weight_norm

tfsnippet.layers.weight_norm(*args, **kwargs)

Weight normalization proposed by (Salimans & Kingma, 2016).

Roughly speaking, the weight normalization is defined as:

kernel = scale * kernel / tf.sqrt(
    tf.reduce_sum(kernel ** 2, axis=<dimensions not in `axis`>,
                  keepdims=True)
)

This function does not support data-dependent initialization for scale. If you do need this feature, you have to turn off scale, and use act_norm() along with weight_norm().

Parameters:
  • kernel – Tensor, the weight w to be normalized.
  • axis (int or tuple[int]) – The axis to apply weight normalization. See above description to know what axis exactly is.
  • use_scale (bool) – Whether or not to use scale. Default True.
  • scale (Tensor) – Instead of creating a new variable, use this tensor.
  • scale_initializer – The initializer for scale.
  • scale_regularizer – The regularizer for scale.
  • scale_constraint – The constraint for scale.
  • trainable (bool) – Whether or not the variables are trainable?
  • epsilon – Small float number to avoid dividing by zero.
  • name (str) – Default name of the variable scope. Will be uniquified. If not specified, generate one according to the class name.
  • scope (str) – The name of the variable scope.