weight_norm¶
-
tfsnippet.layers.
weight_norm
(*args, **kwargs)¶ Weight normalization proposed by (Salimans & Kingma, 2016).
Roughly speaking, the weight normalization is defined as:
kernel = scale * kernel / tf.sqrt( tf.reduce_sum(kernel ** 2, axis=<dimensions not in `axis`>, keepdims=True) )
This function does not support data-dependent initialization for scale. If you do need this feature, you have to turn off scale, and use
act_norm()
along withweight_norm()
.Parameters: - kernel – Tensor, the weight w to be normalized.
- axis (int or tuple[int]) – The axis to apply weight normalization. See above description to know what axis exactly is.
- use_scale (bool) – Whether or not to use scale. Default
True
. - scale (Tensor) – Instead of creating a new variable, use this tensor.
- scale_initializer – The initializer for scale.
- scale_regularizer – The regularizer for scale.
- scale_constraint – The constraint for scale.
- trainable (bool) – Whether or not the variables are trainable?
- epsilon – Small float number to avoid dividing by zero.
- name (str) – Default name of the variable scope. Will be uniquified. If not specified, generate one according to the class name.
- scope (str) – The name of the variable scope.