nalp.models.layers

Pre-defined layers.

A package for already-implemented machine learning layers.

class nalp.models.layers.GumbelSoftmax(axis: Optional[int] = -1, **kwargs)

Bases: tensorflow.keras.layers.Layer

A GumbelSoftmax class is the one in charge of a Gumbel-Softmax layer implementation.

References

E. Jang, S. Gu, B. Poole. Categorical reparameterization with gumbel-softmax. Preprint arXiv:1611.01144 (2016).

__init__(self, axis: Optional[int] = -1, **kwargs)

Initialization method.

Parameters

axis – Axis to perform the softmax operation.

call(self, inputs: tensorflow.Tensor, tau: float)

Method that holds vital information whenever this class is called.

Parameters

  • x – A tensorflow’s tensor holding input data.

  • tau – Gumbel-Softmax temperature parameter.

Returns

Gumbel-Softmax output and its argmax token.

Return type

(Tuple[tf.Tensor, tf.Tensor])

get_config(self)

Gets the configuration of the layer for further serialization.

Returns

Configuration dictionary.

Return type

(Dict[str, Any])

class nalp.models.layers.MultiHeadAttention(n_features: int, n_heads: int, **kwargs)

Bases: tensorflow.keras.layers.Layer

A MultiHeadAttention class is the one in charge of a Multi-Head Attention layer implementation.

References

A.Vaswani, et al. Attention is all you need. Advances in neural information processing systems (2017).

__init__(self, n_features: int, n_heads: int, **kwargs)

Initialization method.

Parameters

  • n_features – Number of input features.

  • n_heads – Number of attention heads.

_split_heads(self, x: tensorflow.Tensor)

Split the last tensor dimension into (n_heads, depth) and transposes its result.

Parameters

x – Tensor to be splitted and transposed.

Returns

Splitted and transposed tensor into shape equal to (batch_size, n_heads, length, depth).

Return type

(tf.Tensor)

call(self, q: tensorflow.Tensor, k: tensorflow.Tensor, v: tensorflow.Tensor, mask: Optional[tensorflow.Tensor] = None)

Method that holds vital information whenever this class is called.

Parameters

  • q – Query tensor.

  • k – Key tensor.

  • v – Value tensor.

  • mask – Mask to be applied.

Returns

An attention-based output tensor and its attention weights.

Return type

(Tuple[tf.Tensor, tf.Tensor])

get_config(self)

Gets the configuration of the layer for further serialization.

Returns

Configuration dictionary.

Return type

(Dict[str, Any])

class nalp.models.layers.RelationalMemoryCell(n_slots: int, n_heads: int, head_size: int, n_blocks: Optional[int] = 1, n_layers: Optional[int] = 3, activation: Optional[str] = 'tanh', recurrent_activation: Optional[str] = 'hard_sigmoid', forget_bias: Optional[float] = 1.0, kernel_initializer: Optional[str] = 'glorot_uniform', recurrent_initializer: Optional[str] = 'orthogonal', bias_initializer: Optional[str] = 'zeros', kernel_regularizer: Optional[str] = None, recurrent_regularizer: Optional[str] = None, bias_regularizer: Optional[str] = None, kernel_constraint: Optional[str] = None, recurrent_constraint: Optional[str] = None, bias_constraint: Optional[str] = None, **kwargs)

Bases: tensorflow.keras.layers.AbstractRNNCell

A RelationalMemoryCell class is the one in charge of a Relational Memory cell implementation.

References

A. Santoro, et al. Relational recurrent neural networks. Advances in neural information processing systems (2018).

__init__(self, n_slots: int, n_heads: int, head_size: int, n_blocks: Optional[int] = 1, n_layers: Optional[int] = 3, activation: Optional[str] = 'tanh', recurrent_activation: Optional[str] = 'hard_sigmoid', forget_bias: Optional[float] = 1.0, kernel_initializer: Optional[str] = 'glorot_uniform', recurrent_initializer: Optional[str] = 'orthogonal', bias_initializer: Optional[str] = 'zeros', kernel_regularizer: Optional[str] = None, recurrent_regularizer: Optional[str] = None, bias_regularizer: Optional[str] = None, kernel_constraint: Optional[str] = None, recurrent_constraint: Optional[str] = None, bias_constraint: Optional[str] = None, **kwargs)

Initialization method.

Parameters

  • n_slots – Number of memory slots.

  • n_heads – Number of attention heads.

  • head_size – Size of each attention head.

  • n_blocks – Number of feed-forward networks.

  • n_layers – Amout of layers per feed-forward network.

  • activation – Output activation function.

  • recurrent_activation – Recurrent step activation function.

  • forget_bias – Forget gate bias values.

  • kernel_initializer – Kernel initializer function.

  • recurrent_initializer – Recurrent kernel initializer function.

  • bias_initializer – Bias initializer function.

  • kernel_regularizer – Kernel regularizer function.

  • recurrent_regularizer – Recurrent kernel regularizer function.

  • bias_regularizer – Bias regularizer function.

  • kernel_constraint – Kernel constraint function.

  • recurrent_constraint – Recurrent kernel constraint function.

  • bias_constraint – Bias constraint function.

_attend_over_memory(self, inputs: tensorflow.Tensor, memory: tensorflow.Tensor)

Performs an Attention mechanism over the current memory.

Parameters

  • inputs – An input tensor.

  • memory – Current memory tensor.

Returns

Updated current memory based on Multi-Head Attention mechanism.

Return type

(tf.Tensor)

build(self, input_shape: tensorflow.Tensor)

Builds up the cell according to its input shape.

Parameters

input_shape – Tensor holding the input shape.

call(self, inputs: tensorflow.Tensor, states: List[tensorflow.Tensor])

Method that holds vital information whenever this class is called.

Parameters

  • inputs – An input tensor.

  • states – A list holding previous states and memories.

Returns

Output states as well as current state and memory.

Return type

(Tuple[tf.Tensor, List[tf.Tensor]])

get_config(self)

Gets the configuration of the layer for further serialization.

Returns

Configuration dictionary.

Return type

(Dict[str, Any])

get_initial_state(self, inputs: Optional[tensorflow.Tensor] = None, batch_size: Optional[int] = None, dtype: Optional[tensorflow.DType] = None)

Gets the cell initial state by creating an identity matrix.

Parameters

  • inputs – An input tensor.

  • batch_size – Size of the batch.

  • dtype – Dtype from input tensor.

Returns

Initial states.

Return type

(Tuple[tf.Tensor, tf.Tensor])

property output_size(self)
property state_size(self)