nalp.models.layers¶
Pre-defined layers.
A package for already-implemented machine learning layers.
- class nalp.models.layers.GumbelSoftmax(axis: Optional[int] = -1, **kwargs)¶
Bases:
tensorflow.keras.layers.Layer
A GumbelSoftmax class is the one in charge of a Gumbel-Softmax layer implementation.
References
E. Jang, S. Gu, B. Poole. Categorical reparameterization with gumbel-softmax. Preprint arXiv:1611.01144 (2016).
- __init__(self, axis: Optional[int] = -1, **kwargs)¶
Initialization method.
- Parameters
axis – Axis to perform the softmax operation.
- call(self, inputs: tensorflow.Tensor, tau: float)¶
Method that holds vital information whenever this class is called.
- Parameters
x – A tensorflow’s tensor holding input data.
tau – Gumbel-Softmax temperature parameter.
- Returns
Gumbel-Softmax output and its argmax token.
- Return type
(Tuple[tf.Tensor, tf.Tensor])
- get_config(self)¶
Gets the configuration of the layer for further serialization.
- Returns
Configuration dictionary.
- Return type
(Dict[str, Any])
- class nalp.models.layers.MultiHeadAttention(n_features: int, n_heads: int, **kwargs)¶
Bases:
tensorflow.keras.layers.Layer
A MultiHeadAttention class is the one in charge of a Multi-Head Attention layer implementation.
References
A.Vaswani, et al. Attention is all you need. Advances in neural information processing systems (2017).
- __init__(self, n_features: int, n_heads: int, **kwargs)¶
Initialization method.
- Parameters
n_features – Number of input features.
n_heads – Number of attention heads.
- _split_heads(self, x: tensorflow.Tensor)¶
Split the last tensor dimension into (n_heads, depth) and transposes its result.
- Parameters
x – Tensor to be splitted and transposed.
- Returns
Splitted and transposed tensor into shape equal to (batch_size, n_heads, length, depth).
- Return type
(tf.Tensor)
- call(self, q: tensorflow.Tensor, k: tensorflow.Tensor, v: tensorflow.Tensor, mask: Optional[tensorflow.Tensor] = None)¶
Method that holds vital information whenever this class is called.
- Parameters
q – Query tensor.
k – Key tensor.
v – Value tensor.
mask – Mask to be applied.
- Returns
An attention-based output tensor and its attention weights.
- Return type
(Tuple[tf.Tensor, tf.Tensor])
- get_config(self)¶
Gets the configuration of the layer for further serialization.
- Returns
Configuration dictionary.
- Return type
(Dict[str, Any])
- class nalp.models.layers.RelationalMemoryCell(n_slots: int, n_heads: int, head_size: int, n_blocks: Optional[int] = 1, n_layers: Optional[int] = 3, activation: Optional[str] = 'tanh', recurrent_activation: Optional[str] = 'hard_sigmoid', forget_bias: Optional[float] = 1.0, kernel_initializer: Optional[str] = 'glorot_uniform', recurrent_initializer: Optional[str] = 'orthogonal', bias_initializer: Optional[str] = 'zeros', kernel_regularizer: Optional[str] = None, recurrent_regularizer: Optional[str] = None, bias_regularizer: Optional[str] = None, kernel_constraint: Optional[str] = None, recurrent_constraint: Optional[str] = None, bias_constraint: Optional[str] = None, **kwargs)¶
Bases:
tensorflow.keras.layers.AbstractRNNCell
A RelationalMemoryCell class is the one in charge of a Relational Memory cell implementation.
References
A. Santoro, et al. Relational recurrent neural networks. Advances in neural information processing systems (2018).
- __init__(self, n_slots: int, n_heads: int, head_size: int, n_blocks: Optional[int] = 1, n_layers: Optional[int] = 3, activation: Optional[str] = 'tanh', recurrent_activation: Optional[str] = 'hard_sigmoid', forget_bias: Optional[float] = 1.0, kernel_initializer: Optional[str] = 'glorot_uniform', recurrent_initializer: Optional[str] = 'orthogonal', bias_initializer: Optional[str] = 'zeros', kernel_regularizer: Optional[str] = None, recurrent_regularizer: Optional[str] = None, bias_regularizer: Optional[str] = None, kernel_constraint: Optional[str] = None, recurrent_constraint: Optional[str] = None, bias_constraint: Optional[str] = None, **kwargs)¶
Initialization method.
- Parameters
n_slots – Number of memory slots.
n_heads – Number of attention heads.
head_size – Size of each attention head.
n_blocks – Number of feed-forward networks.
n_layers – Amout of layers per feed-forward network.
activation – Output activation function.
recurrent_activation – Recurrent step activation function.
forget_bias – Forget gate bias values.
kernel_initializer – Kernel initializer function.
recurrent_initializer – Recurrent kernel initializer function.
bias_initializer – Bias initializer function.
kernel_regularizer – Kernel regularizer function.
recurrent_regularizer – Recurrent kernel regularizer function.
bias_regularizer – Bias regularizer function.
kernel_constraint – Kernel constraint function.
recurrent_constraint – Recurrent kernel constraint function.
bias_constraint – Bias constraint function.
- _attend_over_memory(self, inputs: tensorflow.Tensor, memory: tensorflow.Tensor)¶
Performs an Attention mechanism over the current memory.
- Parameters
inputs – An input tensor.
memory – Current memory tensor.
- Returns
Updated current memory based on Multi-Head Attention mechanism.
- Return type
(tf.Tensor)
- build(self, input_shape: tensorflow.Tensor)¶
Builds up the cell according to its input shape.
- Parameters
input_shape – Tensor holding the input shape.
- call(self, inputs: tensorflow.Tensor, states: List[tensorflow.Tensor])¶
Method that holds vital information whenever this class is called.
- Parameters
inputs – An input tensor.
states – A list holding previous states and memories.
- Returns
Output states as well as current state and memory.
- Return type
(Tuple[tf.Tensor, List[tf.Tensor]])
- get_config(self)¶
Gets the configuration of the layer for further serialization.
- Returns
Configuration dictionary.
- Return type
(Dict[str, Any])
- get_initial_state(self, inputs: Optional[tensorflow.Tensor] = None, batch_size: Optional[int] = None, dtype: Optional[tensorflow.DType] = None)¶
Gets the cell initial state by creating an identity matrix.
- Parameters
inputs – An input tensor.
batch_size – Size of the batch.
dtype – Dtype from input tensor.
- Returns
Initial states.
- Return type
(Tuple[tf.Tensor, tf.Tensor])
- property output_size(self)¶
- property state_size(self)¶