nalp.models

Each neural network architecture is defined in this package. From naïve RNNs to BiLSTMs, from GANs to TextGANs, you can use whatever suits your needs.

A package for already-implemented machine learning architectures, divided into discriminators and generators models.

class nalp.models.DCGAN(input_shape: Optional[Tuple[int, int, int]] = (28, 28, 1), noise_dim: Optional[int] = 100, n_samplings: Optional[int] = 3, alpha: Optional[float] = 0.3, dropout_rate: Optional[float] = 0.3)

Bases: nalp.core.Adversarial

A DCGAN class is the one in charge of Deep Convolutional Generative Adversarial Networks implementation.

References

A. Radford, L. Metz, S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. Preprint arXiv:1511.06434 (2015).

__init__(self, input_shape: Optional[Tuple[int, int, int]] = (28, 28, 1), noise_dim: Optional[int] = 100, n_samplings: Optional[int] = 3, alpha: Optional[float] = 0.3, dropout_rate: Optional[float] = 0.3)

Initialization method.

Parameters

  • input_shape – An input shape for the Generator.

  • noise_dim – Amount of noise dimensions for the Generator.

  • n_samplings – Number of down/up samplings to perform.

  • alpha – LeakyReLU activation threshold.

  • dropout_rate – Dropout activation rate.

class nalp.models.GAN(input_shape: Optional[Tuple[int, Ellipsis]] = (784,), noise_dim: Optional[int] = 100, n_samplings: Optional[int] = 3, alpha: Optional[float] = 0.01)

Bases: nalp.core.Adversarial

A GAN class is the one in charge of naïve Generative Adversarial Networks implementation.

References

I. Goodfellow, et al. Generative adversarial nets. Advances in neural information processing systems (2014).

__init__(self, input_shape: Optional[Tuple[int, Ellipsis]] = (784,), noise_dim: Optional[int] = 100, n_samplings: Optional[int] = 3, alpha: Optional[float] = 0.01)

Initialization method.

Parameters

  • input_shape – An input shape for the Generator.

  • noise_dim – Amount of noise dimensions for the Generator.

  • n_samplings – Number of down/up samplings to perform.

  • alpha – LeakyReLU activation threshold.

class nalp.models.GSGAN(encoder: Optional[nalp.encoders.integer.IntegerEncoder] = None, vocab_size: Optional[int] = 1, embedding_size: Optional[int] = 32, hidden_size: Optional[int] = 64, tau: Optional[float] = 5)

Bases: nalp.core.Adversarial

A GSGAN class is the one in charge of Gumbel-Softmax Generative Adversarial Networks implementation.

References

M. Kusner, J. Hernández-Lobato. Gans for sequences of discrete elements with the gumbel-softmax distribution. Preprint arXiv:1611.04051 (2016).

G_pre_step(self, x: tensorflow.Tensor, y: tensorflow.Tensor)

Performs a single batch optimization pre-fitting step over the generator.

Parameters

  • x – A tensor containing the inputs.

  • y – A tensor containing the inputs’ labels.

__init__(self, encoder: Optional[nalp.encoders.integer.IntegerEncoder] = None, vocab_size: Optional[int] = 1, embedding_size: Optional[int] = 32, hidden_size: Optional[int] = 64, tau: Optional[float] = 5)

Initialization method.

Parameters

  • encoder – An index to vocabulary encoder for the generator.

  • vocab_size – The size of the vocabulary for both discriminator and generator.

  • embedding_size – The size of the embedding layer for both discriminator and generator.

  • hidden_size – The amount of hidden neurons for the generator.

  • tau – Gumbel-Softmax temperature parameter.

_discriminator_loss(self, y_real: tensorflow.Tensor, y_fake: tensorflow.Tensor)

Calculates the loss out of the discriminator architecture.

Parameters

  • y_real – A tensor containing the real data targets.

  • y_fake – A tensor containing the fake data targets.

Returns

The loss based on the discriminator network.

Return type

(tf.Tensor)

_generator_loss(self, y_fake: tensorflow.Tensor)

Calculates the loss out of the generator architecture.

Parameters

y_fake – A tensor containing the fake data targets.

Returns

The loss based on the generator network.

Return type

(tf.Tensor)

compile(self, pre_optimizer: tensorflow.keras.optimizers, d_optimizer: tensorflow.keras.optimizers, g_optimizer: tensorflow.keras.optimizers)

Main building method.

Parameters

  • pre_optimizer – An optimizer instance for pre-training the generator.

  • d_optimizer – An optimizer instance for the discriminator.

  • g_optimizer – An optimizer instance for the generator.

fit(self, batches: nalp.core.dataset.Dataset, epochs: Optional[int] = 100)

Trains the model.

Parameters

  • batches – Training batches containing samples.

  • epochs – The maximum number of training epochs.

generate_batch(self, x: tensorflow.Tensor)

Generates a batch of tokens by feeding to the network the current token (t) and predicting the next token (t+1).

Parameters

x (tf.tensor) – A tensor containing the inputs.

Returns

A (batch_size, length) tensor of generated tokens and a (batch_size, length, vocab_size) tensor of predictions.

Return type

(Tuple[tf.Tensor, tf.Tensor])

property init_tau(self)

Gumbel-Softmax initial temperature.

pre_fit(self, batches: nalp.core.dataset.Dataset, epochs: Optional[int] = 100)

Pre-trains the model.

Parameters

  • batches – Pre-training batches containing samples.

  • epochs – The maximum number of pre-training epochs.

step(self, x: tensorflow.Tensor, y: tensorflow.Tensor)

Performs a single batch optimization step.

Parameters

  • x – A tensor containing the inputs.

  • y – A tensor containing the inputs’ labels.

property vocab_size(self)

The size of the vocabulary.

class nalp.models.MaliGAN(encoder: Optional[nalp.encoders.integer.IntegerEncoder] = None, vocab_size: Optional[int] = 1, max_length: Optional[int] = 1, embedding_size: Optional[int] = 32, hidden_size: Optional[int] = 64, n_filters: Optional[Tuple[int, Ellipsis]] = 64, filters_size: Optional[Tuple[int, Ellipsis]] = 1, dropout_rate: Optional[float] = 0.25, temperature: Optional[float] = 1.0)

Bases: nalp.core.Adversarial

A MaliGAN class is the one in charge of Maximum-Likelihood Augmented Discrete Generative Adversarial Networks implementation.

References

T. Che, et al. Maximum-likelihood augmented discrete generative adversarial networks. Preprint arXiv:1702.07983 (2017).

D_step(self, x: tensorflow.Tensor, y: tensorflow.Tensor)

Performs a single batch optimization step over the discriminator.

Parameters

  • x – A tensor containing the inputs.

  • y – A tensor containing the inputs’ labels.

G_pre_step(self, x: tensorflow.Tensor, y: tensorflow.Tensor)

Performs a single batch optimization pre-fitting step over the generator.

Parameters

  • x – A tensor containing the inputs.

  • y – A tensor containing the inputs’ labels.

G_step(self, x: tensorflow.Tensor, y: tensorflow.Tensor, rewards: tensorflow.Tensor)

Performs a single batch optimization step over the generator.

Parameters

  • x – A tensor containing the inputs.

  • y – A tensor containing the inputs’ labels.

  • rewards – A tensor containing the rewards for the input.

property T(self)

Temperature value to sample the token.

__init__(self, encoder: Optional[nalp.encoders.integer.IntegerEncoder] = None, vocab_size: Optional[int] = 1, max_length: Optional[int] = 1, embedding_size: Optional[int] = 32, hidden_size: Optional[int] = 64, n_filters: Optional[Tuple[int, Ellipsis]] = 64, filters_size: Optional[Tuple[int, Ellipsis]] = 1, dropout_rate: Optional[float] = 0.25, temperature: Optional[float] = 1.0)

Initialization method.

Parameters

  • encoder – An index to vocabulary encoder for the generator.

  • vocab_size – The size of the vocabulary for both discriminator and generator.

  • max_length – Maximum length of the sequences for the discriminator.

  • embedding_size – The size of the embedding layer for both discriminator and generator.

  • hidden_size – The amount of hidden neurons for the generator.

  • n_filters – Number of filters to be applied in the discriminator.

  • filters_size – Size of filters to be applied in the discriminator.

  • dropout_rate – Dropout activation rate.

  • temperature – Temperature value to sample the token.

_get_reward(self, x: tensorflow.Tensor)

Calculates rewards over an input using a Maximum-Likelihood approach.

Parameters

x – A tensor containing the inputs.

Returns

Reward over input.

Return type

(tf.Tensor)

compile(self, pre_optimizer: tensorflow.keras.optimizers, d_optimizer: tensorflow.keras.optimizers, g_optimizer: tensorflow.keras.optimizers)

Main building method.

Parameters

  • pre_optimizer – An optimizer instance for pre-training the generator.

  • d_optimizer – An optimizer instance for the discriminator.

  • g_optimizer – An optimizer instance for the generator.

fit(self, batches: nalp.core.dataset.Dataset, epochs: Optional[int] = 10, d_epochs: Optional[int] = 5)

Trains the model.

Parameters

  • batches – Training batches containing samples.

  • epochs – The maximum number of total training epochs.

  • d_epochs – The maximum number of discriminator epochs per total epoch.

generate_batch(self, batch_size: Optional[int] = 1, length: Optional[int] = 1)

Generates a batch of tokens by feeding to the network the current token (t) and predicting the next token (t+1).

Parameters

  • batch_size – Size of the batch to be generated.

  • length – Length of generated tokens.

Returns

A (batch_size, length) tensor of generated tokens.

Return type

(tf.Tensor)

pre_fit(self, batches: nalp.core.dataset.Dataset, g_epochs: Optional[int] = 50, d_epochs: Optional[int] = 10)

Pre-trains the model.

Parameters

  • batches – Pre-training batches containing samples.

  • g_epochs – The maximum number of pre-training generator epochs.

  • d_epochs – The maximum number of pre-training discriminator epochs.

property vocab_size(self)

The size of the vocabulary.

class nalp.models.RelGAN(encoder: Optional[nalp.encoders.integer.IntegerEncoder] = None, vocab_size: Optional[int] = 1, max_length: Optional[int] = 1, embedding_size: Optional[int] = 32, n_slots: Optional[int] = 3, n_heads: Optional[int] = 5, head_size: Optional[int] = 10, n_blocks: Optional[int] = 1, n_layers: Optional[int] = 3, n_filters: Optional[Tuple[int, Ellipsis]] = 64, filters_size: Optional[Tuple[int, Ellipsis]] = 1, dropout_rate: Optional[float] = 0.25, tau: Optional[float] = 5.0)

Bases: nalp.core.Adversarial

A RelGAN class is the one in charge of Relational Generative Adversarial Networks implementation.

References

W. Nie, N. Narodytska, A. Patel. Relgan: Relational generative adversarial networks for text generation. International Conference on Learning Representations (2018).

G_pre_step(self, x: tensorflow.Tensor, y: tensorflow.Tensor)

Performs a single batch optimization pre-fitting step over the generator.

Parameters

  • x – A tensor containing the inputs.

  • y – A tensor containing the inputs’ labels.

__init__(self, encoder: Optional[nalp.encoders.integer.IntegerEncoder] = None, vocab_size: Optional[int] = 1, max_length: Optional[int] = 1, embedding_size: Optional[int] = 32, n_slots: Optional[int] = 3, n_heads: Optional[int] = 5, head_size: Optional[int] = 10, n_blocks: Optional[int] = 1, n_layers: Optional[int] = 3, n_filters: Optional[Tuple[int, Ellipsis]] = 64, filters_size: Optional[Tuple[int, Ellipsis]] = 1, dropout_rate: Optional[float] = 0.25, tau: Optional[float] = 5.0)

Initialization method.

Parameters

  • encoder – An index to vocabulary encoder for the generator.

  • vocab_size – The size of the vocabulary for both discriminator and generator.

  • max_length – Maximum length of the sequences for the discriminator.

  • embedding_size – The size of the embedding layer for both discriminator and generator.

  • n_slots – Number of memory slots for the generator.

  • n_heads – Number of attention heads for the generator.

  • head_size – Size of each attention head for the generator.

  • n_blocks – Number of feed-forward networks for the generator.

  • n_layers – Amout of layers per feed-forward network for the generator.

  • n_filters – Number of filters to be applied in the discriminator.

  • filters_size – Size of filters to be applied in the discriminator.

  • dropout_rate – Dropout activation rate.

  • tau – Gumbel-Softmax temperature parameter.

_discriminator_loss(self, y_real: tensorflow.Tensor, y_fake: tensorflow.Tensor)

Calculates the loss out of the discriminator architecture.

Parameters

  • y_real – A tensor containing the real data targets.

  • y_fake – A tensor containing the fake data targets.

Returns

The loss based on the discriminator network.

Return type

(tf.Tensor)

_generator_loss(self, y_real: tensorflow.Tensor, y_fake: tensorflow.Tensor)

Calculates the loss out of the generator architecture.

Parameters

  • y_real – A tensor containing the real data targets.

  • y_fake – A tensor containing the fake data targets.

Returns

The loss based on the generator network.

Return type

(tf.Tensor)

compile(self, pre_optimizer: tensorflow.keras.optimizers, d_optimizer: tensorflow.keras.optimizers, g_optimizer: tensorflow.keras.optimizers)

Main building method.

Parameters

  • pre_optimizer – An optimizer instance for pre-training the generator.

  • d_optimizer – An optimizer instance for the discriminator.

  • g_optimizer – An optimizer instance for the generator.

fit(self, batches: nalp.core.dataset.Dataset, epochs: Optional[int] = 100)

Trains the model.

Parameters

  • batches – Training batches containing samples.

  • epochs – The maximum number of training epochs.

generate_batch(self, x: tensorflow.Tensor)

Generates a batch of tokens by feeding to the network the current token (t) and predicting the next token (t+1).

Parameters

x – A tensor containing the inputs.

Returns

A (batch_size, length) tensor of generated tokens and a (batch_size, length, vocab_size) tensor of predictions.

Return type

(Tuple[tf.Tensor, tf.Tensor])

property init_tau(self)

Gumbel-Softmax initial temperature.

pre_fit(self, batches: nalp.core.dataset.Dataset, epochs: Optional[int] = 100)

Pre-trains the model.

Parameters

  • batches – Pre-training batches containing samples.

  • epochs – The maximum number of pre-training epochs.

step(self, x: tensorflow.Tensor, y: tensorflow.Tensor)

Performs a single batch optimization step.

Parameters

  • x – A tensor containing the inputs.

  • y – A tensor containing the inputs’ labels.

property vocab_size(self)

The size of the vocabulary.

class nalp.models.SeqGAN(encoder: Optional[nalp.encoders.integer.IntegerEncoder] = None, vocab_size: Optional[int] = 1, max_length: Optional[int] = 1, embedding_size: Optional[int] = 32, hidden_size: Optional[int] = 64, n_filters: Optional[Tuple[int, Ellipsis]] = 64, filters_size: Optional[Tuple[int, Ellipsis]] = 1, dropout_rate: Optional[float] = 0.25, temperature: Optional[float] = 1.0)

Bases: nalp.core.Adversarial

A SeqGAN class is the one in charge of Sequence Generative Adversarial Networks implementation.

References

L. Yu, et al. Seqgan: Sequence generative adversarial nets with policy gradient. 31th AAAI Conference on Artificial Intelligence (2017).

D_step(self, x: tensorflow.Tensor, y: tensorflow.Tensor)

Performs a single batch optimization step over the discriminator.

Parameters

  • x – A tensor containing the inputs.

  • y – A tensor containing the inputs’ labels.

G_pre_step(self, x: tensorflow.Tensor, y: tensorflow.Tensor)

Performs a single batch optimization pre-fitting step over the generator.

Parameters

  • x – A tensor containing the inputs.

  • y – A tensor containing the inputs’ labels.

G_step(self, x: tensorflow.Tensor, y: tensorflow.Tensor, rewards: tensorflow.Tensor)

Performs a single batch optimization step over the generator.

Parameters

  • x – A tensor containing the inputs.

  • y – A tensor containing the inputs’ labels.

  • rewards – A tensor containing the rewards for the input.

property T(self)

Temperature value to sample the token.

__init__(self, encoder: Optional[nalp.encoders.integer.IntegerEncoder] = None, vocab_size: Optional[int] = 1, max_length: Optional[int] = 1, embedding_size: Optional[int] = 32, hidden_size: Optional[int] = 64, n_filters: Optional[Tuple[int, Ellipsis]] = 64, filters_size: Optional[Tuple[int, Ellipsis]] = 1, dropout_rate: Optional[float] = 0.25, temperature: Optional[float] = 1.0)

Initialization method.

Parameters

  • encoder – An index to vocabulary encoder for the generator.

  • vocab_size – The size of the vocabulary for both discriminator and generator.

  • max_length – Maximum length of the sequences for the discriminator.

  • embedding_size – The size of the embedding layer for both discriminator and generator.

  • hidden_size – The amount of hidden neurons for the generator.

  • n_filters – Number of filters to be applied in the discriminator.

  • filters_size – Size of filters to be applied in the discriminator.

  • dropout_rate – Dropout activation rate.

  • temperature – Temperature value to sample the token.

_get_reward(self, x: tensorflow.Tensor, n_rollouts: int)

Calculates rewards over an input using a Monte Carlo search strategy.

Parameters

  • x – A tensor containing the inputs.

  • n_rollouts – Number of rollouts for conducting the Monte Carlo search.

Returns

Reward over input.

Return type

(tf.Tensor)

compile(self, pre_optimizer: tensorflow.keras.optimizers, d_optimizer: tensorflow.keras.optimizers, g_optimizer: tensorflow.keras.optimizers)

Main building method.

Parameters

  • pre_optimizer – An optimizer instance for pre-training the generator.

  • d_optimizer – An optimizer instance for the discriminator.

  • g_optimizer – An optimizer instance for the generator.

fit(self, batches: nalp.core.dataset.Dataset, epochs: Optional[int] = 10, g_epochs: Optional[int] = 1, d_epochs: Optional[int] = 5, n_rollouts: Optional[int] = 16)

Trains the model.

Parameters

  • batches – Training batches containing samples.

  • epochs – The maximum number of total training epochs.

  • g_epochs – The maximum number of generator epochs per total epoch.

  • d_epochs – The maximum number of discriminator epochs per total epoch.

  • n_rollouts – Number of rollouts for conducting the Monte Carlo search.

generate_batch(self, batch_size: Optional[int] = 1, length: Optional[int] = 1)

Generates a batch of tokens by feeding to the network the current token (t) and predicting the next token (t+1).

Parameters

  • batch_size – Size of the batch to be generated.

  • length – Length of generated tokens.

Returns

A (batch_size, length) tensor of generated tokens.

Return type

(tf.Tensor)

pre_fit(self, batches: nalp.core.dataset.Dataset, g_epochs: Optional[int] = 50, d_epochs: Optional[int] = 10)

Pre-trains the model.

Parameters

  • batches – Pre-training batches containing samples.

  • g_epochs – The maximum number of pre-training generator epochs.

  • d_epochs – The maximum number of pre-training discriminator epochs.

property vocab_size(self)

The size of the vocabulary.

class nalp.models.WGAN(input_shape: Optional[Tuple[int, int, int]] = (28, 28, 1), noise_dim: Optional[int] = 100, n_samplings: Optional[int] = 3, alpha: Optional[float] = 0.3, dropout_rate: Optional[float] = 0.3, model_type: Optional[str] = 'wc', clip: Optional[float] = 0.01, penalty: Optional[int] = 10)

Bases: nalp.core.Adversarial

A WGAN class is the one in charge of Wasserstein Generative Adversarial Networks implementation with weight clipping or gradient penalty algorithms.

References

M. Arjovsky, S. Chintala, L. Bottou. Wasserstein gan. Preprint arXiv:1701.07875 (2017).

I. Gulrajani, et al. Improved training of wasserstein gans. Advances in neural information processing systems (2017).

D_step(self, x: tensorflow.Tensor)

Performs a single batch optimization step over the discriminator.

Parameters

x – A tensor containing the inputs.

G_step(self, x: tensorflow.Tensor)

Performs a single batch optimization step over the generator.

Parameters

x – A tensor containing the inputs.

__init__(self, input_shape: Optional[Tuple[int, int, int]] = (28, 28, 1), noise_dim: Optional[int] = 100, n_samplings: Optional[int] = 3, alpha: Optional[float] = 0.3, dropout_rate: Optional[float] = 0.3, model_type: Optional[str] = 'wc', clip: Optional[float] = 0.01, penalty: Optional[int] = 10)

Initialization method.

Parameters

  • input_shape – An input shape for the Generator.

  • noise_dim – Amount of noise dimensions for the Generator.

  • n_samplings – Number of down/up samplings to perform.

  • alpha – LeakyReLU activation threshold.

  • dropout_rate – Dropout activation rate.

  • model_type – Whether should use weight clipping (wc) or gradient penalty (gp).

  • clip – Clipping value for the Lipschitz constrain.

  • penalty – Coefficient for the gradient penalty.

_gradient_penalty(self, x: tensorflow.Tensor, x_fake: tensorflow.Tensor)

Performs the gradient penalty procedure.

Parameters

  • x – A tensor containing the real inputs.

  • x_fake – A tensor containing the fake inputs.

Returns

The penalization to be applied over the loss function.

Return type

(tf.Tensor)

property clip(self)

Clipping value for the Lipschitz constrain.

fit(self, batches: nalp.core.dataset.Dataset, epochs: Optional[int] = 100, critic_steps: Optional[int] = 5)

Trains the model.

Parameters

  • batches – Training batches containing samples.

  • epochs – The maximum number of training epochs.

  • critic_steps – Amount of discriminator epochs per training epoch.

property model_type(self)

Whether should use weight clipping (wc) or gradient penalty (gp).

property penalty_lambda(self)

Coefficient for the gradient penalty.