nn_rnn function

RNN module

Applies a multi-layer Elman RNN with $\tanh$ or $\mbox{ReLU}$ non-linearity to an input sequence.

nn_rnn(
  input_size,
  hidden_size,
  num_layers = 1,
  nonlinearity = NULL,
  bias = TRUE,
  batch_first = FALSE,
  dropout = 0,
  bidirectional = FALSE,
  ...
)

Arguments

• input_size: The number of expected features in the input x

• hidden_size: The number of features in the hidden state h

• num_layers: Number of recurrent layers. E.g., setting num_layers=2

  would mean stacking two RNNs together to form a stacked RNN, with the second RNN taking in outputs of the first RNN and computing the final results. Default: 1

• nonlinearity: The non-linearity to use. Can be either 'tanh' or 'relu'. Default: 'tanh'

• bias: If FALSE, then the layer does not use bias weights b_ih and b_hh. Default: TRUE

• batch_first: If TRUE, then the input and output tensors are provided as (batch, seq, feature). Default: FALSE

• dropout: If non-zero, introduces a Dropout layer on the outputs of each RNN layer except the last layer, with dropout probability equal to dropout. Default: 0

• bidirectional: If TRUE, becomes a bidirectional RNN. Default: FALSE

• ...: other arguments that can be passed to the super class.

Details

For each element in the input sequence, each layer computes the following function:

where $h_t$ is the hidden state at time t, $x_t$ is the input at time t, and $h_{(t-1)}$ is the hidden state of the previous layer at time t-1 or the initial hidden state at time 0. If nonlinearity is 'relu', then $\mbox{ReLU}$ is used instead of $\tanh$.

Inputs

• input of shape (seq_len, batch, input_size): tensor containing the features of the input sequence. The input can also be a packed variable length sequence.
• h_0 of shape (num_layers * num_directions, batch, hidden_size): tensor containing the initial hidden state for each element in the batch. Defaults to zero if not provided. If the RNN is bidirectional, num_directions should be 2, else it should be 1.

Outputs

• output of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features (h_t) from the last layer of the RNN, for each t. If a :class:nn_packed_sequence has been given as the input, the output will also be a packed sequence. For the unpacked case, the directions can be separated using output$view(seq_len, batch, num_directions, hidden_size), with forward and backward being direction 0 and 1 respectively. Similarly, the directions can be separated in the packed case.
• h_n of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len. Like output, the layers can be separated using h_n$view(num_layers, num_directions, batch, hidden_size).

Shape

• Input1: $(L, N, H_{in})$ tensor containing input features where $H_{in}=\mbox{input\_size}$ and L represents a sequence length.

• Input2: $(S, N, H_{out})$ tensor containing the initial hidden state for each element in the batch. $H_{out}=\mbox{hidden\_size}$

  Defaults to zero if not provided. where $S=\mbox{num\_layers} * \mbox{num\_directions}$

  If the RNN is bidirectional, num_directions should be 2, else it should be 1.

• Output1: $(L, N, H_{all})$ where $H_{all}=\mbox{num\_directions} * \mbox{hidden\_size}$

• Output2: $(S, N, H_{out})$ tensor containing the next hidden state for each element in the batch

Attributes

• weight_ih_l[k]: the learnable input-hidden weights of the k-th layer, of shape (hidden_size, input_size) for k = 0. Otherwise, the shape is (hidden_size, num_directions * hidden_size)
• weight_hh_l[k]: the learnable hidden-hidden weights of the k-th layer, of shape (hidden_size, hidden_size)
• bias_ih_l[k]: the learnable input-hidden bias of the k-th layer, of shape (hidden_size)
• bias_hh_l[k]: the learnable hidden-hidden bias of the k-th layer, of shape (hidden_size)

Note

All the weights and biases are initialized from $\mathcal{U}(-\sqrt{k}, \sqrt{k})$

where $k = \frac{1}{\mbox{hidden\_size}}$

Examples

if (torch_is_installed()) {
rnn <- nn_rnn(10, 20, 2)
input <- torch_randn(5, 3, 10)
h0 <- torch_randn(2, 3, 20)
rnn(input, h0)
}