Understanding RNNs & Transformers in Detail:

Predicting the Next Letter in a Sequence

We have been focusing on NLP on today article and our other two articles of Natural Language Processing (NLP) -RNN – Day 63 & The Revolution of Transformer Models – day 65.

In this article explanation, we’ll delve deeply into how Recurrent Neural Networks (RNNs) and Transformers work, especially in the context of predicting the next letter “D” in the sequence “A B C”. We’ll walk through every step, including actual numerical calculations for a simple example, to make the concepts clear. We’ll also explain why Transformers are considered as neural networks and how they fit into the broader context of deep learning.

Recurrent Neural Networks (RNNs)

Introduction to RNNs

RNNs are a type of neural network designed to process sequential data by maintaining a hidden state that captures information about previous inputs. This makes them suitable for tasks like language modeling, where the context provided by earlier letters influences the prediction of the next letter.

Problem Statement

Lets say, Given the sequence of “A B C”, we want the RNN to predict the next letter, which is “D”.

Input Representation

We need to represent each letter numerically. We’ll use one-hot encoding for simplicity. Let’s define our vocabulary as letters A, B, C, D.

A: $\begin{bmatrix} 1, 0, 0, 0 \end{bmatrix}$

B: $\begin{bmatrix} 0, 1, 0, 0 \end{bmatrix}$

C: $\begin{bmatrix} 0, 0, 1, 0 \end{bmatrix}$

D: $\begin{bmatrix} 0, 0, 0, 1 \end{bmatrix}$

Network Architecture and Parameters

Lets define the architecture and set the parameters for simplicity. The input size $n_x = 4$ (since we have four letters), the hidden size $n_h = 2$ , and the output size $n_y = 4$ .

**Weights and Biases**:

Input-to-Hidden Weights $W_{xh}$ :

$W_{xh} = \begin{bmatrix} 0.1 & 0.2 & 0.3 & 0.4 \\ 0.5 & 0.6 & 0.7 & 0.8 \end{bmatrix}$

Hidden-to-Hidden Weights $W_{hh}$ :

$W_{hh} = \begin{bmatrix} 0.1 & 0.2 \\ 0.3 & 0.4 \end{bmatrix}$

Hidden-to-Output Weights $W_{hy}$ :

$W_{hy} = \begin{bmatrix} 0.1 & 0.2 \\ 0.3 & 0.4 \\ 0.5 & 0.6 \\ 0.7 & 0.8 \end{bmatrix}$

Biases:

Hidden bias $b_h = \begin{bmatrix} 0 \\ 0 \end{bmatrix}$ , Output bias $b_y = \begin{bmatrix} 0 \\ 0 \\ 0 \\ 0 \end{bmatrix}$

Forward Pass

We’ll process the input sequence one time step at a time.

Time Step 1: Processing ‘A’

The input for ‘A’ is represented as $x_1 = \begin{bmatrix} 1 \\ 0 \\ 0 \\ 0 \end{bmatrix}$ .

To compute the hidden state:

$W_{xh} \cdot x_1 = \begin{bmatrix} 0.1 & 0.2 & 0.3 & 0.4 \\ 0.5 & 0.6 & 0.7 & 0.8 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 0 \\ 0 \\ 0 \end{bmatrix} = \begin{bmatrix} 0.1 \\ 0.5 \end{bmatrix}$

The hidden state update at time step 1 is:

$h_1 = \tanh(W_{xh} \cdot x_1 + W_{hh} \cdot h_0 + b_h) = \tanh\left(\begin{bmatrix} 0.1 \\ 0.5 \end{bmatrix}\right) = \begin{bmatrix} 0.0997 \\ 0.4621 \end{bmatrix}$

Time Step 2: Processing ‘B’

The input for ‘B’ is $x_2 = \begin{bmatrix} 0 \\ 1 \\ 0 \\ 0 \end{bmatrix}$ .

The hidden state update at time step 2 is:

$h_2 = \tanh(W_{xh} \cdot x_2 + W_{hh} \cdot h_1 + b_h) = \tanh\left(\begin{bmatrix} 0.30239 \\ 0.81476 \end{bmatrix}\right) = \begin{bmatrix} 0.2935 \\ 0.6729 \end{bmatrix}$

Time Step 3: Processing ‘C’

The input for ‘C’ is $x_3 = \begin{bmatrix} 0 \\ 0 \\ 1 \\ 0 \end{bmatrix}$ .

The hidden state update at time step 3 is:

$h_3 = \tanh(W_{xh} \cdot x_3 + W_{hh} \cdot h_2 + b_h) = \tanh\left(\begin{bmatrix} 0.46393 \\ 1.05721 \end{bmatrix}\right) = \begin{bmatrix} 0.4339 \\ 0.7843 \end{bmatrix}$

Output Prediction

We now predict the output based on the final hidden state $h_3$ . The output is computed as:

$o_3 = W_{hy} \cdot h_3 + b_y = \begin{bmatrix} 0.1 & 0.2 \\ 0.3 & 0.4 \\ 0.5 & 0.6 \\ 0.7 & 0.8 \end{bmatrix} \cdot \begin{bmatrix} 0.4339 \\ 0.7843 \end{bmatrix} = \begin{bmatrix} 0.20025 \\ 0.44389 \\ 0.68753 \\ 0.93117 \end{bmatrix}$

Loss Calculation

We compute the cross-entropy loss by comparing the predicted probabilities with the true label (which is “D”). The cross-entropy loss is calculated as:

$\text{Loss} = -\log(0.3473) = 1.057$

Backpropagation Through Time (BPTT) Continued

During backpropagation, we update the weights by computing gradients of the loss with respect to the parameters. This process is performed across time steps to adjust the weights in the hidden layers, input-to-hidden weights, and hidden-to-output weights.

Step 1: Compute Gradients at Output Layer

The gradient of the loss with respect to the output scores is calculated as:

$\delta o_3 = y_{\text{pred}} - y_{\text{true}} = \begin{bmatrix} 0.1672 \\ 0.2133 \\ 0.2722 \\ -0.6527 \end{bmatrix}$

Step 2: Compute Gradients w.r.t Weights

The gradient with respect to the hidden-to-output weights $W_{hy}$ is calculated as:

$\delta W_{hy} = \delta o_3 \cdot h_3^T$

The bias gradients are:

$\delta b_y = \delta o_3$

Step 3: Compute Gradient w.r.t Hidden State

The gradient with respect to the hidden state $h_3$ is:

$\delta h_3 = W_{hy}^T \cdot \delta o_3$

Step 4: Backpropagate to Previous Time Steps

The error is propagated back to previous time steps by computing the gradients with respect to the previous hidden states. This process continues back in time, updating the weights for the hidden-to-hidden $W_{hh}$ and input-to-hidden $W_{xh}$ matrices.

Step 5: Update Weights

After computing the gradients, we update the weights using gradient descent:

$W \leftarrow W - \eta \cdot \delta W$

where $\eta$ is the learning rate.

& This Process goes on …,

( Please note, this was just a very brief overview of how the NLP would look like with RNN in very simple approach to just have a overlook of the math in mind)

Now Lets See How The Same Example Look In Transformers

How the Same Example Looks in Transformers

1. Introduction to Transformers

Transformers model relationships between sequence elements using self-attention and feed-forward layers, making them highly effective for tasks like sequence prediction. Unlike RNNs, Transformers do not rely on sequential data processing, allowing parallel computations and better handling of long-range dependencies.

2. Problem Statement

We aim to predict the next letter, “D,” given the input sequence “A B C” using a Transformer model.

3. Input Representation

Instead of one-hot encoding, we represent the input letters as embeddings. Let the embedding matrix $E$ (of size $4 \times d_{\text{model}}$ ) map each letter to a 2-dimensional vector:

$E = \begin{bmatrix} 1.0 & 0.0 \\ 0.0 & 1.0 \\ 1.0 & 1.0 \\ 0.0 & 0.0 \end{bmatrix}$

Mapping:

$A \to \begin{bmatrix} 1.0 \\ 0.0 \end{bmatrix}$
$B \to \begin{bmatrix} 0.0 \\ 1.0 \end{bmatrix}$
$C \to \begin{bmatrix} 1.0 \\ 1.0 \end{bmatrix}$
$D \to \begin{bmatrix} 0.0 \\ 0.0 \end{bmatrix}$

4. Self-Attention Mechanism

Step 1: Compute Queries, Keys, and Values

Transformers use Queries (Q), Keys (K), and Values (V) to compute self-attention. The weight matrices $W_Q, W_K, W_V$ are used to linearly transform the embeddings. For simplicity:

$W_Q = W_K = W_V = \begin{bmatrix} 1.0 & 0.0 \\ 0.0 & 1.0 \end{bmatrix}$

For each input $x_i$ , compute:

$Q_i = x_i \cdot W_Q$
$K_i = x_i \cdot W_K$
$V_i = x_i \cdot W_V$

For example, for $C$ :

$x_3 = \begin{bmatrix} 1.0 \\ 1.0 \end{bmatrix}, \, Q_3 = K_3 = V_3 = \begin{bmatrix} 1.0 \\ 1.0 \end{bmatrix}$

Step 2: Compute Attention Scores

Attention scores between $Q_i$ and all $K_j$ are computed using the scaled dot product:

$\text{Score}_{i,j} = \frac{Q_i \cdot K_j^T}{\sqrt{d_k}}$

For simplicity, compute scores for $Q_3$ (for $C$ ):

$\text{Score}_{3,j} = \frac{\begin{bmatrix} 1.0 & 1.0 \end{bmatrix} \cdot K_j}{\sqrt{2}}$

With $A$ : $\frac{(1.0 \cdot 1.0) + (1.0 \cdot 0.0)}{\sqrt{2}} = 0.707$
With $B$ : $\frac{(1.0 \cdot 0.0) + (1.0 \cdot 1.0)}{\sqrt{2}} = 0.707$
With $C$ : $\frac{(1.0 \cdot 1.0) + (1.0 \cdot 1.0)}{\sqrt{2}} = 1.414$

Scores: $\text{Scores}_3 = \begin{bmatrix} 0.707 & 0.707 & 1.414 \end{bmatrix}$

Step 3: Compute Attention Weights

Apply softmax to normalize the scores:

$\alpha_{i,j} = \frac{e^{\text{Score}_{i,j}}}{\sum_k e^{\text{Score}_{i,k}}}$

Softmax for $\text{Scores}_3$ :

$\alpha_{3,j} = \frac{e^{\text{Score}_{3,j}}}{e^{0.707} + e^{0.707} + e^{1.414}}$

Exponential values:

$e^{0.707} \approx 2.028$
$e^{1.414} \approx 4.113$

Normalized weights: $\alpha_{3,j} = \begin{bmatrix} 0.25 & 0.25 & 0.5 \end{bmatrix}$

Step 4: Compute Attention Output

The attention output is a weighted sum of the values:

$\text{Attention Output}_3 = \sum_j \alpha_{3,j} \cdot V_j$

Values:

$V_A = \begin{bmatrix} 1.0 \\ 0.0 \end{bmatrix}$
$V_B = \begin{bmatrix} 0.0 \\ 1.0 \end{bmatrix}$
$V_C = \begin{bmatrix} 1.0 \\ 1.0 \end{bmatrix}$

$\text{Attention Output}_3 = 0.25 \cdot \begin{bmatrix} 1.0 \\ 0.0 \end{bmatrix} + 0.25 \cdot \begin{bmatrix} 0.0 \\ 1.0 \end{bmatrix} + 0.5 \cdot \begin{bmatrix} 1.0 \\ 1.0 \end{bmatrix} = \begin{bmatrix} 0.75 \\ 0.75 \end{bmatrix}$

5. Feed-Forward Neural Network

The attention output is passed through a feed-forward layer:

$\text{FFN}_3 = \text{ReLU}(W_{\text{FFN}} \cdot \text{Attention Output}_3 + b_{\text{FFN}})$

Assuming identity weights and zero bias:

$\text{FFN}_3 = \text{ReLU}(\begin{bmatrix} 0.75 \\ 0.75 \end{bmatrix}) = \begin{bmatrix} 0.75 \\ 0.75 \end{bmatrix}$

6. Output Layer and Prediction

The FFN output is mapped to the vocabulary size using output weights $W_{\text{out}}$ and bias $b_{\text{out}}$ :

$o = W_{\text{out}} \cdot \text{FFN}_3 + b_{\text{out}}$

Assuming:

$W_{\text{out}} = \begin{bmatrix} 1.0 & 0.0 \\ 0.0 & 1.0 \\ 1.0 & 1.0 \\ 0.0 & 0.0 \end{bmatrix}, \, b_{\text{out}} = \begin{bmatrix} 0 \\ 0 \\ 0 \\ 0 \end{bmatrix}$

$o = \begin{bmatrix} 0.75 \\ 0.75 \\ 1.5 \\ 0.0 \end{bmatrix}$

7. Apply Softmax for Probabilities

Convert logits to probabilities using softmax:

$P = \text{Softmax}(o)$

Probabilities:

$P = \begin{bmatrix} 0.209 \\ 0.209 \\ 0.569 \\ 0.014 \end{bmatrix}$

Prediction: $D$ (highest probability = 0.569).

8. Loss Calculation

Cross-entropy loss for the true label $D$ :

$\text{Loss} = -\log(P_{\text{true}}) = -\log(0.569) \approx 0.563$

9. Backpropagation

Compute gradients for all weights ( $W_Q, W_K, W_V, W_{\text{out}}$ ) and update them using gradient descent.

This process continues iteratively, with further training and optimization improving the efficiency and accuracy of the model. However, this was very simple far overview of Transformer approach for the same example can help you understand the Math Behind RNN and transformers better

Why Transformers Are considered as Neural Networks

Transformers are considered neural networks because they consist of layers of neurons (units) that perform linear transformations (e.g., matrix multiplications) followed by non-linear activations (e.g., ReLU). They also have learnable parameters (weights and biases), and the model is trained using backpropagation to minimize a loss function. The attention mechanism, though different from traditional convolutional or recurrent layers, operates within the framework of a neural network.

KEY Notes:

In this explanation, we covered how RNNs and Transformers work to predict the next letter in a sequence. While RNNs process inputs sequentially and rely on maintaining a hidden state over time, Transformers leverage self-attention to process all inputs in parallel, making them more efficient and effective for capturing long-range dependencies in data.

This makes Transformers particularly well-suited for tasks like language modeling, where understanding the relationships between words (or letters) over long distances is essential.

Key Differences Between RNNs and Transformers

Sequential vs. Parallel Processing: RNNs process inputs one at a time, maintaining a hidden state for each step, while Transformers process the entire sequence in parallel using self-attention mechanisms.
Handling Long-Range Dependencies: RNNs can struggle with capturing long-term dependencies due to vanishing gradients, whereas Transformers handle long-range dependencies efficiently through self-attention.
Training Efficiency: Transformers, due to their parallel nature, are faster to train on large datasets compared to RNNs, which must process the data sequentially.
Architectural Complexity: While RNNs are simpler in their design, Transformers are more complex due to their multi-head self-attention layers and feed-forward networks. However, the complexity allows Transformers to capture more nuanced relationships in the data.

Back To Answer Our Initial Main Question , Why Transformers Are Considered Superior For Nlp Compared To Rnn

Let’s revisit why Transformers are considered superior for NLP tasks compared to RNNs, and directly relate this to the detailed examples of RNN and Transformer architectures we previously explored.

1. Parallel Processing and Speed

In the earlier RNN explanation, we saw that RNNs process sequences one step at a time. For each letter in the sequence “A B C”, the model updates the hidden state sequentially, meaning that at each step, the model can only “see” one input at a time. This limits the model’s ability to fully utilize parallel computing, resulting in slower training times.

In contrast, the Transformer processes the entire sequence at once, thanks to its self-attention mechanism. This allows it to process all letters (“A B C”) simultaneously and compute relationships between every pair of tokens at the same time, drastically improving training speed. The self-attention mechanism also ensures that Transformers can leverage parallel computing on GPUs more efficiently. In the example, we saw how self-attention enabled the model to focus on relevant parts of the input sequence all at once, making it more scalable for large datasets compared to the sequential processing of RNNs.

2. Handling Long-Range Dependencies

When discussing RNNs, we highlighted that they can struggle with long-range dependencies due to the vanishing gradient problem. This issue arises because the gradients diminish as they are backpropagated through many time steps, making it hard for the model to update weights and learn relationships between distant words (like in long sentences or documents). In our RNN example, predicting the next letter “D” from “A B C” would become increasingly difficult if the sequence were much longer due to this vanishing gradient problem.

Transformers, on the other hand, excel at capturing long-range dependencies. Using the self-attention mechanism, Transformers can compute relationships between any two tokens, regardless of their distance in the sequence. In the Transformer example, we calculated how each word’s relevance to the others was determined through attention scores. This mechanism is not impacted by token position, which enables Transformers to model long sequences effectively, solving the core problem faced by RNNs.

3. Efficiency in Gradient Flow

In our RNN example, backpropagation through time (BPTT) was computationally expensive and prone to vanishing gradients because the gradients had to pass through many sequential hidden states. The sequential nature of RNNs makes gradient flow less efficient, especially over long sequences, which limits their ability to learn complex dependencies.

In comparison, Transformers avoid the vanishing gradient problem entirely. With their non-sequential nature, gradients can flow more easily through the network during backpropagation, allowing for smoother training over large sequences. This makes Transformers more robust for learning from complex data, as seen in the transformer’s ability to handle the entire sequence “A B C” in parallel without encountering these gradient issues.

4. Scalability and Model Size

In the RNN architecture we explored, adding more layers or neurons to handle longer sequences becomes computationally challenging. RNNs face limitations in scalability because their sequential structure prevents efficient scaling with modern GPU resources.

Transformers, however, scale much more effectively. In the Transformer example, we demonstrated how adding more layers of self-attention and feed-forward networks (as done in models like GPT-3 and BERT) allows the architecture to handle massive datasets and complex tasks without suffering from computational inefficiencies. This scalability is a key reason why Transformers have become the foundation for large language models that can be trained on vast corpora of text.

5. Self-Attention vs. Sequential Memory

In RNNs, the hidden state serves as a “memory” of previous tokens, but as we saw in the example, this memory decays over time, especially with long sequences. RNNs rely on this hidden state to retain information, which limits their ability to maintain accurate representations of earlier tokens as the sequence grows longer.

In the Transformer example, we saw that the self-attention mechanism allows the model to compare all tokens in the sequence directly, providing a more precise and dynamic representation of how each token relates to every other token. This makes Transformers far better at capturing the relationships between distant tokens, which is crucial in tasks like machine translation, text summarization, and long-form text generation.

Conclusion: Why Transformers are considered Superior for NLP

To summarize, based on both the theoretical explanations and the practical examples, Transformers are better suited for NLP tasks because:

Parallel processing allows for faster training and scalability.
Self-attention helps capture long-range dependencies more effectively.
Transformers avoid the vanishing gradient problem, making gradient flow smoother.
The architecture is inherently more scalable, which is why large models like GPT-3 and BERT are based on Transformers.

In essence, the efficiency, ability to handle long dependencies, and scalability make Transformers the go-to architecture for many NLP tasks today.

Don’t forget to check our apps! Visit here.

Why transformers are better for NLP ? Let’s see the math behind it – Day 64