Understanding Recurrent Neural Networks and LSTM Cells

The Need for Sequence Persistence

Humans do not reset their cognitive state from scratch with every passing second. As you read this sentence, you understand each incoming word based on the context established by the words that preceded it. You do not discard your working memory after every punctuation mark and start your comprehension from zero. Your thoughts possess persistence.

Traditional feed forward neural networks cannot maintain persistence. In a conventional multilayer perceptron or convolutional network, each input sample is processed entirely independently of all previous samples:

$y = \sigma(W x + b)$

This architecture works well for static tasks such as classifying isolated photographs, but it represents a severe shortcoming when analyzing sequential information. Consider an artificial intelligence system trying to predict the next word in a conversational stream, transcribe audio phonemes from continuous speech, or summarize a paragraph of text. To make an accurate prediction at time step $t$, the network must retain context from time steps $t-1, t-2$, and earlier.

Recurrent Neural Networks (commonly abbreviated as RNNs) resolve this challenge. They are neural networks equipped with feedback loops, allowing information to cycle through internal states and persist across sequential transitions.

In the diagram above, a modular block of neural network, labeled $A$, receives an input vector $x_t$ at step $t$ and emits an output activation vector $h_t$. The feedback loop allows internal state signals to pass from one processing cycle to the next.

Unrolling Recurrent Neural Networks Through Time

These recurrent feedback loops can initially appear mysterious. However, unrolling the network through time reveals that a recurrent network is simply multiple copies of the exact same neural network layer chained together, with each copy passing its hidden state to its successor:

At any individual discrete time step $t$, the standard Elman recurrent network computes its updated hidden state $h_t$ and emission prediction $y_t$ through two matrix transformations:

$h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$

$y_t = \text{softmax}(W_{hy} h_t + b_y)$

Here:

• $x_t \in \mathbb{R}^d$ represents the input vector at time step $t$.
• $h_{t-1} \in \mathbb{R}^h$ represents the hidden state vector from the previous time step.
• $W_{xh} \in \mathbb{R}^{h \times d}$ is the parameter weight matrix mapping inputs to hidden states.
• $W_{hh} \in \mathbb{R}^{h \times h}$ is the shared recurrent parameter weight matrix mapping prior hidden activations to new hidden activations.
• $b_h \in \mathbb{R}^h$ is the hidden state bias vector.
• $\tanh$ is the hyperbolic tangent activation function, squashing values into the bounded range $[-1, 1]$.

This unrolled chain structure reveals that recurrent neural networks are fundamentally designed for sequences and ordered lists. They formed the dominant architecture for machine translation, language modeling, and voice recognition for over two decades.

However, standard recurrent networks suffer from a fundamental mathematical flaw that crippled their ability to solve complex, long sequence tasks.

The Dilemma of Long Distance Dependencies

The primary promise of recurrent networks is that they can connect earlier context to a downstream decision. If an autonomous agent reads a continuous text, previous paragraphs should inform current interpretations. Whether a basic RNN can accomplish this depends entirely on the distance between the context and the prediction.

Sometimes, only immediate recent context is required. For example, consider a language model predicting the final word in the phrase:

“The clouds gather in the sky.”

The relevant context word “clouds” sits only four tokens before the blank. Because the distance is minimal, standard recurrent loops easily propagate the contextual signal forward.

Now consider a case requiring distant contextual memory:

“I spent my early childhood in France… After twenty five years of living overseas, I can still converse fluently in French.”

Recent tokens such as “converse fluently in” suggest that the missing word is the name of a language. But to determine which language, the network must recall the location “France” mentioned dozens of words earlier.

As the temporal distance between the relevant evidence and the target prediction increases, standard recurrent neural networks become mathematically incapable of connecting the information.

The Mathematics of Vanishing and Exploding Gradients

To understand why standard RNNs fail over long distances, we examine how gradients flow backward during training. In Backpropagation Through Time (abbreviated as BPTT), the total loss $L$ is calculated across sequence steps, and gradients are propagated backward from step $T$ to earlier step $k$:

$\frac{\partial L}{\partial h_k} = \frac{\partial L}{\partial h_T} \prod_{j=k+1}^T \frac{\partial h_j}{\partial h_{j-1}}$

Let us compute the Jacobian matrix $\frac{\partial h_j}{\partial h_{j-1}}$ representing the rate of change of state $h_j$ with respect to state $h_{j-1}$:

$\frac{\partial h_j}{\partial h_{j-1}} = \text{diag}\left(1 - \tanh^2(W_{hh} h_{j-1} + W_{xh} x_j + b_h)\right) W_{hh}^T$

Notice what occurs across the chain rule product:

$\prod_{j=k+1}^T \frac{\partial h_j}{\partial h_{j-1}} = \prod_{j=k+1}^T \text{diag}(1 - h_j^2) W_{hh}^T$

This expression contains two critical terms:

1. The derivative of the hyperbolic tangent function $1 - \tanh^2(z)$, which achieves a maximum value of $1.0$ at $z=0$ and decays rapidly toward $0.0$ as activations saturate.
2. The recurrent weight matrix $W_{hh}^T$ multiplied repeatedly by itself $T - k$ times.

If the largest eigenvalue of the recurrent weight matrix is less than $1.0$ (or if activations saturate), the continuous matrix multiplication causes gradient signals to decay exponentially toward zero as they travel backward. Over a modest sequence length of 50 steps, a gradient factor of $0.8$ shrinks to:

$0.8^{50} \approx 0.000014$

The error signal vanishes. The weights in early layers receive virtually zero gradient updates, rendering the network blind to long range historical dependencies.

Conversely, if the eigenvalues exceed $1.0$, the gradient values explode exponentially, causing numerical instability, arithmetic overflow, and divergence during gradient descent updates.

The Core Architecture of Long Short Term Memory

Long Short Term Memory networks (commonly known as LSTMs) were invented in 1997 by Sepp Hochreiter and Jürgen Schmidhuber specifically to resolve the vanishing gradient problem.

While a standard recurrent neural network contains a simple repeating module with a single hyperbolic tangent layer, an LSTM cell features an intricate internal control structure containing four separate neural network layers that interact in a carefully choreographed manner.

To understand how vectors flow through these interacting layers, let us review the diagram notation guide:

The Constant Error Carousel and Cell State

The central breakthrough of the LSTM is the cell state, denoted as $C_t$.

The cell state runs straight along the top of the unrolled chain, functioning like a physical conveyor belt or memory bus. Information can travel along this linear channel with minimal multiplicative interference.

While a standard RNN continuously multiplies hidden states by the weight matrix $W_{hh}$, the LSTM updates its cell state through linear additions and subtractions. In calculus, the derivative of a linear addition is identity:

$\frac{\partial (A + B)}{\partial A} = 1.0$

By preserving a linear pathway where error signals can flow backward through time without continuous matrix decay, Hochreiter and Schmidhuber created the Constant Error Carousel (CEC), allowing neural networks to maintain memories across hundreds or thousands of steps.

The Role of Gating Layers

To regulate what information enters, remains in, or exits the cell state, LSTMs utilize specialized mechanisms called gates.

A gate consists of an artificial neural network layer with a sigmoid activation function followed by a pointwise elementwise multiplication operation:

$\sigma(z) = \frac{1}{1 + e^{-z}}$

The sigmoid function maps any real numbered input into a continuous range between zero and one:

• An activation value of 0 represents a closed gate: “let nothing pass through”.
• An activation value of 1 represents an open gate: “let everything pass through”.

An LSTM contains three fundamental gates:

1. The Forget Gate ($f_t$): Decides what information to purge from the cell state.
2. The Input Gate ($i_t$): Decides what incoming information to write into the cell state.
3. The Output Gate ($o_t$): Decides what filtered version of the cell state to emit as the hidden state $h_t$.

Step by Step Mathematical Derivation of LSTM Gating

Let us walk through the four mathematical operations executed within an LSTM cell during time step $t$.

Step 1: The Forget Gate Layer

The first decision in the LSTM cell is to identify which information from historical memory should be discarded. This choice is calculated by the forget gate layer:

$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$

In this equation:

• $[h_{t-1}, x_t]$ denotes the concatenation of the previous hidden state vector $h_{t-1} \in \mathbb{R}^h$ and the current input vector $x_t \in \mathbb{R}^d$, producing a combined vector in $\mathbb{R}^{h + d}$.
• $W_f \in \mathbb{R}^{h \times (h + d)}$ is the learned forget gate weight matrix.
• $b_f \in \mathbb{R}^h$ is the forget gate bias vector.
• $\sigma$ is the sigmoid activation applied element by element.

The resulting vector $f_t \in (0, 1)^h$ outputs a value between 0 and 1 for each coordinate in the cell state $C_{t-1}$. For example, in a language model tracking the grammatical gender of a subject, when a new subject noun appears, the forget gate can emit near zero values to purge the gender of the former subject.

Step 2: The Input Gate and Candidate Memory

The second phase determines what new information should be stored in the cell state. This process consists of two coordinated sub steps:

First, the input gate layer applies a sigmoid activation to decide which memory dimensions should be updated:

$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$

Second, a candidate state layer applies a hyperbolic tangent activation to synthesize a vector of new candidate values, denoted as $\tilde{C}_t$, that could be added to the cell state:

$\tilde{C}_t = \tanh(W_c \cdot [h_{t-1}, x_t] + b_c)$

Here:

• $W_i \in \mathbb{R}^{h \times (h + d)}$ and $b_i \in \mathbb{R}^h$ parameterize the input gate.
• $W_c \in \mathbb{R}^{h \times (h + d)}$ and $b_c \in \mathbb{R}^h$ parameterize the candidate memory layer.
• The $\tanh$ function bounds candidate values to $[-1, 1]$, preventing unchecked numerical growth.

Together, $i_t$ and $\tilde{C}_t$ determine both what to write and how strongly to write it.

Step 3: Updating the Cell State

We now combine the forget gate decisions and candidate updates to compute the new cell state $C_t$:

$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$

Where $\odot$ represents the Hadamard product (element by element pointwise vector multiplication).

Let us break down the physical meaning of this equation:

1. $f_t \odot C_{t-1}$: We scale the old memory state by our forget factors. Any dimension where $f_t \approx 0$ is cleared; any dimension where $f_t \approx 1$ is retained intact.
2. $i_t \odot \tilde{C}_t$: We scale our new candidate values by our input gate factors, determining how much fresh evidence to inject.
3. The $+$ operator: We combine the preserved old state with the new filtered input through simple vector addition.

This additive update is what prevents gradients from vanishing. During backpropagation, the derivative of $C_t$ with respect to $C_{t-1}$ contains the additive term $f_t$:

$\frac{\partial C_t}{\partial C_{t-1}} = f_t + \dots$

As long as the network learns to keep the forget gate open ($f_t \approx 1$), gradient error signals can backpropagate indefinitely through the sequence without exponential decay.

Step 4: The Output Gate and Hidden State

The final phase produces the outgoing hidden state $h_t$, which serves two purposes: it is passed to time step $t+1$ as recurrent context, and it is passed to downstream prediction layers (such as a classification head or vocabulary projection).

The output is based on the updated cell state, but passes through an active filter:

First, the output gate layer runs a sigmoid function to decide which parts of the cell state to expose:

$o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$

Second, we pass the cell state $C_t$ through a hyperbolic tangent function to squash values into the range $[-1, 1]$, and multiply it elementwise by the output gate:

$h_t = o_t \odot \tanh(C_t)$

Here:

• $W_o \in \mathbb{R}^{h \times (h + d)}$ and $b_o \in \mathbb{R}^h$ are the output gate parameters.
• $h_t \in \mathbb{R}^h$ represents the visible hidden state emitted at step $t$.

For example, if the cell state remembers that the current grammatical subject is plural, the output gate can expose this property when encountering a verb position so the language model correctly predicts a plural verb form.

Architectural Variants of the Standard LSTM

While the four step formulation described above represents the standard LSTM architecture (often credited to Graves and Schmidhuber), researchers developed several influential architectural variations.

Variant 1: Peephole Connections

Introduced by Felix Gers and Jürgen Schmidhuber in 2000, peephole connections allow the gate layers to inspect the internal cell state $C_{t-1}$ directly, rather than relying exclusively on the external hidden state $h_{t-1}$:

$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + W_{cf} \odot C_{t-1} + b_f)$

$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + W_{ci} \odot C_{t-1} + b_i)$

$o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + W_{co} \odot C_t + b_o)$

Notice that the forget and input gates inspect the previous cell state $C_{t-1}$, while the output gate inspects the newly updated cell state $C_t$. Peephole connections allow the network to time intervals and measure sequence durations with higher precision.

Variant 2: Coupled Forget and Input Gates

Another common variation ties the forget decision and input decision together into a single choice. Instead of computing independent gates $f_t$ and $i_t$, the network only decides what to forget, and automatically fills that freed capacity with new input:

$C_t = f_t \odot C_{t-1} + (1 - f_t) \odot \tilde{C}_t$

Here, the factor $1 - f_t$ replaces $i_t$. The network only writes new information to a dimension when it is actively forgetting older information.

Variant 3: The Gated Recurrent Unit (GRU)

Perhaps the most prominent variation is the Gated Recurrent Unit (GRU), introduced by Kyunghyun Cho and colleagues in 2014. The GRU simplifies the LSTM cell by eliminating the separate cell state $C_t$, merging all memory into a single hidden state $h_t$:

The GRU uses two gates:

1. The Reset Gate ($r_t$): Controls how much previous memory to disregard when computing new candidate states: $r_t = \sigma(W_r \cdot [h_{t-1}, x_t] + b_r)$
2. The Update Gate ($z_t$): Combines the functions of the forget and input gates: $z_t = \sigma(W_z \cdot [h_{t-1}, x_t] + b_z)$

The candidate hidden state is computed by applying the reset gate to the prior hidden state:

$\tilde{h}_t = \tanh(W \cdot [r_t \odot h_{t-1}, x_t] + b)$

The final hidden state interpolation is:

$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$

Because the GRU features fewer parameter matrices and requires fewer tensor multiplications per step, it computes faster and requires less memory while achieving competitive performance on many sequential modeling benchmarks.

Summary Comparison of Recurrent Architectures

From Recurrence to Self Attention: The Evolution to Transformers

For more than a decade, Long Short Term Memory networks powered state of the art sequence applications, from Google Translate to Apple Siri and voice dictation.

However, as dataset sizes surged from millions to billions of tokens, recurrent architectures collided with two fundamental hardware and algorithmic constraints:

1. Sequential Execution Bottleneck

To compute step $t$, an LSTM strictly requires the output of step $t-1$:

$h_t = \text{LSTM}(h_{t-1}, x_t)$

Because step $t$ depends on step $t-1$, recurrent networks cannot parallelize training across sequence positions. Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs) achieve their extraordinary computational throughput by running thousands of matrix multiply operations simultaneously across large parallel cores. When training an RNN, these cores must wait sequentially for each token transition, creating severe training latency.

2. Maximum Signal Path Length

In any recurrent architecture, information from token $1$ must travel through $T - 1$ physical network steps to reach token $T$. Even with constant error carousels, memory compression remains lossy; squeezing an entire sequence into a fixed dimension hidden vector inevitably creates an information bottleneck.

In 2017, Ashish Vaswani and colleagues published the landmark paper “Attention Is All You Need”, introducing the Transformer architecture. The Transformer completely dismantled recurrence:

• Instead of processing sequentially in $O(T)$ operations, the Transformer calculates pairwise relationships across all tokens simultaneously using Self Attention.
• The path length between any two arbitrary tokens in a sequence drops from $O(T)$ in an RNN to $O(1)$ in a Transformer layer.
• Training is fully parallelized across all sequence tokens, allowing modern models to train across trillions of words.

Understanding recurrent neural networks and LSTM gating equations illuminates why transformers were designed as they were. The core quest of deep learning sequence modeling has always been the same: preserving long distance contextual relationships without losing gradients or hitting hardware limits.

To explore how weights, bias terms, and backpropagation gradients behave in real time, launch our interactive laboratory visualizers below.