The Vanilla RNN Problem
ht = tanh(Wh ht-1 + Wx xt)
Each new hidden state overwrites the old one. Over long sequences, gradients either explode or vanish.
test_html/RNN/LSTM_Gates_Explained.html
LSTM Gates — Interactive Notes
Created: 2026年5月16日 Modified: 2026年5月16日
LSTM Gates Explained
An RNN with a protected memory lane.
Long Short-Term Memory networks solve the vanishing gradient problem by introducing a cell state (conveyor belt) and three gating mechanisms that carefully add or remove information.
The Conveyor Belt
Imagine an assembly line (the cell state) running straight through. At each station, gates decide:
- Forget Gate: drop old products?
- Input Gate: add new ones?
- Output Gate: what to expose?
Forget Gate
ft = σ(Wf · [ht-1, xt] + bf)
Discards information from the cell state when a new context starts.
Input + Output Gates
it = σ(Wi · [ht-1, xt] + bi)
ot = σ(Wo · [ht-1, xt] + bo)
Interactive Demo: The Conveyor Belt
Thecatsatonthemat
h0 h1 h2 h3 h4 h5
Cell State (conveyor) Hidden State (output)
Retrieval-Augmented Generation
Giving LLMs an open-book exam instead of a closed-book memory test.
RAG augments a language model by fetching relevant external documents from a knowledge base at inference time, grounding generation in facts.
The Exam Analogy
| Paradigm | Analogy | Hallucination |
|---|---|---|
| Plain LLM | Closed-book | High |
| Fine-tuning | Studying | Medium |
| RAG | Open-book | Low |
Step 1: Indexing
Documents are chunked and embedded into a high-dimensional vector space.
vdoc = fenc(chunk) ∈ ℝd
Step 2: Retrieval
score(q, d) = (vq · vd) / (‖vq‖ ‖vd‖)
Return the top-k chunks using approximate search like HNSW for billion-scale retrieval.
Step 3: Generation
Use the following context to answer.
If the answer is not in the context, say "I don't know".
Context: {retrieved chunks}
Question: {query}
Answer: Interactive Demo: Vector Search
Convolution Operators
Sliding a flashlight over an image to find patterns.
Convolution is a local, translation-equivariant operator that learns spatial hierarchies by sliding learnable filters across an input grid.
The Flashlight Metaphor
Imagine shining a small flashlight (the kernel) over a photograph, one patch at a time. The collection becomes a feature map.
Discrete 2D Convolution
(I * K)(i, j) = Σm Σn I(i+m, j+n) · K(m, n)
Where I = input image, K = kernel weights.
Classic Hand-Crafted Kernels
| Kernel | Effect |
|---|---|
| Sobel X | Horizontal edges |
| Gaussian | Smoothing |
| Sharpen | Enhance detail |
Padding & Stride
O = ⌊(I − K + 2P) / S⌋ + 1
Padding adds a zero border. Stride controls how many pixels the flashlight jumps.
Interactive Demo: Kernel Slideshow
Input (7×7)
Kernel (Sobel X)
-101 -202 -101
Position: (0, 0)
Output (5×5)
Seq2Seq & Cross-Attention
One RNN reads; another writes. Attention lets them whisper to each other.
Sequence-to-sequence models map variable-length inputs to variable-length outputs using an encoder-decoder pair. Attention liberates the decoder from a single fixed-size context vector.
The Encoder-Decoder Bottleneck
A classic Seq2Seq compresses the entire input into one context vector c.
Fitting a 50-word paragraph into 256 dims is like fitting a novel into a tweet.
Attention to the Rescue
ct = Σj=1T αtj · hjenc
The decoder gets a different context vector at every step.
Alignment Weights
αtj = exp(etj) / Σk exp(etk)
where etj = score(st-1, hj).
From Seq2Seq to Transformers
Seq2Seq attention is cross-attention: queries from decoder, keys/values from encoder. This exact pattern survives in Transformers today.
Interactive Demo: Alignment Heatmap
Source (EN)
Iloveyouverymuch
Target (FR)
Je t'aime beaucoup