Abdullah Erzin
Back to the scenes

Self-study, 2026

Backpropagation and attention, from first principles

Worked through deep learning from the maths up: gradients and backpropagation by hand, RNNs written from scratch, and the attention mechanism behind BERT and GPT.

Why

At work I use trained models every day: detectors, segmentation networks, depth pipelines. I wanted to know what happens inside them, not only how to call them. So I worked through deep learning from the maths up, writing the core pieces by hand before using the library versions.

How a network learns

Training is one idea repeated: measure how wrong the output is, then move every weight a little in the direction that makes it less wrong. The gradient gives that direction, and backpropagation computes it with the chain rule, one local derivative at a time, from the loss back to each weight.

I worked these derivatives out by hand before relying on autograd. The case below is my favourite: for a sigmoid followed by cross-entropy, two messy factors cancel and the gradient becomes simply ŷ − y. The error itself is the learning signal.

Computation graph of one neuron: inputs x, w and b feed z = w·x + b, then ŷ = sigmoid of z, then the cross-entropy loss L against the label y. Forward values are shown in each box; gradients flowing backwards are shown in the accent colour under each edge.∂L/∂ŷ−6.75∂ŷ/∂z0.126∂z/∂w = x1.50x1.50w−1.50b0.50z = w·x + b−1.75ŷ = σ(z)0.148L = BCE(ŷ, y)1.910label y1Computation graph of one neuron, top to bottom: x, w and b feed z = w·x + b, then ŷ = sigmoid of z, then the loss L against the label y. Gradients flowing backwards are shown in the accent colour beside each edge.∂z/∂w = x1.50∂ŷ/∂z0.126∂L/∂ŷ−6.75x1.50w−1.50b0.50z = w·x + b−1.75ŷ = σ(z)0.148L = BCE(ŷ, y)1.910label y1

∂L/∂w = ∂L/∂ŷ · ∂ŷ/∂z · ∂z/∂w = −6.75 · 0.126 · 1.50 = −1.278

For a sigmoid with cross-entropy the first two factors cancel: ∂L/∂z = ŷ − y = −0.852, so ∂L/∂w = (ŷ − y)·x.

Loss as a function of the weight w, with the current weight marked and the tangent line whose slope is the gradient ∂L/∂w.0246−4−2024weight wLoss
One neuron and one training example, computed live in your browser. Drag the weight to see every gradient change, then take steps downhill. When the prediction is confidently wrong the gradient is large; as ŷ approaches y it shrinks towards zero.

The same picture explains vanishing gradients. In a deep network the chain rule multiplies many local derivatives, and a saturated sigmoid’s ∂ŷ/∂z is close to zero, so the product shrinks layer by layer until the early layers barely learn.

From recurrence to attention

A recurrent network reads a sentence one token at a time and carries everything it knows in one hidden state, ht = tanh(Whh·ht−1 + Wxh·xt + b). I wrote one from scratch to see this. Gradients have to travel back through every step, so they fade over long sequences; the gated cell state of an LSTM is what keeps them alive. The sequential bottleneck stays, though: the fiftieth token can reach the first only through forty-nine updates.

Attention removes that bottleneck. Every token looks at every other token directly, in a single step.

  1. Each token turns its vector into a query (what it is looking for) and a key (what it offers).
  2. The query is compared with every key by a dot product, divided by √d, and passed through a softmax. The result is one row of weights that sums to 1.
  3. The token's new vector is the weighted mix of every token's value.

“lag” attends most to “service” (0.46).

A shortened “Great food, but the service lagged.” “lagged” is split into the pieces “lag” and “##ged”, the way WordPiece splits a word it has not stored whole.

Attention matrix. Each row is a query token, each column a key token; darker cells mean more attention.Greatfoodbutservicelag##gedGreatfoodbutservicelag##ged0.170.570.070.070.070.050.570.170.070.070.070.050.180.180.120.180.270.080.070.070.100.150.510.100.060.060.090.460.130.200.040.040.060.090.660.13
Hand-set toy scores chosen to show patterns trained models tend to learn, such as an adjective attending to its noun and a suffix to its root. Not weights from a trained model. The softmax, the √d scaling and the mask are computed live in your browser.

A transformer runs several of these in parallel (multi-head attention, each head free to learn a different relation), adds positional encodings because attention on its own ignores word order, and stacks the result with feed-forward layers. In GPT-2 small, for example, 12 heads of 64 dimensions make up the 768-dimensional model, and each feed-forward layer widens it to 3,072 and back.

BERT uses the bidirectional version and learns by filling in masked words. GPT uses the causal mask and predicts the next token. Because earlier tokens never see later ones, GPT can keep the keys and values it has already computed (the KV cache) instead of recomputing them for every new token.

Along the way

The exercises were small and standard; the point was the understanding. Linear algebra and calculus first, then logistic and linear regression on Kaggle’s Titanic and House Prices, where I learned that more feature engineering can make a model worse. Then MNIST, first with a fully connected network and then with a CNN, debugging my own training loop on the way. Then a character-level LSTM, and finally fine-tuning a Turkish BERT (dbmdz/bert-base-turkish-cased) for sentiment. Its first version called “Berbat bir deneyimdi” (“It was a terrible experience”) positive; I traced that to class imbalance in the training data and fixed it by rebalancing the classes.