Transformers from scratch

A language model does one thing: given some text, it predicts what comes next. It does not look up answers in a database. It produces a list of every word it knows, each with a probability, then picks one.

To write a whole answer, it adds the chosen word to the text and predicts again, over and over. Everything a chatbot does, from translating to writing code, comes from repeating that one step.

The transformer is the machine that makes those predictions good. The rest of the course opens it up.

First the text is cut into tokens: whole words, pieces of words, or punctuation marks. Each token has a fixed ID number from a vocabulary list.

Common words like "the" get one token. A rare word like "unvaccinated" is built from pieces such as "un" + "vacc" + "inated". That way a vocabulary of about 50,000 to 200,000 pieces can spell any word, even ones the model has never seen.

From here on, the model only sees a list of ID numbers.

Each token ID is swapped for a long list of numbers called an embedding (a vector). You can think of those numbers as coordinates that place the word on a map of meaning.

Nobody writes these numbers by hand. They are learned during training, and they settle so that words used in similar ways end up close together. Directions on the map come to mean something too, such as "male → female" or "country → its capital".

Real models use 768 to over 12,000 numbers per token. We'll squash it to 2 so you can see it.

The attention step you'll meet next looks at all words at once, like a bag of words. On its own, it has no idea which word came first.

So before attention, each embedding gets a positional encoding added to it: another list of numbers that is different for position 1, position 2, position 3, and so on. The original transformer made these from sine waves of different speeds, so each position gets a unique pattern and neighbouring positions get similar patterns.

After this step, each vector carries two things: what the word is and where it sits.

This is the idea that made transformers famous (the 2017 paper was titled Attention Is All You Need). For every word, the model asks: which other words in this text matter for understanding me?

Each word gives every other word an attention weight, a share of its focus that adds up to 100%. Then it updates its own vector by blending in information from the words it focuses on. After attention, the vector for "she" carries information from "patient", so it now means something closer to "the sick patient".

Older models read word by word and forgot the start of long sentences. Attention lets any word look directly at any other word, however far away.

Each token's vector is turned into three new vectors by multiplying it with three learned tables of numbers:

• Query: what am I looking for? ("she" is looking for a female person who could be sick.)
• Key: what do I offer? (a label each word holds up: "patient" says "I'm a person, I'm sick".)
• Value: what I pass on if you pick me (the actual information).

The score between two words is the dot product of one word's query with the other's key: multiply matching numbers and add them up. Matching directions give a big score. Scores are scaled down, turned into percentages with softmax, and used to blend the values.

Multi-head attention. A transformer runs several attention steps side by side, each with its own query, key and value tables. These are called heads. During training each head drifts toward tracking a different kind of link: one follows grammar, one links pronouns to names, one simply looks at the previous word.

The block. Attention is wrapped with a few helpers into a transformer block. A feed-forward network lets each token "think" on its own about what it just gathered; much of the model's stored knowledge lives here. Residual connections add each step's input back to its output so nothing is lost, and normalisation keeps the numbers in a steady range.

The stack. Blocks are stacked: GPT-2 small has 12, GPT-3 has 96. Early layers pick up simple patterns, later layers more abstract meaning.

Training is a guessing game played trillions of times. Take real text, hide the next token, let the model predict, then measure how surprised it was by the true answer. That surprise is the loss: if the model gave the right word a high probability, the loss is small.

Then every number in the model (its parameters or weights, 175 billion of them in GPT-3) is nudged a tiny bit in the direction that would have lowered the loss. This is gradient descent. The size of each nudge is the learning rate.

This first stage, called pre-training, gives a model that can continue any text. It is then fine-tuned on examples of good conversations and on human feedback to become a helpful assistant.

After the last block, the vector of the final token is turned into a probability for every token in the vocabulary (Module 1). One is picked, added to the text, and the whole thing runs again. This loop is called autoregressive generation.

Chat models use a causal mask: each token may only attend to tokens before it, never after. That matches the job of predicting the future from the past.

The model can attend to everything inside its context window (from a few thousand to over a million tokens today). It is trained to produce plausible text, not to check facts, which is why it can state wrong things confidently. People call this hallucination; always verify important claims.