Attention and transformers

Attention lets each token look at the other tokens and decide which ones matter to it. Each token makes a query vector and offers a key vector; the dot product of a query with a key says how relevant they are, softmax turns those scores into weights, and the token takes a weighted mix of the others' value vectors. A transformer stacks many attention and network layers. It is the T in GPT.

Why it matters: Attention is the breakthrough that made today's language models work.

Builds on: Embeddings, Matrices and matrix multiplication, Layers and networks

Learn it, then prove it

Sign in (free) to take this step's mastery check and save your journey.