Next-token prediction

A language model does one thing: given the text so far, it outputs a probability for every token in its vocabulary being next. A softmax step turns raw scores into probabilities using eˣ, so they are positive and add up to 1. The model picks a token (usually one of the likely ones), adds it to the text and repeats. A long answer is this loop run hundreds of times.

Why it matters: This loop is what a chatbot is doing every time it answers.

Builds on: Tokens, Probability basics, Exponents and logarithms, Layers and networks

Learn it, then prove it

Sign in (free) to take this step's mastery check and save your journey.