Tokens

Language models don't read letters or whole words; they read tokens, common chunks of text. A long word like "unbelievable" may be split into a few pieces. Each token has an ID number from a fixed vocabulary of tens of thousands. Prices, context limits and speed are all counted in tokens; very roughly, 100 tokens is about 75 English words.

Why it matters: Tokens are what a language model actually reads and writes.

Builds on: Training versus using a model

Learn it, then prove it

Sign in (free) to take this step's mastery check and save your journey.