Fundamentals
Glossary
Token
A chunk of text, roughly a word or part of a word, that a language model reads and generates.
Last updated
Language models do not see whole words or characters; they work with tokens. A token might be a short word, a common word ending or punctuation. "Unbelievable" could split into several tokens. Token counts matter because they determine context limits, pricing and generation speed. Providers usually bill by the number of input and output tokens.
Related terms
Tokenization
The process of splitting text into tokens before it is fed to a language model.
Context Window
The maximum amount of text, measured in tokens, that a model can consider at once.
Large Language Model
A neural network trained on massive text corpora to predict the next token, enabling it to generate and understand langu...
Keep exploring
Browse the full AI glossary or compare AI tools that use this technology.