How LLMs Actually Work: Tokens, Attention and Next-Word Prediction

Ask a large language model why it writes so fluently and it cannot tell you — it has no idea. It is not reading your question and thinking. It is running an enormous statistical machine that predicts the next word, over and over, until it reaches a stopping point.

Understanding that single fact reframes everything. It explains why LLMs hallucinate, why they are so good at rephrasing, and why you can get them to do remarkable things with a well-written prompt.

From words to tokens

Models do not read letters or words directly. They read tokens — chunks of text that are usually part of a word. The word "unbelievable" might be split into "un", "believ", "able". Tokens let the model handle a fixed vocabulary of about 32,000 to 200,000 pieces instead of an infinite set of words.

Every token has a number associated with it. The model converts those numbers into vectors, which are just long lists of numbers that encode meaning. Words that are similar — "cat" and "kitten" — end up with vectors that point in similar directions.

The transformer and attention

Modern LLMs use a neural network architecture called the transformer. Its defining trick is the attention mechanism, which decides how much each token should "look at" every other token in the text.

When the model processes "The cat sat on the …", attention lets the model weigh which previous tokens matter most for predicting the next one. The word "cat" should strongly influence what comes next; "The" matters far less. The model learns these relationships from trillions of examples during training.

Training: predict the next token, billions of times

Training a large language model is conceptually simple and practically enormous. You feed it a huge amount of text, one chunk at a time. For each chunk, the model tries to predict the next token. When it guesses wrong, you adjust its billions of internal parameters slightly, in the direction that would have made the guess right.

Repeat that for months on thousands of GPUs and the model develops a deep statistical model of language — not rules, but patterns. That is why it can write in styles it never explicitly memorised: it learned the distribution of text, not a copy of it.

Inference: one prediction at a time

When you type a question, the model generates the most likely next token given everything so far — your question and every token it has already produced. Then it does it again, and again, each time feeding its own previous output back in.

This is why output speed feels slower than the model's "brains": it cannot skip ahead. It must produce one token at a time, and the whole response length is limited by how many times it repeats the loop.

Why temperature and sampling matter

If the model always picked the single most likely token, its writing would be flat and repetitive. So inference adds randomness via temperature. A low temperature produces predictable, focused output. A high temperature produces more creative, varied output — and more mistakes.

Temperature is a dial on creativity versus reliability. For code and data analysis, keep it low. For brainstorming, raise it.

Where LLMs fall short

• Hallucination: the model is optimised to produce plausible text, not true text. It has no ground truth to check against.
• Recency limits: knowledge stops at training data unless retrieval or tools are added.
• No persistent memory: each conversation is effectively a fresh start unless you supply context.
• No real reasoning: what looks like reasoning is pattern completion at enormous scale.

Putting it together

Once you internalise that an LLM is a next-token prediction engine, every practical skill follows. Prompt engineering is about giving the predictor better context. RAG is about supplying facts the model never memorised. Fine-tuning is about reshaping the probabilities toward your style. Tools and agents are about letting the model act on its predictions instead of just describing them.

None of this makes LLMs less impressive. A statistical machine that can hold a conversation, write a draft, and debug a function is remarkable. But knowing how it works lets you use it precisely instead of hoping.