How LLMs Work: How AI Generates Text, One Token at a Time

Okay so you type something into ChatGPT or Claude, you hit send, and letters start showing up on your screen one by one. It feels kind of magical. But there is actually a pretty simple process happening underneath.

So how do LLMs work, really? How does an LLM like ChatGPT actually generate text? Let me show you what’s really going on. No heavy math, no PhD level stuff. Just the basic idea, broken into five steps, with some interactive demos so you can see it happen.

The short version: a large language model turns your text into tokens, predicts the next token over and over, and streams the result back to you one piece at a time. Now let’s walk through each step.

Step 1: Your text becomes tokens (tokenization)

The model does not see letters or words the way you do. Your text first gets cut up into little pieces called tokens.

A token can be a whole word, part of a word, or just punctuation. For common English, one token is usually about 3 to 4 characters. So “hello” is one token. But a big rare word like “antidisestablishmentarianism” gets chopped into several tokens.

Think of it like lego blocks. You cannot build a castle by handing someone a photo of a castle. You have to break the idea into blocks first. The model only works with blocks.

Try typing in the demo. Watch your text get cut into chunks. Each chunk also has a number. The model does not actually see the letters “cat” or “hello”. It only sees the ID number of that token. Everything inside the model is just numbers.

If you want to see how a real tokenizer splits your text, OpenAI has a nice interactive tool at platform.openai.com/tokenizer. Paste anything in and see the actual tokens and IDs.

Step 2: The model predicts the next token

Okay now the model has your tokens. What does it do with them?

This part surprises people. The model only does ONE thing. It looks at all the tokens so far and predicts what token should come next.

That’s it. That is the whole trick, and it has a name: next token prediction. An LLM is basically a really smart autocomplete.

Here is a simple way to think about it. Give your friend a sentence starting with “The cat sat on the” and ask them to guess the next word. They will probably say “mat” or “floor” or “bed”. Your friend just used their whole life of reading books and stuff to guess what comes next.

The model does the same thing but it read pretty much all of the internet.

Click next to generate one token at a time. The bars show what the model thinks each candidate token has a probability of being chosen. The highest bar is the most likely one, but the model does not always pick that one.

Step 3: How the model picks one token (sampling)

Good question. When the model says “mat is 42 percent, floor is 18 percent, bed is 12 percent…” it still has to pick ONE actual word to put next. There are a few ways to do that.

Greedy: always pick the most likely one. Sounds smart but it makes the output super boring and repetitive. If you always pick the safest word, you get very dull writing that sounds robotic.

Weighted random (sampling): treat the probabilities like a weighted dice roll. “mat” has 42 percent chance, “floor” has 18 percent chance, and so on. Roll the dice. Whichever word wins, you pick that one.

Most modern models use weighted sampling with some extra rules. They usually cut off the tail of very unlikely words so you don’t get weird random stuff showing up. This is called top-k (keep only the top 10 candidates) or top-p (keep enough candidates to cover 90 percent of the probability).

Step 4: Temperature, the LLM creativity knob

There is one more important thing. Temperature.

Temperature is the LLM setting that controls how much the model follows its own probabilities. It is a single number, usually between 0 and 2, and it is the main knob you get for tuning how creative or predictable the output is.

At a low temperature like 0.1, the model basically always picks the top choice. Very predictable, very safe, very boring.

At a high temperature like 1.5, the model treats all the choices as almost equally likely. You get wild creative outputs but also a lot of weird nonsense.

Here is one way to think about it. Imagine a room full of people voting on what word comes next. At low temperature, the loudest most confident person always wins. At high temperature, everyone is shouting over each other and the vote is basically random.

Drag the temperature slider. Watch the probability bars change. Low temperature makes one bar huge and the others tiny. High temperature makes them all look similar.

Step 5: The loop that generates the whole text

So now you know the basic move. Predict the next token, pick one, done. But that is just ONE token. How does the model write a whole paragraph?

It just does this over and over.

1. You give it your prompt as tokens.
2. Model predicts the next token. Picks one.
3. That new token gets added to the input.
4. Model predicts the next next token based on your prompt plus the new token.
5. Pick one. Add it. Repeat.

It keeps going until it hits a stop condition. Usually that means it generates a special “end of text” token, or you hit a max length limit.

This is why you see text streaming to you one word at a time. The model is literally generating one token at a time, and the app is showing you each one as soon as it pops out.

So, how do LLMs work? That’s basically it

LLMs are just very good next-token predictors. They take your text, chop it into tokens, look at the whole thing, guess what comes next, pick one, add it, and do it all again.

Everything else, all the fancy abilities like writing code or answering questions, comes from the fact that a model big enough and trained on enough text becomes really really good at “what comes next”. Writing working code becomes a matter of predicting the tokens that make the code actually work. Answering a question is predicting the tokens of a good answer.

Once you understand this, a lot of weird LLM behaviors start to make sense. Why it hallucinates (it is just guessing the most likely next token, not fact checking). Why it struggles with counting letters (letters are not tokens, whole chunks are). Why prompts matter so much (your prompt is literally the starting context it is predicting from).

Next time you use one, remember. It is just a very good autocomplete that read the whole internet. That is the magic.

Frequently asked questions

How does ChatGPT work?

ChatGPT is a large language model, so it works by predicting one token at a time. It turns your prompt into tokens, guesses the most likely next token, adds it to the sequence, and repeats until it hits a stop token. The text you see streaming in is that loop running over and over.

What are tokens in AI?

Tokens are the small pieces of text an AI model actually reads. A token can be a whole word, part of a word, or punctuation. For English, one token is roughly 3 to 4 characters. The model never sees your letters directly, it only sees the ID numbers of these tokens.

What is tokenization?

Tokenization is the step that chops your text into tokens before the model does anything with it. It is always the first thing that happens. You can watch it live in the demo above or in OpenAI’s tokenizer.

What is next token prediction?

Next token prediction is the single job an LLM does: look at all the tokens so far and predict what token most likely comes next. Every ability a model has, from writing code to answering questions, comes out of doing this one thing extremely well.

What does temperature do in an LLM?

Temperature controls how much randomness goes into picking the next token. A low temperature (like 0.1) makes the model almost always pick its top choice, so the output is safe and predictable. A high temperature (like 1.5) flattens the odds so the model takes more risks, so you get more creative but less reliable text.

Why do LLMs hallucinate?

Because an LLM is guessing the most likely next token, not checking facts. If a confident-sounding but wrong sequence of tokens is the statistically likely continuation, the model will happily produce it. There is no built-in truth check, only probability.

Sources

• OpenAI Tokenizer. Paste any text and see exactly how GPT models chop it into tokens.
• “Let’s build the GPT Tokenizer” by Andrej Karpathy on YouTube. Two-hour deep dive that builds a BPE tokenizer from scratch. Best video on the internet for understanding tokens.
• Anthropic Messages API. Where temperature, top_p, and top_k actually live in the request payload.