How LLMs Generate Text: A Builder's Mental Model

You don't need to understand transformer math to build good AI features. You do need a working model of what happens between sending a request and getting text back. Without one, every odd behavior looks random: the classifier that's right 49 times and wrong once, the summary that invents an order number, the bot that "forgets" a rule you clearly wrote down.

This lesson gives you that mental model in builder terms. You'll learn what a model actually computes, why it writes one token at a time, why the same request can give different answers, where its knowledge comes from, and why chat models follow instructions at all. Each idea comes with the concrete design decision it drives in the features you'll build for the rest of this course.

A model is a function from context to the next token#

Strip away the chat interface and a language model does one thing: given a sequence of tokens (chunks of text, usually a word or part of a word), it produces a probability for every token that could come next. Everything else is built on that single step.

Three consequences follow, and they shape almost every feature you'll write.

The model only knows what's in the request. It has no memory of your previous call, no access to your database, and no idea who the user is unless you put that in the input. When a feature feels dumb, the first question to ask is: what did the model actually see? You'll build this skill properly in Context Assembly.

Conversations are replayed, not remembered. A chat feature that seems to remember the last ten messages works because your code sends all ten messages again on every turn. That's why long conversations get slower and more expensive, and why managing conversation history is real engineering work.

Everything in the input is just more tokens. Your system prompt, the user's message, a pasted email, a retrieved help article: by the time they reach the model, they're one long sequence. The model has learned conventions that separate instructions from data, but there is no hard wall between them. Keep that in mind whenever untrusted text goes into a prompt.

Generation is a loop, one token at a time#

To produce a full response, the model runs that single step in a loop. It predicts a distribution over the next token, one token is chosen, that token is appended to the input, and the whole thing runs again. This is autoregressive generation: each new token is conditioned on everything before it, including the model's own output so far.

The loop explains several things you'll see in production.

The model can't go back#

Once a token is written, it's part of the input for everything that follows. A model can't revise its first sentence after writing the third. If it commits early to a wrong answer, it will often spend the rest of the response justifying it.

That's why order matters in structured output. Compare two shapes for a Deskmate triage result:

{
  "priority": "urgent",
  "reason": "The customer mentions their whole team is blocked..."
}

{
  "evidence": "Login fails for every user on a Team plan account after a password reset; customer says the whole team is blocked.",
  "priority": "urgent"
}

Current models also think privately before answering (more on that below), which reduces how much this matters, but putting supporting fields before conclusions costs nothing. Designing Output Schemas covers field order in depth.

Output length drives latency#

Reading your input happens in parallel and is fast. Writing the output happens one token after another, so it's sequential and slow by comparison. For most features, the number of output tokens is the biggest factor in how long a response takes.

Two design moves fall straight out of this. Ask for the shortest output that does the job: a triage label, not a paragraph explaining the label, unless someone will read the paragraph. And when output has to be long, stream it so users see the first words immediately. You'll do that in Streaming End to End.

Probabilities, not lookups: why answers vary#

At each step, the model produces probabilities, not a single answer. For a ticket that says "my whole team is blocked," the next-token distribution after "priority": " might put most of the weight on urgent, some on high, and a little on normal. The serving system then samples a token from that distribution, which means a less likely token sometimes wins.

Usually the top choice dominates and you get the same answer every run. On genuinely borderline inputs the distribution is split, and you'll see the answer flip between runs. Older APIs exposed a temperature setting to sharpen or flatten the distribution; current Claude models don't accept it at all, for reasons covered in Output Controls.

What it means for you as a builder:

• Don't snapshot-test exact strings. A test that asserts the summary equals a stored paragraph will fail randomly. Test properties instead: the label is one of the allowed values, every order number in the output appears in the input, the reply mentions the customer's actual question. Module 9 builds this out.
• Variation is a signal. If the same input produces different answers across runs, the input is probably ambiguous, or your instructions are. That's useful information for both your prompt and your product.
• Store outputs. A user regenerating a summary and getting a new one is fine. Your backend silently overwriting a label someone already acted on is a bug.

Thinking Like an AI Builder promised you'd run this experiment. Here's a quick way to see variation for yourself. This script sends the same borderline ticket to the model ten times and counts the answers:

import Anthropic from "@anthropic-ai/sdk";
 
const anthropic = new Anthropic();
 
const ticket = `Subject: Exports are slow today
CSV exports that usually take a few seconds are taking about two minutes.
Not urgent for me, but our finance team runs month-end reports tomorrow.`;
 
async function triageOnce(): Promise<string> {
  const response = await anthropic.messages.create({
    model: "claude-haiku-5-5",
    max_tokens: 1000,
    output_config: { effort: "low" },
    system: "You triage support tickets. Reply with exactly one word: low, normal, or urgent.",
    messages: [{ role: "user", content: ticket }],
  });
  const block = response.content.find((b): b is Anthropic.TextBlock => b.type === "text");
  return block ? block.text.trim().toLowerCase() : `(no text: ${response.stop_reason})`;
}
 
const answers = await Promise.all(Array.from({ length: 10 }, () => triageOnce()));
const counts = new Map<string, number>();
for (const answer of answers) counts.set(answer, (counts.get(answer) ?? 0) + 1);
console.log(counts);

On a clear-cut ticket you'll likely get ten identical answers. On a ticket like this one, where "not urgent for me" pulls one way and "month-end reports tomorrow" pulls the other, don't be surprised by a split.

Where the knowledge comes from: pretraining#

A model's general knowledge comes from pretraining: training on an enormous amount of text to get better and better at predicting the next token. To predict text well, the model has to absorb grammar, facts, writing styles, code patterns, and a great deal of reasoning ability. That knowledge ends up compressed into the model's parameters, the billions of numbers that define it.

That compression is lossy. Facts that appear thousands of times in the training data (how HTTP status codes work, what a refund is) are stored reliably. Facts that appeared a handful of times (the exact rate limit of a niche API, the founding year of a small company) are stored poorly, and the model reconstructs them by pattern. A reconstruction looks exactly as fluent as a memory. That's the root of most hallucination, which you'll study in Why Models Get Things Wrong.

Pretraining also stops at a date, the knowledge cutoff. Anything that happened later, and anything private (your product's docs, your customer's account, last week's pricing change), isn't in the model at all.

The builder's rule: use the model's built-in knowledge for language and general reasoning, and supply facts yourself. If a Deskmate reply needs the refund policy, put the refund policy in the request. If it needs the customer's plan, look it up and include it. Grounding answers in your own data is what Module 5 is about.

Why chat models follow instructions: post-training#

A model that has only been pretrained is a document completer. Give it "How do I reset my password?" and it might continue with three more questions, as if writing an FAQ page.

The models you call through an API have gone through post-training: further training on examples of instructions paired with good responses (often called instruction tuning), followed by training on human and AI feedback about which responses are better. This is what turns a completer into an assistant that answers questions, follows formatting rules, declines harmful requests, and adopts the persona you give it.

A few practical consequences:

• Roles are a learned convention. When you send a system prompt and a list of user and assistant messages, the provider formats them into a single token sequence with special markers. The model was trained to treat text after the system marker as standing instructions and text after the user marker as the person it's helping. It follows that convention very reliably, but it's training, not enforcement. That's the opening that prompt injection exploits, covered in Prompt Injection in Apps.
• Models are trained to be helpful and agreeable. That's mostly what you want, but it also produces sycophancy: going along with a user's wrong premise. If a customer writes "your docs say refunds are allowed within 90 days," a model may agree unless your prompt gives it the actual policy and permission to correct people.
• Instructions work best when they look like the training data. Clear tasks, explicit constraints, examples of the output you want, and data clearly separated from instructions. That's what Module 2 teaches.

Thinking: generation before the answer#

Current models add one more stage. Before writing the visible answer, a reasoning model generates a stretch of private working-out, usually called thinking. It's the same token-by-token loop, aimed at scratch work instead of the final response: breaking the problem down, checking intermediate results, reconsidering.

Thinking eases the "can't go back" problem: the model can try an approach, notice it's wrong, and switch before committing anything visible. It isn't free. Thinking tokens are generated sequentially like any other output, so they add latency, and they're billed as output tokens. Claude's current models decide adaptively how much to think, and you steer that with an effort setting.

For a builder, the question isn't "should models think?" but "how much thinking does this feature need?" A triage label rarely needs much. A reply that has to reconcile a refund policy, an order history, and an angry customer might. You'll learn to make that call in Reasoning Models: When Thinking Is Worth Paying For.

Putting the mental model to work#

When a feature misbehaves, trace the symptom back to the mechanism. It tells you where the fix lives:

None of these fixes require understanding the model's internals, only what it sees, how it writes, and what it was trained to do.

Next: Tokens, Context Windows, and What They Cost — how text becomes tokens, how to count them, and how they turn into your bill.