Why Models Get Things Wrong

Every AI feature you ship will be wrong some of the time. That isn't pessimism; it's the starting assumption that separates products people trust from demos that embarrass their makers. The useful question is never "how do I make the model perfect?" It's "what kinds of wrong will this feature produce, how often, and what happens to the user when it does?"

The failures aren't random. They come in a handful of recognizable types, each with a cause you can trace back to how models are built, and each with product patterns that contain it. In this lesson you'll learn the five you'll meet most: hallucination, knowledge cutoffs, sycophancy, instruction drift, and refusals. For each, you'll see how it shows up inside real features and what to build so a mistake becomes a minor annoyance instead of an incident.

Hallucination: fluent guesses#

A hallucination is output that's plausible, confident, and not supported by reality or by the input. It's the failure everyone has heard of, and its cause follows directly from how models generate text: a model produces likely-sounding tokens, and when it doesn't have a fact, the likely-sounding continuation is a fabricated one that looks exactly like a real one.

Inside product features, hallucinations cluster around specifics:

• Identifiers: order numbers, ticket IDs, invoice amounts, dates.
• References: URLs to help articles that don't exist, citations to the wrong document, API endpoints that sound right.
• Policies and facts about your own product that the model was never told, like "refunds are processed within 3 to 5 business days."
• Gap-filling: a summary that resolves an ambiguity the source left open.

The containment patterns, roughly in order of power:

1. Give the model the facts. A model asked about your refund policy without the policy will invent a plausible one. With the policy in the prompt, it mostly quotes it. Grounding in your own data is what Module 5 is about.
2. Make "I don't know" a valid output. If your schema requires an answer, you'll get one. Add an explicit way out, such as a nullable field or an answerable flag, and say in the prompt when to use it.
3. Verify specifics in code. Any ID, amount, or reference the model outputs can be checked against the source or your database. This is cheap and catches the most expensive mistakes.
4. Show sources. When users can see where an answer came from, they catch what your checks miss.

Here's the second and third pattern together, for a Deskmate feature that answers customer questions from help articles:

import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { z } from "zod";
 
const anthropic = new Anthropic();
 
const KbAnswer = z.object({
  answerable: z.boolean().describe("False if the articles do not contain the answer."),
  answer: z.string().nullable().describe("The answer, using only the articles. Null if not answerable."),
  sourceArticleIds: z.array(z.string()).describe("IDs of the articles the answer is based on."),
});
 
type Article = { id: string; title: string; body: string };
 
export async function answerFromKb(question: string, articles: Article[]) {
  let res;
  try {
    res = await anthropic.messages.parse({
      model: "claude-sonnet-5-5",
      max_tokens: 4000,
      output_config: { format: zodOutputFormat(KbAnswer), effort: "low" },
      system:
        "Answer the customer's question using only the help articles provided. " +
        "If the articles don't answer it, set answerable to false. Never guess at policies, prices, or timelines.",
      messages: [
        {
          role: "user",
          content:
            articles.map((a) => `<article id="${a.id}" title="${a.title}">\n${a.body}\n</article>`).join("\n") +
            `\n<question>${question}</question>`,
        },
      ],
    });
  } catch (err) {
    // Network, rate-limit, and server errors: let the caller retry or report them.
    if (err instanceof Anthropic.APIError) throw err;
    // parse() throws when the text isn't valid JSON for the schema (a refusal or a cut-off reply).
    return { kind: "handoff" as const };
  }
 
  const out = res.parsed_output;
  if (res.stop_reason !== "end_turn" || !out || !out.answerable || !out.answer) return { kind: "handoff" as const };
 
  // Code check: every cited article must be one we actually sent.
  const sentIds = new Set(articles.map((a) => a.id));
  if (out.sourceArticleIds.length === 0 || !out.sourceArticleIds.every((id) => sentIds.has(id))) {
    return { kind: "handoff" as const };
  }
  return { kind: "answer" as const, text: out.answer, sources: out.sourceArticleIds };
}

The fallback, handing the question to a human, is a product decision, not an error. A help widget that says "I'll connect you with the team" is far better than one that confidently invents a policy.