Capstone 5: A Content Product with Evals

Product descriptions are the content every online store needs and almost nobody enjoys writing. A store with 2,000 products and a manufacturer's spec sheet for each is the perfect customer for an AI writer, and also the perfect place for one to fail at scale: a description that invents "machine washable" for a dry-clean-only jacket isn't a style problem, it's a returns problem.

In this capstone you'll build ShelfCopy, a small (fictional) SaaS product that writes descriptions in each store's brand voice. You'll turn brand voice into data, check every draft with code and then with a calibrated model grader, generate whole catalogs at roughly half price with the Message Batches API, and bill merchants per published description through Stripe. The quality system is the product: it's what makes merchants trust the output enough to publish it.

What you'll build#

A merchant signs up, imports their catalog, and fills in a short brand voice form. From there:

• Preview writes one description in a few seconds, so the merchant can tune their voice settings.
• Generate catalog submits every product as a batch. Within the hour (often sooner), each product has a draft with a status: ready, needs_review, or failed.
• Review shows needs_review drafts with the reason: which code check failed, or which rubric criterion the grader flagged, with quoted evidence.
• Publish pushes the description to the store and records one billable unit.

Brand voice form ──► brand_voices (per store)
Catalog import  ──► products (title, category, attributes)
 
Preview (sync)                     Catalog (async, ~50% price)
  generationParams()                 batch 1: generationParams() × N
  └─► codeChecks()                   └─► codeChecks() ── fail ──► needs_review
       └─► gradingParams()               └─ pass ─► batch 2: gradingParams() × M
            └─► status                                └─► status per product
 
Merchant reviews ──► publish ──► Stripe meter event ("description_published")

The same request-building functions serve both paths, so a prompt change applies everywhere and your evals test exactly what production runs.

This build leans on Few-Shot Examples, Structured Outputs in Practice, Code-Based Checks and Assertions, Model-Graded Evals You Can Trust, Batch APIs and Async Processing, and Metering Usage and Billing with Stripe.

Prerequisites: Node 22+, a TypeScript project with Postgres, npm i @anthropic-ai/sdk zod postgres stripe, an Anthropic API key, and a Stripe account in test mode. The code uses an @/ import alias for src/ and the shared src/lib/ai.ts:

import Anthropic from "@anthropic-ai/sdk";
 
export const anthropic = new Anthropic();
 
export const MODELS = {
  fast: "claude-haiku-5-5",
  default: "claude-sonnet-5-5",
  best: "claude-opus-5-5",
} as const;