Cost, Latency & Scale
AI features that are fast and pay for themselves
Understand and control the economics of AI features: reading your bill, unit economics, prompt caching, response caching, batch processing, latency optimization, rate limits and per-user budgets, token efficiency, and scaling workloads.
- Lessons
- 9
- Reading time
- 97 min
- Level
- Advanced
- Access
- Full course
Lessons in this module
- 10.111 min
Understanding Your AI Bill
Where AI spend actually comes from — input, output, thinking, cached, and tool tokens — and how to attribute every dollar to a feature and a customer.
- 10.211 min
Unit Economics of AI Features
Calculate cost per request, per user, and per outcome; model gross margin under different usage patterns; and spot features that lose money at scale.
- 10.311 min
Prompt Caching in Practice
Structure prompts so stable content is cached, verify cache hits in usage data, and avoid the silent invalidators that quietly double your bill.
- 10.410 min
Exact and Semantic Response Caching
Skip the model entirely for repeated requests: exact-match caches, semantic caches with embeddings, and the correctness traps of each.
- 10.511 min
Batch APIs and Async Processing
Process large volumes at roughly half price with batch APIs: when they fit, how to submit and collect results, and designing features around delay.
- 10.611 min
Cutting Latency: Time to First Token and Beyond
Measure where time goes, then cut it: smaller models, lower effort, shorter outputs, parallel calls, speculative work, and regional endpoints.
- 10.711 min
Rate Limits, Quotas, and Per-User Budgets
Handle provider rate limits gracefully and enforce your own: per-user quotas, spend caps, plan-based limits, and abuse protection.
- 10.810 min
Token Efficiency: Same Quality, Fewer Tokens
Trim prompts, compress context, constrain output length, and prune history — and prove with evals that quality didn't drop.
- 10.911 min
Scaling AI Workloads
Queues, concurrency limits, backpressure, and capacity planning for AI traffic that spikes — without blowing through rate limits or budgets.
About this module
Every AI feature has a running cost that traditional software doesn't. Each summary, drafted reply, or chat message is billed by the token, and those tokens come in several kinds with very different prices: uncached input, cached input, visible output, and the hidden reasoning tokens that thinking models produce before they answer. Latency works the same way. The model you pick, the context you send, and the length of the answer all show up as seconds your users wait.
That makes cost and latency product decisions, not just infrastructure details. A feature that customers love can lose money on its heaviest users, and growing adoption can shrink your margin instead of improving it. A feature that takes eight seconds to respond can go unused even when its output is excellent. Teams that understand where the money and time go can keep improving a feature without being surprised by the invoice.
This module starts with understanding your AI bill and the unit economics of AI features: cost per request, per user, and per outcome, and how margins behave under different usage patterns. It then covers the main levers in order of typical payoff: prompt caching, exact and semantic response caching, batch APIs for work that can wait, cutting time to first token and total latency, and token efficiency that keeps quality while sending and generating less.
The last lessons cover operating at scale: handling provider rate limits gracefully, enforcing per-user quotas and spend caps, and using queues, concurrency limits, and backpressure so traffic spikes don't blow through your limits or your budget. Examples use TypeScript and the Anthropic API with current Claude models, and every cost-cutting change is paired with evals that prove quality didn't drop.
Questions about Cost, Latency & Scale
How much does it cost to run an LLM feature?
Multiply the tokens each call uses by the model's per-token prices, then multiply by how often the feature runs. Input, output, thinking, and cached tokens are priced differently, so read the usage data each API response returns rather than estimating from text length. Per-call costs range from fractions of a cent for short classification calls on small models to several cents for long, context-heavy generation on larger models.
How can I reduce my LLM API costs?
Start by measuring cost per feature so you know where the money goes. The biggest levers are usually choosing the smallest model that passes your evals, caching stable prompt prefixes, sending less context, lowering reasoning effort, and moving non-urgent work to a batch API. Check every change against your evals so cheaper doesn't quietly mean worse.
Do thinking tokens cost money?
Yes. Reasoning models bill their thinking as output tokens, even when the thinking text isn't returned to you. On reasoning-heavy requests thinking can cost more than the visible answer, so lowering the effort setting for simple tasks is one of the quickest savings.
How much does prompt caching save?
Cached input is billed at a small fraction of the normal input price, about a tenth or less on current Claude models, while writing to the cache costs somewhat more than normal input. Savings depend on how much of each request is a stable prefix and how often it's reused, so verify hits in the usage data rather than assuming them.
How do I make an LLM feature respond faster?
Stream the response so users see output immediately, then reduce the work itself: use a smaller model or lower effort where quality allows, shorten outputs, cache stable prompt prefixes, and run independent calls in parallel. Measure time to first token and total time separately, because they have different fixes.