Choosing a Model per Feature

Most products pick a model once, early, usually the one the first prototype happened to use, and then every feature inherits it. The result is predictable: the autocomplete feels sluggish because it runs on a model built for hard reasoning, the nightly tagging job costs more than the feature earns, and the one feature that genuinely needs a stronger model is stuck on the default because nobody wants to touch the shared setting.

Model choice belongs to the feature, not the app. In this lesson you'll learn a selection process you can repeat for every feature: write down the bar before you test anything, shortlist candidates using hard constraints first, measure them on your own examples with a small harness that reports quality, latency, and cost side by side, and record the decision so you can revisit it when the next model generation arrives.

Define the bar before you look at results#

If you test models before deciding what "good enough" means, you'll rationalize whatever you see. The fast model looks fine on the three examples you glanced at, or the big model's answers feel nicer and you convince yourself the latency doesn't matter. Write the bar down first, as numbers, for each feature.

Every bar has three parts:

• Quality: the share of test examples that must pass your check. What counts as a pass depends on the feature type: an exact label match for classification, correct fields for extraction, a rubric for open-ended text.
• Latency: a budget for the slowest typical request, usually the 95th percentile (p95: the latency that 95% of requests come in under). Averages hide the slow requests users actually notice.
• Cost: a ceiling per request, or per thousand requests, derived from what the feature is worth. Unit Economics of AI Features shows how to calculate it.

Here's what that looks like for four Deskmate features. The numbers are targets a team might set, not measurements:

Notice how differently these features weigh the same three dimensions. Triage is volume-bound, so cost dominates. The summary sits in the critical path of opening a ticket, so latency dominates. The escalation review is rare and high-stakes, so quality dominates and cost barely matters. One model for all four would be wrong for at least three of them.

Then turn each bar into something code can check:

export type FeatureBar = {
  minPassRate: number; // e.g. 0.95
  maxP95Ms: number; // the latency budget for this surface
  maxUsdPer1k: number; // from your unit economics
};
 
export type Result = { candidate: string; passRate: number; p95ms: number; usdPer1k: number };
 
/** The cheapest candidate that clears every bar, or null if none does. */
export function choose(results: Result[], bar: FeatureBar): Result | null {
  const passing = results.filter(
    (r) => r.passRate >= bar.minPassRate && r.p95ms <= bar.maxP95Ms && r.usdPer1k <= bar.maxUsdPer1k,
  );
  return passing.sort((a, b) => a.usdPer1k - b.usdPer1k)[0] ?? null;
}

"Cheapest that clears every bar" is a deliberate rule. Once a candidate meets the bar, extra quality you didn't ask for is usually not worth paying for. If you find yourself wanting the more expensive model anyway, that's a sign your bar is wrong, and the fix is to raise the bar explicitly, not to override the result.