Building with Open-Weight Models
When open-weight models make sense, how licenses differ, where to run them (hosted inference vs. your own GPUs), and how to serve them with vLLM.
Sooner or later someone asks whether you should just run your own model. Maybe a large customer's security review says no data may leave their cloud. Maybe triage now processes millions of tickets a month and the invoice is getting attention. Maybe you want to fine-tune a model on your own data and own the result.
Open-weight models make all of that possible, and they also make it easy to trade a predictable API bill for an unpredictable infrastructure project. You'll learn when open-weight models genuinely make sense for a product feature, how their licenses differ and what to check, the three ways to run them (hosted inference, dedicated deployments, and your own GPUs), and how to serve a model with vLLM and call it from your app with code you already know.
When open-weight models make sense
An open-weight model is one whose weights (the learned parameters that make up the model) are published for anyone to download and run. Families you'll see often include Llama (Meta), Qwen (Alibaba), Mistral, Gemma (Google), DeepSeek, and gpt-oss (OpenAI). New versions arrive every few months, so check each family's current release rather than relying on a name from a blog post.
For a product team, the reasons to use one come down to a short list, and each comes with a price:
| Reason | What you gain | What you take on |
|---|---|---|
| Data control | Prompts and documents stay in infrastructure you or your customer control | Securing and operating that infrastructure |
| Customer deployment | You can offer an on-premises or customer-cloud version of an AI feature | Supporting your model on hardware you don't run |
| Cost at sustained volume | For a narrow, high-volume task, a smaller model on busy GPUs can cost less per request than an API | Paying for GPUs whether or not they're busy, plus engineering time |
| Stability | The model never changes or gets retired unless you decide to upgrade | You also miss improvements until you do the upgrade work |
| Customization | Full fine-tuning of the weights for your task | A training and evaluation pipeline to maintain |
One reason that isn't good enough on its own: "it's free." The weights are free; running them isn't. Before committing, check whether something simpler solves the problem: provider data-retention options and regional processing for privacy concerns (see PII, Data Retention, and Provider Policies), or a smaller hosted model and the batch API for cost (see Batch APIs and Async Processing).
Capability is the other half. The strongest closed models generally lead on the hardest, most open-ended tasks, while open-weight models are often competitive on narrow ones like classification and extraction. That makes them a natural fit for the high-volume features where cost matters most. Your own test set decides.