Running Models Locally with Ollama and llama.cpp
Run capable models on a laptop for development, privacy, or offline features — quantization explained, and an OpenAI-compatible local endpoint in minutes.
At some point you'll want a model that runs on your own machine. Maybe your test suite makes hundreds of model calls and you're tired of paying for them on every push. Maybe you want to prototype against real customer tickets without sending them anywhere. Maybe you're building a feature that has to work on a plane. Ten years ago "run the model yourself" meant a research cluster. Today a capable model runs on a recent laptop, and serving it behind an API takes one command.
In this lesson you'll run open-weight models locally with Ollama and llama.cpp, understand quantization well enough to pick the right file for your hardware, call a local model from TypeScript through an OpenAI-compatible endpoint, and decide which Deskmate jobs a local model can genuinely handle.
Why run a model on your own machine
Running locally is a trade: you give up the best available quality and take on the operations work, and in return you get control. For a product team, the reasons that hold up are specific:
| Reason | What it looks like in practice |
|---|---|
| Development loop | Tests, demos, and prompt iteration hit a local endpoint; no API key in CI, no bill per run |
| Data that can't leave | A customer's contract or a regulator says ticket text stays on their hardware |
| Offline features | Field workers, flights, factory floors, and air-gapped networks |
| Predictable cost at volume | A bulk backfill over millions of rows on hardware you already own |
| No surprise changes | The model file never changes unless you replace it |
And the reasons that don't: "it's free" (your time and hardware aren't), and "it's private, so it's safe" (a local model is just as vulnerable to prompt injection and just as able to be wrong).
The quality gap is real. Small local models are good at narrow, well-specified tasks and noticeably worse at long reasoning, nuanced writing, and following complicated instructions. Treat any local model as a candidate you have to test, exactly like the process in Choosing a Model per Feature.
This lesson is about one machine: a laptop, a workstation, a single server. Serving open-weight models to many users on GPU servers with vLLM, and the license questions that come with shipping them, are covered in Building with Open-Weight Models.