Local LLM

A local LLM is a model you run yourself rather than reach through a provider's API. In practice that means an open-weight model, usually quantized, served by a tool such as Ollama, llama.cpp, LM Studio, or vLLM. Ollama and vLLM expose OpenAI-compatible endpoints, so the same client code can talk to a local model or a hosted one by changing a base URL.

Builders reach for local models for a few reasons: data that must not leave a machine or network, workloads large enough that per-token pricing hurts, offline products, and development loops where you want to iterate without spending. It's also a cheap way to prototype a feature before choosing a provider.

The mistake is assuming local means free and equivalent. You pay in hardware, memory limits, setup, and operations, and a model small enough for a laptop will be weaker than a frontier model on open-ended or reasoning-heavy work. Local models shine on narrow, well-specified tasks: classification, extraction into a schema, short rewrites, embeddings. Treat the choice like any model decision: run your eval set against the local candidate and compare it with the hosted baseline before you switch.