On-Device and In-Browser AI
Run small models in the browser and on phones with Transformers.js, WebGPU, and built-in platform models — and what they're genuinely good for.
Every AI feature you've built so far has the same shape: the user's data travels to a server, a model runs, and the answer travels back. That costs money per call, adds network latency, and means the data leaves the user's device. For some features, none of that is necessary. A model small enough to download can run right in the browser tab or on the phone, with no server call at all.
On-device AI is not a replacement for frontier models: the models are small, and devices vary enormously. But for the right jobs (suggesting help articles as someone types, detecting a language, a quick summary, filtering text before it's uploaded) it's instant, private, and free per call. You'll learn the three ways to run models on users' devices, build in-browser semantic search with Transformers.js, use the browser's built-in model with a server fallback, and design for devices that can't run them.
What running on the device buys you, and what it costs
Running the model on the user's hardware changes the economics and the trust story at the same time:
| You gain | You pay |
|---|---|
| Zero marginal cost: a million inferences cost you nothing | Download size: models are tens to hundreds of megabytes, sometimes gigabytes |
| No network latency: results as fast as the device can compute | Device variability: a new laptop and a five-year-old phone are different products |
| Privacy: text never leaves the device, which is easy to explain to users | Small-model quality: narrow tasks only, and you must test them |
| Offline: works on a plane or a flaky connection | No secrecy: anything you ship to the client, including a fine-tuned model, can be extracted |
| Battery and heat on mobile, plus a first-run delay while the model loads |
That table points to what on-device AI is genuinely good for: small, frequent, latency-sensitive tasks where a slightly worse answer is fine, and tasks where keeping data on the device is the point.
- Embeddings and semantic search over a small, mostly static corpus: help articles, a product catalog, a user's own notes.
- Classification and detection: language detection, topic suggestions, spam or toxicity pre-screening before upload.
- Light text transformations: short summaries, rewording, translation, proofreading.
- Vision and audio basics: background removal, image classification, and on-device speech recognition for dictation (see Speech-to-Text Features).
What it's not good for: anything where a wrong answer is costly, long reasoning, or anything that needs your private data or business logic, which belongs on the server anyway.