On-device AI

On-device AI means inference happens on the user's hardware. In a web app, that might be Transformers.js running a small model with WebGPU, or a built-in browser model such as Chrome's task APIs backed by Gemini Nano. In native apps, it might be Apple's Foundation Models framework or Android's ML Kit GenAI APIs.

The benefits are concrete: no network round trip, no marginal cost per call, offline support, and an easy privacy story because the text never leaves the device. That makes it a good fit for small, frequent tasks where a slightly weaker answer is acceptable: suggesting help articles as someone types, detecting language, searching a small corpus, or screening text before upload.

The costs are just as concrete. Models are small, downloads can be large, and devices vary enormously, so a feature that's instant on a new laptop may be slow or unavailable on an older phone. With built-in platform models you don't choose the model, and they aren't on every device. Anything you ship to the client can be extracted, so keep secrets and business logic on the server. Design every on-device feature with a capability check and a server fallback.