A phone with no signal can now summarize an email, rewrite a paragraph, or answer a question about a photo — entirely offline. No API call, no data leaving the device, no spinner waiting on a network round trip. That capability didn't exist in any practical form three years ago. It exists now because of a convergence of smaller, denser models and purpose-built silicon, and it's changing where teams choose to put the "AI" in their AI features.
On-device AI means running inference for a machine learning model — often a large language model, though the same ideas apply to vision and speech models — directly on the user's hardware instead of sending a request to a remote server. The model weights live on the device. The computation happens on the device's CPU, GPU, or a dedicated neural processing unit. Nothing about the interaction requires connectivity.
This post explains how that actually works, why it's become viable now rather than five years ago, what it changes for teams building products, and where it still falls short of cloud-hosted alternatives.
It's worth being precise about terminology up front, because "on-device," "edge," and "local" AI get used almost interchangeably, and the overlap causes confusion in planning conversations. On-device AI specifically means inference on the end-user's personal hardware — a phone, laptop, tablet, or wearable. Edge AI is a broader term that also includes inference on nearby infrastructure the user doesn't own outright, such as a retail store's local server, a factory floor gateway, or a cell tower's compute unit. Local AI is often used as a catch-all for both, and sometimes for self-hosted server deployments that simply aren't a public cloud API. This post focuses specifically on the end-user-hardware case, since that's where the LLM conversation has moved fastest.
How On-Device AI Actually Works
Running a large language model requires two things: the model's parameters (weights) stored somewhere accessible, and enough compute to perform the matrix multiplications that turn an input into an output, token by token. Cloud inference does this on a server with a full-size GPU or TPU, often with tens of billions of parameters loaded across multiple accelerators. On-device inference has to do the same job on hardware with a fraction of the memory bandwidth and power budget.
Three techniques make this feasible:
- Quantization — reducing the numerical precision of model weights, typically from 16-bit or 32-bit floating point down to 8-bit, 4-bit, or even lower integer representations. A model quantized to 4-bit weights takes roughly a quarter of the memory of its 16-bit original, with a measurable but often tolerable accuracy cost.
- Smaller, denser architectures — rather than shrinking a large model naively, model builders train compact architectures from scratch (or distill them from larger "teacher" models) so that a 1-8 billion parameter model retains more capability per parameter than earlier generations of small models did.
- Hardware acceleration — modern phone and laptop chips now ship with a dedicated neural processing unit (NPU) alongside the CPU and GPU, designed specifically for the low-precision tensor operations that inference requires. This is distinct from training hardware; NPUs are optimized for fast, low-power forward passes, not backpropagation.
Together, these mean a model that once needed a rack-mounted GPU can now fit in a few gigabytes of RAM and run at usable speed on a chip smaller than a fingernail.
The Inference Pipeline, Simplified
A typical on-device LLM pipeline looks like this:
- The app loads a quantized model file (commonly in a format like GGUF, ONNX, or a vendor-specific runtime format) into memory.
- A tokenizer converts the input text into numerical tokens.
- The runtime — llama.cpp, ONNX Runtime, Core ML, TensorFlow Lite, MediaPipe, or a vendor SDK — feeds tokens through the model's layers using whichever accelerator is available (NPU first, falling back to GPU or CPU).
- Output tokens are generated one at a time and streamed back to the interface, exactly as a cloud API would stream a response.
The user-facing behavior looks identical to a cloud-backed chat interface. The difference is entirely in where step 3 happens.
Why It Matters Right Now
For most of the current LLM era, "AI feature" has meant "network call to a hosted model." That default made sense when only frontier-scale models were capable enough to be useful, and those models were too large to run anywhere but a data center. That constraint has loosened. Open-weight models in the 1-10 billion parameter range have gotten good enough, and consumer hardware has gotten capable enough, that a meaningful slice of everyday AI tasks — summarization, autocomplete, simple classification, basic reasoning, image captioning — no longer requires a frontier model or a network trip.
This shift shows up in product decisions, not just benchmarks. Operating systems now ship built-in on-device model runtimes as a platform feature rather than a novelty, and app developers can call into them the way they'd call any other system framework. That's a structural change: on-device inference stopped being something only ML specialists could wire up with a custom C++ build, and became an SDK call.
The practical effect is that "should this feature call the cloud or run locally" is now a real design decision for a wide range of products, not a hypothetical. Teams are weighing it feature-by-feature: a smart-reply suggestion might run locally; a complex multi-step research agent still needs a frontier model in the cloud.
What Runs Well Locally — and What Doesn't
Not every LLM task is a good fit for on-device execution. The deciding factors are model size versus task difficulty, latency sensitivity, and how much the task benefits from up-to-date or broad world knowledge.
| Task type | Fits on-device well | Better suited to cloud |
|---|---|---|
| Autocomplete / text prediction | Yes — low latency need, narrow scope | |
| Grammar and tone rewriting | Yes | |
| Short summarization (single document) | Yes, with smaller models | |
| Simple intent classification / routing | Yes | |
| Multi-step agentic workflows | Yes — needs stronger reasoning | |
| Long-context document analysis | Yes — memory-constrained on device | |
| Tasks needing current/real-time info | Yes — local models are static snapshots | |
| Code generation for complex systems | Usually yes — frontier models still lead | |
| Voice transcription, simple Q&A | Yes |
The general rule: the more a task depends on breadth of knowledge, long context windows, or complex multi-step reasoning, the more it benefits from a large model that only a data center can currently host affordably. The more a task is narrow, repetitive, and latency-sensitive, the better a candidate it is for local execution.
On-Device, Cloud, or Hybrid: Choosing an Approach
Framing this as an either/or decision undersells how most real products end up architected. In practice, three patterns dominate, and the choice between them tends to track the product's risk tolerance around latency, cost, and correctness as much as raw capability.
Cloud-only is still the right default for anything that needs the strongest available reasoning, the widest world knowledge, or frequent model upgrades without shipping a new app build. It's also the simplest to build and monitor, since all inference happens in one controlled environment.
On-device-only makes sense for narrowly scoped features where offline reliability or privacy is a hard requirement — an on-device transcription feature for a note-taking app, for instance, or a keyboard's next-word prediction. The tradeoff is accepting a capability ceiling and the fragmentation costs discussed below.
Hybrid routing is where most sophisticated products are converging: a small on-device model acts as a first pass, handling the majority of routine requests locally and instantly, while harder or ambiguous requests get escalated to a cloud model automatically. Done well, this looks like a single feature to the user — they never see the seam. Done poorly, it introduces its own failure mode, where the escalation logic itself becomes a source of bugs and inconsistent behavior between the "fast path" and "smart path" responses.
Deciding among these isn't purely technical. It also depends on how much control the team wants over the user experience during a network outage, how sensitive the underlying data is, and how much engineering capacity exists to maintain two inference paths instead of one.
Practical Implications for Builders
For teams shipping AI features, the on-device option changes the calculus on a few fronts.
Cost and Latency
Every cloud inference call has a marginal cost — API pricing per token, plus the infrastructure cost if you're self-hosting. At scale, for high-frequency, low-complexity requests, that adds up. On-device inference shifts that marginal cost to zero (the compute happens on hardware the user already owns) at the price of a one-time integration cost and a larger app download or model asset. Latency follows a similar pattern: a local model has no network round trip, which matters for typing-speed features like autocomplete or real-time suggestions where even 200ms of network latency is noticeable.
Privacy and Compliance
Data that never leaves the device never has to be logged, transmitted, or governed by a third party's data retention policy. For products handling sensitive categories — health notes, financial details, private messages — on-device processing sidesteps an entire category of compliance and vendor-risk questions. This is one of the more durable arguments for on-device AI: it's not just a performance optimization, it changes the data-handling story the product can tell.
Offline Reliability
Cloud-dependent AI features fail silently or visibly whenever connectivity drops — in transit, in low-signal areas, or during outages at the AI provider. A local model keeps working. For products used in the field (logistics, industrial, healthcare, travel), that reliability is not a nice-to-have.
The Integration Cost
None of this is free from an engineering standpoint. Teams adopting on-device AI need to budget for:
- Choosing and validating a quantized model that fits the target device's memory and thermal envelope
- Testing across a fragmented range of hardware (older phones, low-RAM devices, varied NPU support)
- Handling graceful fallback to cloud inference when the device can't run the local model well
- Managing model updates and versioning as part of the app release cycle, rather than a server-side deploy
- Accepting a capability ceiling — the on-device model will not match a frontier cloud model on hard reasoning tasks
Real Limitations and Open Questions
On-device AI is not a strictly better version of cloud AI; it's a different tradeoff, and several limitations are unresolved rather than merely inconvenient.
Capability gap. Even the best small, quantized models trail frontier cloud models on complex reasoning, long-context understanding, and tasks requiring broad world knowledge. Quantization itself introduces a measurable accuracy cost — how much depends on the model and the task, and it is not always predictable in advance without testing.
Device fragmentation. Unlike a controlled cloud environment where every request hits the same GPU cluster, on-device inference has to contend with years of hardware variation. A model that runs smoothly on a current flagship phone may be unusably slow — or simply unable to fit in memory — on a three-year-old mid-range device. Building a feature that degrades gracefully across that spread is genuine engineering work, not a configuration flag.
Battery and thermal cost. Sustained inference is compute-intensive. Running a local model for an extended session can noticeably affect battery life and, on some devices, trigger thermal throttling that slows the model down mid-task.
Static knowledge. A model shipped inside an app is frozen at whatever training cutoff it had when it was packaged. It has no access to current events, no retrieval of fresh information, and no way to be corrected without a full app or model update — unlike a cloud model that can be swapped or supplemented with retrieval infrastructure behind the scenes at any time.
Security of the model file itself. Shipping model weights inside an app means a proprietary model — if it isn't an open-weight one — sits on the user's device and can potentially be extracted. Teams shipping custom fine-tuned models on-device need to think about this differently than they would a server-side model that never leaves their infrastructure.
Tooling maturity. The runtimes and formats (GGUF, ONNX, Core ML, LiteRT, vendor NPU SDKs) are still evolving quickly, and a pipeline built against one runtime's quirks today may need rework as tooling consolidates.
What to Watch Next
A few threads are worth tracking as this space matures:
- Hybrid architectures becoming the default, where a lightweight on-device model handles routine requests and automatically escalates harder ones to a cloud model — designed as a single feature rather than two separate systems bolted together.
- Smaller models closing the capability gap on specific task categories (summarization, classification, structured extraction) even as they remain behind on open-ended reasoning.
- Standardization of on-device model formats, reducing the current fragmentation across GGUF, ONNX, Core ML, and vendor-specific runtimes, which would lower the integration cost for teams supporting multiple platforms.
- NPU capability becoming a baseline assumption in new hardware, the way GPU acceleration became assumed for graphics — shifting on-device AI from an opportunistic feature to a default design assumption.
- Regulatory and privacy pressure pushing more categories of data processing toward on-device by default, particularly in regions with strict data residency or health-data rules.
None of these trends make cloud inference obsolete. Frontier capability will likely keep living in data centers for the foreseeable future, simply because the largest, most capable models are too large to fit on a phone. What's changing is the boundary of which tasks need that scale at all.
FAQ
What is on-device AI in simple terms?
On-device AI means a machine learning model runs its calculations directly on your phone, laptop, or other hardware instead of sending your request to a remote server. The model and the compute both live locally, so the feature can work without an internet connection.
Can a phone really run a large language model?
Yes, within limits. Phones run smaller, quantized versions of LLMs — typically in the 1-8 billion parameter range — rather than the largest frontier models, which still require data-center-scale hardware. These smaller models handle tasks like summarization, rewriting, and simple Q&A well, but not complex multi-step reasoning.
Is on-device AI more private than cloud AI?
Generally yes, because the data being processed never has to leave the device or pass through a third party's servers. This reduces exposure for sensitive data categories and simplifies certain compliance considerations, though it doesn't eliminate every privacy concern — for instance, how the app itself stores or logs data locally still matters.
What is quantization and why does it matter for on-device AI?
Quantization reduces the numerical precision of a model's weights, for example from 16-bit to 4-bit representations, which shrinks the memory footprint dramatically. This is what makes it possible to fit a multi-billion-parameter model into the RAM of a phone, at some cost to accuracy that varies by model and task.
Does on-device AI work without internet access?
Yes, that's one of its main advantages. Once the model is downloaded and installed as part of the app, inference happens entirely locally, so the feature keeps working in airplane mode, in low-signal areas, or during a network outage.
Why don't all AI features just run on-device?
Because on-device models are smaller and less capable than frontier cloud models, and they can't access real-time information or extremely long context windows. Tasks that need deep reasoning, broad knowledge, or the latest information still perform better through a cloud-hosted model.
What tools are used to build on-device AI features?
Common runtimes include llama.cpp, ONNX Runtime, Apple's Core ML, Google's LiteRT (formerly TensorFlow Lite), and MediaPipe, along with platform-level on-device model frameworks that operating systems now expose as built-in SDKs. Choice of tool usually depends on the target platform and the model format available.
Teams weighing whether a feature belongs on-device, in the cloud, or split between both can get that architecture right faster with hands-on help from Woyce Technologies.