NPUs and AI Chips: A Buyer's Guide to the New Hardware

Open a laptop spec sheet today and you'll find a new line item sitting next to the CPU and GPU: an NPU, quoted in TOPS, with a marketing sticker calling the machine an "AI PC." Phone makers have run the same play for longer — every new flagship chip announcement leads with its neural engine's throughput number. Almost none of that number tells you what you actually need to know, which is whether the chip does anything useful for the software you run.

That matters whether you're buying a fleet of laptops, choosing hardware for a product, or deciding where your app's AI features should run. Pay for an NPU your software never uses and you've wasted budget; ignore one your software depends on and features drain batteries or fall back to the cloud.

This guide cuts through the spec-sheet noise. It explains what an NPU actually is, how it differs from the CPU and GPU already in your machine, why every hardware vendor suddenly wants one, where the real limitations sit, and what questions to ask before treating "NPU inside" as a reason to buy.

What an NPU Actually Is

A neural processing unit is a chip — or more often a block of circuitry embedded alongside a CPU and GPU on the same die — designed to do one narrow class of math very efficiently: the matrix multiplications and additions that make up a neural network's forward pass. That's it. It isn't a general-purpose processor, and it isn't magic; it's a piece of silicon shaped around one repeated arithmetic pattern.

To see why that shape matters, it helps to compare the three processors that now typically ship together in a single device:

A CPU executes instructions one after another (with some pipelining and parallel cores), which makes it excellent at logic, branching, and anything unpredictable, but inefficient at repeating the same simple calculation across millions of data points. A GPU flips that trade-off: thousands of simple cores doing the same operation on different pieces of data simultaneously, which is exactly what both 3D rendering and neural network math require. GPUs became the default AI training and inference chip almost by accident — they already existed for graphics, and it turned out matrix multiplication for rendering triangles and matrix multiplication for neural networks are close enough that the same hardware serves both, a convergence explored further in GPU vs TPU vs ASIC.

An NPU narrows that specialization further. It's built specifically for the operations inside a trained neural network — mostly multiply-accumulate operations at reduced numerical precision (8-bit or even 4-bit integers instead of the 32-bit floats a CPU favors) — and nothing else. Because it does less, it can do that one thing with far less energy per operation than a CPU or GPU spends on the same math. That efficiency, not raw speed, is the entire reason NPUs exist.

Where NPUs sit in the compute stack

An NPU virtually never runs a model on its own. A typical inference pass on an NPU-equipped device still routes through the CPU for orchestration, memory management, and pre/post-processing, with the NPU called in specifically for the heavy matrix math in the middle. So "the NPU ran it" is usually shorthand for "the NPU did the expensive part of the job while the CPU handled everything else around it."

The physical layout reflects this division of labor. On most modern phone and laptop chips, the NPU isn't a separate card or module the way a discrete GPU is — it's one block among several on the same piece of silicon as the CPU cores, GPU cores, image signal processor, and memory controller, all sharing the same pool of on-chip memory. That shared-memory design is deliberate: moving data between separate chips (as happens between a CPU and a discrete GPU over a PCIe bus) costs time and power, a tradeoff that shapes how custom AI silicon gets designed in the first place, and a big share of what makes an NPU efficient for small, frequent inference calls is that it doesn't have to pay that data-movement tax. The trade-off is that an integrated NPU can't be swapped or upgraded independently — it ships with the rest of the chip and its capabilities are fixed for the life of the device.

How inference actually gets scheduled

None of this is visible to an end user, and most of it isn't visible to app developers either. A phone's operating system typically owns the decision about where a given AI task runs, based on rules that weigh the model's size, the precision it needs, current battery and thermal state, and what else is competing for the NPU at that moment. Two identical phones running the identical app can route the same feature to different processors depending on battery level or background load. That's a deliberate design choice — it lets the OS optimize for battery life and responsiveness dynamically — but it also means a spec sheet claim about NPU capability describes a ceiling the software may or may not actually reach in everyday use.

Benefits of NPUs: Why Every Chipmaker Added One

The wave of NPU announcements from 2023 onward — in phone SoCs, then laptop chips, then desktop processors — traces back to one shift: AI features moved from something a cloud server did on your behalf to something users expect to run locally, instantly, and without a network round trip. Autocomplete suggestions, live transcription, background blur in video calls, on-device image search, voice assistants that work offline — all of these are neural network inference jobs that a device now has to do dozens of times a minute, and doing them on a GPU or CPU burns battery fast enough to be noticeable.

The push toward local inference comes from a handful of converging pressures, and each one is also a benefit an NPU delivers when software actually uses it.

Longer battery life for always-on features

Running inference on a CPU or GPU for always-on features (wake-word detection, camera effects, keyboard prediction) drains a laptop or phone battery far faster than routing the same work through a purpose-built low-power block. Because an NPU spends much less energy per operation on low-precision matrix math, features that run constantly in the background stop being a visible drain. For users, that is the difference between leaving a feature switched on and turning it off to get through the day.

Lower latency and offline operation

A round trip to a cloud model adds tens to hundreds of milliseconds and depends on a network connection; local inference on an NPU responds immediately and works offline. Live captions keep up with speech, camera effects track movement without lag, and voice commands work on a plane or in a basement. For features that feel broken if they hesitate, removing the network from the loop matters more than raw model size.

Privacy and data residence

Keeping voice, camera, and personal-file processing on-device avoids sending that data to a third-party server, which matters both for user trust and for regulated industries. If the audio from a meeting or the photos on a phone never leave the device, a whole category of data-handling questions disappears. That makes some features possible in settings where a cloud dependency would have been a non-starter.

Lower cost at scale

Every inference call a device handles locally is one a company doesn't pay a cloud GPU bill to serve, which matters enormously once a feature runs on hundreds of millions of devices. Shifting high-frequency, low-complexity inference to the user's own hardware turns a recurring server cost into a one-time hardware capability.

Headroom for the CPU and GPU

When the NPU takes the steady background AI work, the CPU and GPU stay free for the application itself. A video call with background blur and noise suppression leaves more capacity for the call, and the device runs cooler while doing it.

Operating system vendors then built the software layer to make this practical — frameworks such as Apple's Core ML or Google's AI Edge stack that let an app request "run this model" without the developer needing to write chip-specific code, with the OS deciding at runtime whether to route the job to the NPU, GPU, or CPU depending on what's free and what the model needs. That software layer is arguably as important as the silicon: an NPU sitting idle because no app knows how to call it is just a wasted chunk of die space, and for the first couple of years of the "AI PC" push, that was closer to reality than the marketing suggested.

NPU Use Cases

Live transcription and captions

Turning speech into text in real time is a continuous inference job: audio arrives every fraction of a second and the model has to keep up. Running it in the cloud adds latency and needs a connection; running it on a CPU drains the battery during a long meeting. On an NPU, live captions and meeting transcription can run locally for hours with modest power draw, and the audio never has to leave the device, which matters for confidential conversations.

Camera effects in video calls

Background blur, background replacement, eye-contact correction, and lighting adjustment all run a vision model on every frame of a video call. Doing that on the GPU competes with the call itself and heats the laptop. These effects are among the most common reasons laptop makers point to NPUs, because they are steady, predictable workloads that suit a low-power block well.

Wake words and offline voice commands

Voice assistants need to listen for a trigger phrase constantly and, ideally, handle simple commands without a network. A small always-on model is a natural NPU job: it runs nonstop, uses little power, and responds instantly. The result is voice control that works offline and doesn't keep the main processor awake just to listen.

Keyboard prediction and on-device text features

Next-word prediction, autocorrect, and smart replies run a model on almost every keystroke. Sending each one to a server would be slow and expensive at scale. Phones route this work to the NPU, keeping typing responsive and the text private, which is a large share of the inference a typical phone performs in a day.

On-device photo search and image processing

Searching your photo library for "dog on a beach" requires classifying and indexing images, which phones do locally in the background. Computational photography features, such as noise reduction and scene detection, also lean on the neural engine. Users get fast search and better photos without uploading their library.

Small local language models

Compact, quantized language models can summarise, draft, and answer questions over local files on NPU-equipped devices. They are far less capable than data-center models, but they suit private, narrow tasks where a cloud call is undesirable.

What the TOPS Number Does and Doesn't Tell You

TOPS — trillions of operations per second — is the headline spec on every NPU announcement, and it is the single most misused number in this entire category. It measures peak theoretical throughput for a specific, favorable operation type (usually 8-bit integer multiply-accumulates) under ideal conditions. It does not measure:

1. Real-world model performance. A model that needs more memory bandwidth than the NPU can supply will bottleneck long before it hits peak TOPS, regardless of what the spec sheet says.
2. Precision compatibility. A TOPS figure quoted at INT8 doesn't transfer to a model that needs FP16 precision to stay accurate; some NPUs handle the conversion well, others lose real quality doing it.
3. Software support. A chip with excellent TOPS but immature drivers or a limited operator set (the specific neural network layer types it can execute) may fall back to the CPU for large parts of a model it doesn't fully support, erasing most of the theoretical advantage.
4. Which models actually run there. OS and app vendors decide, often opaquely, which specific features get routed to the NPU versus the GPU or cloud. A high-TOPS chip is irrelevant to a task the software never sends its way.

Comparing two chips by TOPS alone is a bit like comparing two cars by peak horsepower without checking whether either one's transmission can actually put that power to the road under normal driving. It's a useful ceiling, not a guarantee.

Memory bandwidth deserves particular attention because it's the constraint TOPS figures hide most effectively. Running a neural network means repeatedly loading its weights from memory into the compute units, and for many real models — especially language models generating one token at a time — that data movement, not the arithmetic itself, is the slow part. A chip can advertise enormous peak TOPS and still deliver disappointing real-world speed if its memory system can't keep the compute units fed. This is why two devices with similar TOPS ratings can produce noticeably different real-world speeds on the same task, and why independent, task-specific benchmarks matter more than the headline spec.

A more useful way to evaluate a chip

For anyone actually deciding what to buy or spec for a project, TOPS is a filter, not a decision criterion. More productive questions:

• What specific features (transcription, translation, image generation, code completion) is this device advertised to run locally, and can you test them before buying?
• Does the manufacturer publish which model sizes and precisions the NPU actually supports, or only a peak TOPS figure?
• Is there independent benchmark data for real workloads — tokens per second on a specific model, latency for a specific task — rather than synthetic microbenchmarks?
• How much unified/shared memory does the device have, since large models are often memory-bound before they're compute-bound?

Common NPU Buying and Building Mistakes

Comparing chips by TOPS alone

TOPS is a peak figure at a favourable precision under ideal conditions. Choosing between devices on that number ignores memory bandwidth, operator support, and whether your software ever routes work to the NPU. Two chips with similar TOPS can feel very different on the same task, so the headline figure should narrow a shortlist, not decide it.

Paying for an "AI PC" label without checking features

The label confirms the hardware exists, not that any feature you need will use it. Fleet buyers sometimes pay a premium for NPU-equipped laptops when the applications their staff actually use run nothing on the NPU. Confirm which specific features run locally on the exact model, in your region and OS version, before the premium is justified.

Expecting frontier models to run locally

Consumer NPUs target models in the hundreds of millions to low billions of parameters. Teams that plan a product around running a large general-purpose assistant on-device usually discover the model doesn't fit or loses too much quality when quantized. Plan local inference around smaller, task-specific models and keep the heaviest work in the cloud.

Building for one vendor's toolchain

Every chipmaker has its own NPU architecture and compiler. Writing directly against one vendor's toolchain makes the product fragile on other hardware and ties your roadmap to theirs. Cross-platform runtimes and OS frameworks cost a little performance and save a lot of rework when the next hardware generation arrives.

Shipping without a fallback path

A mixed device fleet cannot be assumed to have a capable NPU. Products that only work on NPU hardware, or degrade badly without it, frustrate users on older machines. Designing a CPU or cloud fallback from the start is often more work than the NPU integration, which is exactly why it gets skipped.

NPU Best Practices for Builders and Businesses

If you build software that touches AI inference — a mobile app with on-device features, a desktop tool, an embedded product — NPUs change some of the calculus around where inference should run at all.

When on-device inference via NPU makes sense

• Features that need to work offline or with strict latency (real-time transcription, camera effects, voice commands).
• Workloads sensitive enough that sending data to a cloud API is a non-starter for privacy, compliance, or contractual reasons.
• High-frequency, low-complexity inference calls where cloud API costs would scale badly (keystroke-level prediction, continuous sensor processing).
• Products targeting markets with unreliable or expensive connectivity, where a cloud dependency is a real usability cost.

When it doesn't

• Large or frequently updated models, where the cost of shipping and versioning weights across a device fleet outweighs the latency win.
• Workloads that need the largest, most capable models — those still run meaningfully better on data-center GPUs than on any current NPU, and probably will for a while.
• Anything where the target device fleet is heterogeneous and can't be assumed to have a capable NPU — building a good CPU/cloud fallback path is often more work than the NPU integration itself.
• Early-stage products where engineering time is better spent validating the feature than optimizing where it runs.
• Features you cannot yet measure. If you have no task-specific benchmark for latency or quality, start in the cloud, instrument the feature, and move it on-device once you know what good looks like.

For a business evaluating whether "AI chip" hardware claims matter to a purchasing decision — buying laptops for a workforce, choosing embedded hardware for a product, evaluating a vendor's device roadmap — the practical filter is simple: does a specific feature you need depend on local inference, and has the vendor demonstrated that feature working on the hardware in question. Everything else in the spec sheet is a proxy for that answer, and often an unreliable one.

Limitations and Open Questions

The NPU story has real limits that the marketing tends to skip past.

Software fragmentation is still unresolved. Every major chipmaker has its own NPU architecture, its own compiler toolchain, and its own supported operator set. A model that runs well on one vendor's NPU may need re-tuning or partial CPU fallback on another's, even within the same operating system. Cross-platform frameworks are closing this gap, but it isn't closed.

The largest, most useful models still don't fit. Consumer NPUs are built for models in the hundreds of millions to low billions of parameters, run at reduced precision — the same size class covered in on-device AI. The models people actually find most capable — large general-purpose assistants — are typically far bigger and still run in data centers, meaning "on-device AI" today mostly means smaller, task-specific models handling narrow jobs, not a shrunk-down version of a frontier model.

Benchmarking is genuinely hard and often gamed. Because there's no single agreed-upon workload that represents "real AI usage" the way there is for, say, gaming frame rates, vendors have wide latitude to publish TOPS numbers or cherry-picked demo benchmarks that don't reflect what a buyer's actual software will do on the chip.

The category is still shaking out. It isn't settled whether NPUs remain a permanent fixture of every chip or whether GPU architectures absorb enough of the same efficiency gains to make a separate block unnecessary — GPU vendors have already started adding lower-precision, NPU-like execution paths of their own — part of a broader trend toward specialized silicon that also includes approaches like neuromorphic computing. Buying hardware today on the assumption that "NPU" is a stable, well-defined category for the next five years is a bet, not a certainty.

What to Watch Next

A few developments will determine whether NPUs settle into a durable category or get absorbed into GPUs and CPUs over time:

• Whether app and OS ecosystems mature enough that NPU usage becomes automatic and transparent, rather than something developers have to specifically target.
• Whether a standardized, trusted benchmark for on-device AI performance emerges, the way graphics benchmarking eventually standardized for GPUs.
• Whether memory bandwidth — often the actual bottleneck for on-device models — gets addressed as aggressively as raw compute has been, since a faster NPU feeding on a slow memory bus doesn't translate to a faster feature.
• Whether the industry converges on a shared way to describe and compare "AI chip" capability, replacing the current mix of incomparable TOPS figures.

If you're deciding how AI features in your own product should be split between on-device and cloud inference, Woyce Technologies can help work through that architecture.

FAQ

What does NPU stand for and what does it do?

NPU stands for neural processing unit. It's a chip or chip block designed specifically to run the matrix math inside neural networks — mainly the multiply-accumulate operations of an inference pass — more efficiently than a CPU or GPU can. In practice it handles features like live transcription, background blur, noise suppression, and on-device image or text models. Because it's built for one kind of arithmetic, it uses far less power for those tasks, which matters most on battery-powered laptops and phones.

Is an NPU better than a GPU for AI?

Not better in an absolute sense — better for a specific job. NPUs are more power-efficient for running (not training) smaller neural networks locally, while GPUs remain faster and more flexible for training and for running larger models, especially in data centers. On a laptop, the NPU suits always-on, low-power features, while the GPU handles heavier bursts such as image generation or running a larger local model. Many systems use both, and the operating system or app decides which one gets each task.

What is TOPS and why shouldn't I rely on it alone?

TOPS measures trillions of operations per second, a peak theoretical throughput figure usually quoted for a favorable numerical precision. It ignores memory bandwidth, software support, and precision compatibility, all of which determine whether a real model actually runs fast on the chip. Two devices with similar TOPS can feel very different in use. Look for task-specific benchmarks instead, such as tokens per second on a named model or latency for a feature you actually need.

Do I need an NPU to run AI software?

No. Any CPU or GPU can run AI inference; an NPU just does it more power-efficiently for supported model types. Without an NPU, the same features usually still work, just with higher battery or thermal cost, or by falling back to a cloud service. For most people today, an NPU is a nice-to-have rather than a requirement. It becomes more important if you rely on specific on-device features your operating system or apps only run locally on supported hardware.

What is an "AI PC" and is the label meaningful?

An AI PC is marketing shorthand for a computer with a CPU, GPU, and NPU combination meeting a vendor-defined performance threshold. The label tells you the hardware exists; it doesn't guarantee any specific software feature actually uses the NPU on that machine. Before paying a premium, check which features you'll use are confirmed to run locally on that exact model, and whether they're available in your region and operating system version.

Can NPUs run large language models like the ones behind chatbots?

Only smaller, quantized versions, and with real trade-offs in capability compared to the full-size models running in data centers. Current consumer NPUs target models in the hundreds of millions to low billions of parameters, not the largest general-purpose assistants. Small local models are useful for summarizing, drafting, and private tasks that don't need frontier-level reasoning. For the most capable models, most products still call a cloud service, with the NPU handling lighter work on the device.

Will NPUs become standard in all future chips?

That trend is already underway across most major laptop and phone chipmakers, though whether NPUs remain a distinct chip block long-term or get absorbed into more flexible GPU architectures is still an open question in the industry. For buyers, that means treating today's NPU as a useful efficiency feature rather than a long-term guarantee. For builders, it argues for using cross-platform runtimes so your models aren't tied to one vendor's NPU toolchain.

Conclusion

NPUs are real, useful silicon: blocks built to run neural network inference at a fraction of the power a CPU or GPU would spend on the same job. The confusion comes from how they're sold. A TOPS figure is a peak number at a favorable precision, and it says nothing about memory bandwidth, operator support, or whether your software ever routes work to the NPU.

The practical lens is feature-first. If you need offline, low-latency, or privacy-sensitive inference on battery-powered devices, a capable NPU can make a real difference, provided the vendor shows those exact features running on that exact hardware. If your workloads depend on large, frequently updated models, data-center GPUs and cloud APIs remain the better home, and an "AI PC" sticker changes little.

The caveats are fragmentation across vendor toolchains, benchmarks that are easy to game, and a category that may yet merge back into GPUs. Buy for what works today, not for a spec sheet's promise.

If you're building a product and deciding which AI features should run on the device and which in the cloud, our AI and machine learning team can help you map that split before you commit to hardware.