NPUs and AI Chips: A Buyer's Guide to the New Hardware

Open a laptop spec sheet today and you'll find a new line item sitting next to the CPU and GPU: an NPU, quoted in TOPS, with a marketing sticker calling the machine an "AI PC." Phone makers have run the same play for longer — every new flagship chip announcement leads with its neural engine's throughput number. Almost none of that number tells you what you actually need to know, which is whether the chip does anything useful for the software you run.

This guide cuts through the spec-sheet noise. It explains what an NPU actually is, how it differs from the CPU and GPU already in your machine, why every hardware vendor suddenly wants one, where the real limitations sit, and what questions to ask before treating "NPU inside" as a reason to buy.

What an NPU Actually Is

A neural processing unit is a chip — or more often a block of circuitry embedded alongside a CPU and GPU on the same die — designed to do one narrow class of math very efficiently: the matrix multiplications and additions that make up a neural network's forward pass. That's it. It isn't a general-purpose processor, and it isn't magic; it's a piece of silicon shaped around one repeated arithmetic pattern.

To see why that shape matters, it helps to compare the three processors that now typically ship together in a single device:

A CPU executes instructions one after another (with some pipelining and parallel cores), which makes it excellent at logic, branching, and anything unpredictable, but inefficient at repeating the same simple calculation across millions of data points. A GPU flips that trade-off: thousands of simple cores doing the same operation on different pieces of data simultaneously, which is exactly what both 3D rendering and neural network math require. GPUs became the default AI training and inference chip almost by accident — they already existed for graphics, and it turned out matrix multiplication for rendering triangles and matrix multiplication for neural networks are close enough that the same hardware serves both.

An NPU narrows that specialization further. It's built specifically for the operations inside a trained neural network — mostly multiply-accumulate operations at reduced numerical precision (8-bit or even 4-bit integers instead of the 32-bit floats a CPU favors) — and nothing else. Because it does less, it can do that one thing with far less energy per operation than a CPU or GPU spends on the same math. That efficiency, not raw speed, is the entire reason NPUs exist.

Where NPUs sit in the compute stack

An NPU virtually never runs a model on its own. A typical inference pass on an NPU-equipped device still routes through the CPU for orchestration, memory management, and pre/post-processing, with the NPU called in specifically for the heavy matrix math in the middle. So "the NPU ran it" is usually shorthand for "the NPU did the expensive part of the job while the CPU handled everything else around it."

The physical layout reflects this division of labor. On most modern phone and laptop chips, the NPU isn't a separate card or module the way a discrete GPU is — it's one block among several on the same piece of silicon as the CPU cores, GPU cores, image signal processor, and memory controller, all sharing the same pool of on-chip memory. That shared-memory design is deliberate: moving data between separate chips (as happens between a CPU and a discrete GPU over a PCIe bus) costs time and power, and a big share of what makes an NPU efficient for small, frequent inference calls is that it doesn't have to pay that data-movement tax. The trade-off is that an integrated NPU can't be swapped or upgraded independently — it ships with the rest of the chip and its capabilities are fixed for the life of the device.

How inference actually gets scheduled

None of this is visible to an end user, and most of it isn't visible to app developers either. A phone's operating system typically owns the decision about where a given AI task runs, based on rules that weigh the model's size, the precision it needs, current battery and thermal state, and what else is competing for the NPU at that moment. Two identical phones running the identical app can route the same feature to different processors depending on battery level or background load. That's a deliberate design choice — it lets the OS optimize for battery life and responsiveness dynamically — but it also means a spec sheet claim about NPU capability describes a ceiling the software may or may not actually reach in everyday use.

Why Every Chipmaker Added One

The wave of NPU announcements from 2023 onward — in phone SoCs, then laptop chips, then desktop processors — traces back to one shift: AI features moved from something a cloud server did on your behalf to something users expect to run locally, instantly, and without a network round trip. Autocomplete suggestions, live transcription, background blur in video calls, on-device image search, voice assistants that work offline — all of these are neural network inference jobs that a device now has to do dozens of times a minute, and doing them on a GPU or CPU burns battery fast enough to be noticeable.

The push toward local inference comes from a handful of converging pressures:

• Battery life. Running inference on a CPU or GPU for always-on features (wake-word detection, camera effects, keyboard prediction) drains a laptop or phone battery far faster than routing the same work through a purpose-built low-power block.
• Latency. A round trip to a cloud model adds tens to hundreds of milliseconds and depends on a network connection; local inference on an NPU responds immediately and works offline.
• Privacy and data residence. Keeping voice, camera, and personal-file processing on-device avoids sending that data to a third-party server, which matters both for user trust and for regulated industries.
• Cost at scale. Every inference call a device handles locally is one a company doesn't pay a cloud GPU bill to serve, which matters enormously once a feature runs on hundreds of millions of devices.

Operating system vendors then built the software layer to make this practical — frameworks that let an app request "run this model" without the developer needing to write chip-specific code, with the OS deciding at runtime whether to route the job to the NPU, GPU, or CPU depending on what's free and what the model needs. That software layer is arguably as important as the silicon: an NPU sitting idle because no app knows how to call it is just a wasted chunk of die space, and for the first couple of years of the "AI PC" push, that was closer to reality than the marketing suggested.

What the TOPS Number Does and Doesn't Tell You

TOPS — trillions of operations per second — is the headline spec on every NPU announcement, and it is the single most misused number in this entire category. It measures peak theoretical throughput for a specific, favorable operation type (usually 8-bit integer multiply-accumulates) under ideal conditions. It does not measure:

1. Real-world model performance. A model that needs more memory bandwidth than the NPU can supply will bottleneck long before it hits peak TOPS, regardless of what the spec sheet says.
2. Precision compatibility. A TOPS figure quoted at INT8 doesn't transfer to a model that needs FP16 precision to stay accurate; some NPUs handle the conversion well, others lose real quality doing it.
3. Software support. A chip with excellent TOPS but immature drivers or a limited operator set (the specific neural network layer types it can execute) may fall back to the CPU for large parts of a model it doesn't fully support, erasing most of the theoretical advantage.
4. Which models actually run there. OS and app vendors decide, often opaquely, which specific features get routed to the NPU versus the GPU or cloud. A high-TOPS chip is irrelevant to a task the software never sends its way.

Comparing two chips by TOPS alone is a bit like comparing two cars by peak horsepower without checking whether either one's transmission can actually put that power to the road under normal driving. It's a useful ceiling, not a guarantee.

Memory bandwidth deserves particular attention because it's the constraint TOPS figures hide most effectively. Running a neural network means repeatedly loading its weights from memory into the compute units, and for many real models — especially language models generating one token at a time — that data movement, not the arithmetic itself, is the slow part. A chip can advertise enormous peak TOPS and still deliver disappointing real-world speed if its memory system can't keep the compute units fed. This is why two devices with similar TOPS ratings can produce noticeably different real-world speeds on the same task, and why independent, task-specific benchmarks matter more than the headline spec.

A more useful way to evaluate a chip

For anyone actually deciding what to buy or spec for a project, TOPS is a filter, not a decision criterion. More productive questions:

• What specific features (transcription, translation, image generation, code completion) is this device advertised to run locally, and can you test them before buying?
• Does the manufacturer publish which model sizes and precisions the NPU actually supports, or only a peak TOPS figure?
• Is there independent benchmark data for real workloads — tokens per second on a specific model, latency for a specific task — rather than synthetic microbenchmarks?
• How much unified/shared memory does the device have, since large models are often memory-bound before they're compute-bound?

Practical Implications for Builders and Businesses

If you build software that touches AI inference — a mobile app with on-device features, a desktop tool, an embedded product — NPUs change some of the calculus around where inference should run at all.

When on-device inference via NPU makes sense

• Features that need to work offline or with strict latency (real-time transcription, camera effects, voice commands).
• Workloads sensitive enough that sending data to a cloud API is a non-starter for privacy, compliance, or contractual reasons.
• High-frequency, low-complexity inference calls where cloud API costs would scale badly (keystroke-level prediction, continuous sensor processing).
• Products targeting markets with unreliable or expensive connectivity, where a cloud dependency is a real usability cost.

When it doesn't

• Large or frequently updated models, where the cost of shipping and versioning weights across a device fleet outweighs the latency win.
• Workloads that need the largest, most capable models — those still run meaningfully better on data-center GPUs than on any current NPU, and probably will for a while.
• Anything where the target device fleet is heterogeneous and can't be assumed to have a capable NPU — building a good CPU/cloud fallback path is often more work than the NPU integration itself.
• Early-stage products where engineering time is better spent validating the feature than optimizing where it runs.

For a business evaluating whether "AI chip" hardware claims matter to a purchasing decision — buying laptops for a workforce, choosing embedded hardware for a product, evaluating a vendor's device roadmap — the practical filter is simple: does a specific feature you need depend on local inference, and has the vendor demonstrated that feature working on the hardware in question. Everything else in the spec sheet is a proxy for that answer, and often an unreliable one.

Limitations and Open Questions

The NPU story has real limits that the marketing tends to skip past.

Software fragmentation is still unresolved. Every major chipmaker has its own NPU architecture, its own compiler toolchain, and its own supported operator set. A model that runs well on one vendor's NPU may need re-tuning or partial CPU fallback on another's, even within the same operating system. Cross-platform frameworks are closing this gap, but it isn't closed.

The largest, most useful models still don't fit. Consumer NPUs are built for models in the hundreds of millions to low billions of parameters, run at reduced precision. The models people actually find most capable — large general-purpose assistants — are typically far bigger and still run in data centers, meaning "on-device AI" today mostly means smaller, task-specific models handling narrow jobs, not a shrunk-down version of a frontier model.

Benchmarking is genuinely hard and often gamed. Because there's no single agreed-upon workload that represents "real AI usage" the way there is for, say, gaming frame rates, vendors have wide latitude to publish TOPS numbers or cherry-picked demo benchmarks that don't reflect what a buyer's actual software will do on the chip.

The category is still shaking out. It isn't settled whether NPUs remain a permanent fixture of every chip or whether GPU architectures absorb enough of the same efficiency gains to make a separate block unnecessary — GPU vendors have already started adding lower-precision, NPU-like execution paths of their own. Buying hardware today on the assumption that "NPU" is a stable, well-defined category for the next five years is a bet, not a certainty.

What to Watch Next

A few developments will determine whether NPUs settle into a durable category or get absorbed into GPUs and CPUs over time:

• Whether app and OS ecosystems mature enough that NPU usage becomes automatic and transparent, rather than something developers have to specifically target.
• Whether a standardized, trusted benchmark for on-device AI performance emerges, the way graphics benchmarking eventually standardized for GPUs.
• Whether memory bandwidth — often the actual bottleneck for on-device models — gets addressed as aggressively as raw compute has been, since a faster NPU feeding on a slow memory bus doesn't translate to a faster feature.
• Whether the industry converges on a shared way to describe and compare "AI chip" capability, replacing the current mix of incomparable TOPS figures.

FAQ

What does NPU stand for and what does it do?

NPU stands for neural processing unit. It's a chip or chip block designed specifically to run the matrix math inside neural networks — mainly the multiply-accumulate operations of an inference pass — more efficiently than a CPU or GPU can.

Is an NPU better than a GPU for AI?

Not better in an absolute sense — better for a specific job. NPUs are more power-efficient for running (not training) smaller neural networks locally, while GPUs remain faster and more flexible for training and for running larger models, especially in data centers.

What is TOPS and why shouldn't I rely on it alone?

TOPS measures trillions of operations per second, a peak theoretical throughput figure usually quoted for a favorable numerical precision. It ignores memory bandwidth, software support, and precision compatibility, all of which determine whether a real model actually runs fast on the chip.

Do I need an NPU to run AI software?

No. Any CPU or GPU can run AI inference; an NPU just does it more power-efficiently for supported model types. Without an NPU, the same features usually still work, just with higher battery or thermal cost, or by falling back to a cloud service.

What is an "AI PC" and is the label meaningful?

An AI PC is marketing shorthand for a computer with a CPU, GPU, and NPU combination meeting a vendor-defined performance threshold. The label tells you the hardware exists; it doesn't guarantee any specific software feature actually uses the NPU on that machine.

Can NPUs run large language models like the ones behind chatbots?

Only smaller, quantized versions, and with real trade-offs in capability compared to the full-size models running in data centers. Current consumer NPUs target models in the hundreds of millions to low billions of parameters, not the largest general-purpose assistants.

Will NPUs become standard in all future chips?

That trend is already underway across most major laptop and phone chipmakers, though whether NPUs remain a distinct chip block long-term or get absorbed into more flexible GPU architectures is still an open question in the industry.

If you're deciding how AI features in your own product should be split between on-device and cloud inference, Woyce Technologies can help work through that architecture.