For most of the last decade, if you wanted to run an AI model, you rented or bought an Nvidia GPU. That was the whole conversation. Training needed GPUs, inference needed GPUs, and the only real question was which generation you could get your hands on. That assumption is now visibly cracking. Nvidia's own move to acquire Groq's LPU technology for roughly $20 billion in December 2025 — absorbing rather than merely competing with a chip built specifically to beat GPUs at inference — is the clearest signal yet that the inference layer of AI computing has become a separate, contested market from training.
This matters because inference, not training, is where most AI compute dollars are actually spent once a model ships. Training happens once, in bursts, on a small number of massive clusters. Inference happens continuously, at scale, every time a user sends a prompt, an agent takes an action, or a background job classifies a document. As AI moves from research demos to always-on products, the economics of inference — cost per token, latency per response, watts per query — have become the thing that determines whether an AI feature is profitable or a permanent expense.
Why GPUs Were the Default in the First Place
GPUs became the AI industry's workhorse for a reason that had nothing to do with inference: they were built for parallel matrix math, which happens to be exactly what neural networks need. Nvidia spent years building CUDA, a software ecosystem so deep and so sticky that switching away from it carried real engineering cost even when better hardware existed on paper. That combination — good-enough architecture plus an unmatched software moat — let Nvidia capture the overwhelming majority of AI compute spend, training and inference alike.
But GPUs are general-purpose parallel processors. They were never designed exclusively for the specific, repetitive math pattern that transformer-based language models perform millions of times per second: attention calculations and matrix multiplications against a known, fixed model architecture. A chip designed only for that pattern can, in principle, do it faster and with less wasted energy than a chip designed to also handle graphics rendering, arbitrary parallel workloads, and everything else GPUs are asked to do.
That gap between "general-purpose parallel chip repurposed for AI" and "chip built only to run transformers" is exactly where the new wave of inference hardware lives.
It's worth remembering that CUDA lock-in was never really about raw silicon performance in the first place. It was about the years of accumulated tooling — compilers, debuggers, profiling tools, and a library ecosystem that every major AI framework was written to target. A startup could design a technically superior chip and still lose, because switching costs for developers were higher than the performance gain on offer. What's changed recently is that the performance gap between GPUs and specialized inference chips has grown large enough, in some workloads, that teams are willing to eat the switching cost anyway — which is precisely the condition that makes a challenger's technology worth acquiring rather than ignoring.
What Makes Inference Different From Training
Training and inference are often lumped together as "AI compute," but they stress hardware in almost opposite ways.
| Dimension | Training | Inference |
|---|---|---|
| Workload pattern | Massive batches, long-running jobs | Small batches, often single requests, continuous |
| Priority | Raw throughput across a cluster | Low latency per response |
| Precision needs | Higher precision for stable gradient updates | Lower precision often acceptable (e.g., int8, fp8) |
| Cost sensitivity | Amortized over model lifetime | Recurring, scales with usage/revenue |
| Hardware flexibility needed | High — architectures change often during R&D | Lower — production models are usually fixed |
| Ideal chip trait | Flexible, high memory bandwidth | Specialized, low power per operation |
That last row is the key insight driving the current hardware shift. Because a production inference workload runs a known, fixed model architecture over and over, a chip that hard-wires its design around that specific pattern — rather than staying flexible for whatever comes next — can strip out a lot of the overhead a general-purpose GPU carries. That's the bet behind two distinct classes of new hardware.
The Two Challengers: LPUs and Transformer ASICs
Language Processing Units (LPUs)
Groq's LPU architecture took a different approach to the memory bottleneck that slows down GPU inference. GPUs pair fast compute cores with external memory (HBM) that data has to travel to and from, and that round trip is often the actual speed limit on how fast a model can generate tokens — not the raw compute power. LPUs instead keep the entire model's working data on-chip in fast SRAM, avoiding that round trip almost entirely. The tradeoff is that on-chip memory is expensive and limited in capacity, so LPU systems typically need many chips networked together to hold a full-size model, and they've historically made more economic sense for inference than training.
The result, in Groq's public demonstrations, was inference speed — tokens generated per second — that visibly outpaced GPU-based serving for similar model sizes, at a time when generation speed had become a genuine user-facing differentiator for chat and agent products. Nvidia's decision to acquire that technology rather than simply out-engineer it around its own roadmap suggests the memory-bottleneck problem was harder to solve incrementally than to buy outright.
Transformer ASICs
A more radical bet comes from companies building Application-Specific Integrated Circuits (ASICs) that assume the transformer architecture itself won't change. Etched, a startup that came out of stealth as part of this same wave, is a visible example of this approach: instead of building a chip flexible enough to run any neural network architecture, it built silicon that runs transformer math and essentially nothing else.
The logic is a genuine gamble. If you strip out the flexibility a GPU needs to support arbitrary model architectures, you can pack more transformer-specific throughput into the same die area and power budget — but only for as long as transformers remain the dominant architecture. If the field moves to something structurally different (state-space models, alternative attention mechanisms, or an architecture not yet invented), a transformer-only ASIC becomes far less useful far faster than a general-purpose GPU would.
This is essentially the ASIC playbook that has played out in other corners of computing before. Bitcoin mining moved from general-purpose CPUs to GPUs to purpose-built ASICs as the underlying algorithm stabilized and the economic incentive to squeeze out every last bit of efficiency grew large enough to justify custom silicon. Video codecs followed a similar path, with dedicated hardware decoders eventually outperforming general-purpose processors running the same standard. The bet transformer ASIC makers are placing is that large language model architecture has stabilized enough, and inference volume has grown large enough, to justify the same move — building hardware around one specific, well-understood computational pattern rather than staying flexible for a pattern that hasn't been invented yet.
The counter-argument is that AI research has historically moved faster than hardware design cycles. A chip that takes eighteen months to tape out and ship is a bet that the dominant architecture won't meaningfully change in that window — a bet that would have looked reasonable at several points over the last few years, right before it wasn't.
Why This Is Happening Now
Nvidia's acquisition of Groq's LPU technology is worth sitting with, because it's an unusual move for a dominant incumbent to make. Companies with overwhelming market share more often try to out-build or out-price a challenger than pay a premium to absorb its core technology. That Nvidia chose the latter path, at a scale near $20 billion, points to a few things happening at once:
- Inference has become the larger, stickier revenue pool. Training spend is lumpy and concentrated among a handful of frontier labs; inference spend is distributed across every company that ships an AI feature, and it grows with usage rather than with model releases.
- The memory-bandwidth bottleneck is a real technical threat. If LPU-style architectures can reliably beat GPUs on inference latency and cost-per-token, that's not a marginal feature gap — it's a different value proposition for an entire category of buyer.
- The ASIC threat is credible enough to hedge against. Etched emerging from stealth alongside this deal signals that transformer-specific silicon isn't a one-off experiment; it's a category multiple teams are betting on, and CUDA's software moat matters less when a chip is purpose-built and comes with its own tightly scoped software stack.
- Owning inference hardware diversity protects the platform. By bringing LPU technology in-house rather than letting it become the default alternative to GPUs, Nvidia keeps optionality across chip architectures rather than defending a single design against a faster challenger.
For a company that has defined an era of computing, hedging against your own architecture is a notable signal about how contested the inference layer has become.
It's also worth noting what this deal doesn't settle. Acquiring a technology is not the same as successfully integrating it — Nvidia now has to decide whether LPU-style memory architecture becomes a distinct product line, gets folded into future GPU designs, or ends up shelved as a defensive move rather than a genuine product strategy. Large acquisitions of this kind have a mixed track record across the semiconductor industry, and the value of this one will depend heavily on execution over the next several product cycles, not on the announcement itself.
What This Means for Businesses Deploying AI
Most companies building AI products don't buy chips directly — they consume inference through an API or a cloud instance. But the hardware layer still shapes what shows up on their bill and in their latency metrics, and the choices are starting to multiply in ways worth tracking.
- Cost per token is becoming architecture-dependent, not just vendor-dependent. Two providers running the same open-weight model on different hardware can have meaningfully different cost and speed profiles. Evaluating an inference provider now sometimes means asking what it's running on, not just what model it serves.
- Latency-sensitive products have new options. Voice agents, real-time coding assistants, and interactive tools that were previously bottlenecked by token generation speed can now shop for hardware-backed speed guarantees rather than accepting whatever a general-purpose GPU cluster delivers.
- Vendor lock-in risk is shifting, not disappearing. CUDA lock-in was well understood and widely discussed. Committing to a transformer-only ASIC vendor, or an LPU-specific serving platform, introduces a different kind of lock-in — one tied to an architectural bet that may or may not hold up as models evolve.
- Multi-hardware strategies are becoming more common at the infrastructure layer. Cloud providers and inference platforms increasingly offer a choice of backend silicon for the same model, which pushes the hardware decision down to a configuration choice rather than a company-wide commitment.
- Capacity is no longer a single queue. When GPUs were the only viable inference hardware, GPU shortages meant AI shortages, full stop. A more diverse hardware market — even an imperfect, uneven one — gives buyers more paths to secure capacity when any single supply chain tightens.
For teams building on top of AI rather than selling infrastructure, the practical takeaway isn't "switch to an ASIC." It's that the inference layer is no longer a fixed cost with one shape — it's becoming a genuine set of tradeoffs between speed, cost, flexibility, and architectural risk that's worth revisiting periodically rather than assuming last year's default still holds.
Real Limitations and Open Questions
None of this displaces GPUs outright, and it's worth being clear-eyed about why.
- Software maturity still favors Nvidia. CUDA didn't become dominant by accident — it represents years of tooling, libraries, debugging support, and developer familiarity that alternative hardware ecosystems are still building out. A faster chip with a rougher software experience is a real cost, not a footnote.
- Flexibility has value that's easy to underprice. Model architectures still change. A company that commits heavily to transformer-only silicon is making a bet that the next architectural shift either won't happen soon or won't matter much for their use case — a bet that has been wrong before in AI's short history.
- Memory capacity, not just bandwidth, still constrains LPU-style designs. Keeping model weights on-chip is fast, but on-chip memory is expensive per gigabyte, which is part of why these systems tend to need many chips networked together for large models — a cost and complexity tradeoff that doesn't disappear just because latency improves.
- Consolidation could recreate the same concentration problem. If Nvidia absorbing Groq's technology means the most promising inference architecture ends up inside the same company that already dominates training hardware, the market could end up with more chip diversity in name but not much more in practice, unless independent challengers like transformer ASIC makers stay genuinely independent.
- Benchmarks from vendors deserve scrutiny. Tokens-per-second claims are usually measured under conditions favorable to the hardware being showcased. Real production workloads — with mixed request sizes, concurrent users, and varying prompt lengths — often look different from a clean demo.
What to Watch Next
The next year or two will likely clarify a few open threads: whether Nvidia integrates LPU-style memory architecture into its own GPU roadmap rather than keeping it as a separate product line, whether transformer ASIC vendors can build a software layer good enough to offset their architectural inflexibility, and whether any major model release forces a rethink of the transformer-only bet these ASICs are built on. Cloud providers' willingness to offer real hardware choice — rather than quietly defaulting everyone back to GPUs — will also be a useful signal of how much this diversity actually sticks.
FAQ
What is AI inference hardware, and how is it different from training hardware?
Inference hardware runs an already-trained AI model to generate responses to real user requests, while training hardware is used to build the model in the first place by processing huge datasets over many iterations. Inference workloads prioritize low latency and cost-efficiency per request, while training prioritizes raw throughput across massive, long-running jobs.
What is a Language Processing Unit (LPU)?
An LPU is a chip architecture, popularized by Groq, that keeps a model's data on fast on-chip memory instead of relying on external memory like GPUs do, which reduces a major latency bottleneck in generating tokens. This design tends to favor inference speed over the flexibility needed for training diverse model architectures.
What is a transformer ASIC, and why is it risky?
A transformer ASIC is a chip built specifically to run transformer-based model math and nothing else, trading general-purpose flexibility for higher efficiency on that one workload. The risk is architectural: if the industry moves away from transformers, that specialized hardware loses much of its advantage far faster than a general-purpose chip would.
Why did Nvidia acquire Groq's LPU technology instead of competing with it?
The roughly $20 billion deal in December 2025 suggests Nvidia saw LPU-style memory architecture as a credible threat to its inference dominance, and chose to absorb the technology rather than risk a competitor establishing it as the new default for inference workloads.
Does this mean GPUs are becoming obsolete for AI?
No. GPUs remain the default for training and for many general-purpose inference workloads, largely due to the maturity of the CUDA software ecosystem. Specialized chips are carving out advantages in specific inference scenarios, particularly where latency and cost per token are the dominant concerns, not replacing GPUs outright.
How should a business choose between GPU-based and specialized inference hardware?
It depends on the workload: latency-sensitive, high-volume, architecturally stable applications are where specialized hardware tends to show the clearest gains, while workloads that need flexibility or use less common model architectures are usually still better served by GPUs. Most teams don't need to choose a single hardware strategy — many inference platforms now let you select the backend per use case.
Will inference hardware diversity actually lower AI costs for businesses?
It has the potential to, since competition among chip architectures can push down the cost per token beyond what a single dominant vendor would offer. Whether that potential is realized depends on how much genuine independent competition survives consolidation moves like Nvidia's acquisition of Groq's technology.
Teams weighing where AI inference costs and latency actually come from in their own stack can get hands-on help from Woyce Technologies.