Low-Precision AI: What FP8, FP4, and INT4 Mean for Speed and Cost

A number that used to take 32 bits to store now takes 4. That single change — using fewer bits to represent the weights and activations inside a neural network — is quietly responsible for most of the recent gains in AI inference speed and cost. It is not a new model architecture, a smarter training algorithm, or a bigger cluster. It is arithmetic getting coarser on purpose, and it works because neural networks turn out to be remarkably tolerant of imprecision.

Low-precision computing — running models in FP8, FP4, or INT4 instead of the FP32 or FP16 formats they were historically trained and served in — has moved from a research curiosity to the default assumption baked into new AI silicon. Understanding what these formats actually are, and where they help and hurt, matters for anyone deciding what hardware to buy, what model size to deploy, or how to budget for inference at scale.

What "precision" means in a neural network

Every number inside a neural network — the weights that define what the model learned, and the activations that flow through it during a forward pass — has to be stored and manipulated in some numeric format. That format has a fixed number of bits, and those bits get split between representing the sign, the exponent (how big or small the number can get), and the mantissa (how many significant digits of precision you get).

The formats in common use today:

The pattern across the table is simple: fewer bits means less memory per number, less data to move, and less energy per multiply-accumulate operation — the core arithmetic step that GPUs and NPUs spend nearly all their time on. It also means less precision, which is the whole tension this article is about.

Why fewer bits translates directly into speed and cost

AI accelerators are not primarily bottlenecked by how fast they can do arithmetic — they are bottlenecked by how fast they can move data from memory to the compute units. This is usually called being "memory-bandwidth bound." A model's weights have to travel from high-bandwidth memory into the chip's compute cores for every token generated. If those weights are stored in FP4 instead of FP16, there is a quarter as much data to move, and a quarter as much memory needed to hold them.

The effect compounds three ways:

1. Memory footprint drops, so a model that needed multiple high-end GPUs to fit in memory at FP16 might fit on a single GPU at FP4 or INT4 — or a large model might fit on a laptop or phone at all.
2. Data movement drops, which directly increases achievable throughput on memory-bound workloads like token generation.
3. Raw compute throughput often increases too, because chip designers can pack more low-precision arithmetic units into the same silicon area than high-precision ones, and newer accelerators have dedicated low-precision math units that run at multiples of their FP16 throughput.

None of this is free. Coarser number representations mean each individual value is a rougher approximation of what the model "really" computed, and that approximation error can accumulate in ways that hurt output quality if not handled carefully.

FP8 vs. FP4 vs. INT4: what actually differs

These three formats get lumped together as "low precision," but they solve different problems and carry different risk profiles.

FP8

FP8 is now a mainstream training and inference format, not just an inference shortcut. It keeps a floating-point structure (sign, exponent, mantissa) but squeezes it into 8 bits, typically split as either 4 exponent bits and 3 mantissa bits, or 5 exponent bits and 2 mantissa bits, depending on whether you want more dynamic range or more precision. Because it retains a floating-point exponent, FP8 handles the very large and very small values that show up during training (gradients, in particular) more gracefully than a fixed-point integer format would. This is why FP8 has become viable not just for running trained models but for training them in the first place, with careful per-tensor or per-block scaling to keep values inside a usable range.

FP4

FP4 pushes the same floating-point idea down to 4 bits — usually something like 1 sign bit, 2 exponent bits, and 1 mantissa bit, though exact layouts vary by vendor. At this bit width there are only a handful of representable values, so FP4 is essentially unusable without aggressive per-block scaling: instead of one scale factor for an entire tensor, the values are grouped into small blocks (say, 16 or 32 values at a time), each with its own scale. This lets a block of small numbers and a block of large numbers both be represented reasonably accurately in FP4, even though the format itself can only express a tiny set of distinct values. FP4 is, as of the current generation of hardware, primarily an inference format — training directly in FP4 remains an active research problem rather than a settled practice.

INT4

INT4 is a fixed-point integer format: 4 bits representing a plain integer (0–15, or -8 to 7 signed), with a separate scale and zero-point value used to map those integers back to the real range of numbers the model needs. INT4 has no exponent, so it cannot natively represent a wide dynamic range the way FP4 can — it depends even more heavily on good calibration (choosing the right scale and zero-point per channel or per block) to avoid clipping outlier values or wasting precision on a range the data never uses. INT4 has been around longer than FP4 in practical deployment, especially on mobile and edge neural processing units (NPUs), because integer arithmetic units are simpler and cheaper to build into silicon than floating-point ones.

Quick comparison

Why this matters right now

Precision has always been a knob researchers could turn, but for most of deep learning's history it wasn't turned very far — FP16 and BF16 covered the vast majority of training, and INT8 covered a modest slice of inference optimization. What has changed is that the hardware itself has shifted to make ultra-low precision the default path rather than an edge case.

The 2026 generation of flagship AI accelerators is built around FP4 as a first-class format, with dedicated tensor cores designed to run FP4 matrix multiplication at throughput multiples well beyond what the same silicon achieves in FP8 or FP16. That is a deliberate design choice by chipmakers, not an incidental capability — it reflects a bet that the industry's appetite for larger models and more inference volume will keep outpacing the growth in raw silicon capacity, and that precision reduction is the most reliable lever left for closing that gap. At the same time, edge NPUs — the low-power AI chips built into phones, laptops, and other on-device hardware — have moved to ship hardware INT4 support as standard, not as a niche feature reserved for specialized SKUs.

That combination — flagship datacenter silicon optimized for FP4 and consumer edge silicon optimized for INT4 — means low-precision inference is no longer something only a handful of frontier labs use for cost-cutting at massive scale. It is becoming the default execution path across the stack, from the largest cloud deployments down to the model running locally on a phone.

Practical implications for businesses and builders

If you deploy or build on top of AI models, precision choices show up as concrete line items: GPU hours, memory footprint, latency, and the size of model you can practically ship.

Where low precision pays off most directly

• Inference cost per token. Serving a large language model is dominated by memory bandwidth during token generation. Moving from FP16 to FP8 or FP4 weights can meaningfully cut the cost per million tokens served, because the same GPU can hold a bigger batch in memory and move less data per step.
• Fitting bigger models on smaller hardware. A model quantized to INT4 or FP4 might fit on a single GPU or even a high-end laptop where the FP16 version required a multi-GPU server. This changes the calculus for teams that want to self-host rather than call an API.
• On-device and edge deployment. INT4 support in edge NPUs is what makes it feasible to run a meaningfully capable model directly on a phone or laptop with acceptable battery life, rather than round-tripping every request to a cloud server.
• Throughput under fixed hardware budgets. For teams with a fixed GPU allocation, lower precision often means more requests served per second on the same hardware, which is a direct lever on both cost and latency for user-facing products.

Where the trade-offs bite

Precision reduction is not a free upgrade, and treating it as one is the most common mistake in production deployments.

1. Accuracy degradation is task-dependent, not uniform. A model quantized to INT4 might show negligible quality loss on casual conversation but measurable degradation on tasks that require precise numerical reasoning, code generation, or long multi-step logic — the exact tasks where errors compound.
2. Quantization is not "flip a setting." Getting good results at FP4 or INT4 typically requires calibration against representative data, careful choice of block sizes for scaling, and sometimes quantization-aware fine-tuning rather than simply casting weights down after training.
3. Not all layers should be quantized equally. Production quantization pipelines often keep certain sensitive layers (embeddings, final output layers, or specific attention components) at higher precision while quantizing the bulk of the model, because uniform quantization across every layer tends to concentrate error in the places that matter most.
4. Hardware support is uneven. FP4 tensor core support exists on the newest flagship accelerators but not on older GPU generations still in wide production use, meaning the cost benefits are only available to teams that have upgraded or are renting the newest hardware tier.
5. Benchmarks can mislead. A quantized model can look fine on standard benchmark suites while quietly underperforming on a business's specific use case, because benchmark tasks rarely stress the exact failure modes low precision introduces.

A practical decision framework

For teams deciding how far to push precision reduction, the tradeoff generally maps like this:

How quantization actually gets done

There are two broad approaches, and the choice between them is mostly about how much compute and data you're willing to spend to preserve quality.

Post-training quantization (PTQ) takes an already-trained model and converts its weights to a lower-precision format after the fact, usually using a small calibration dataset to determine good scale factors per layer or per block. This is cheap and fast — often a matter of minutes to hours — which is why it's the default path for most teams quantizing an off-the-shelf model.

Quantization-aware training (QAT) simulates low-precision arithmetic during training or fine-tuning itself, so the model's weights adapt to the rounding and clipping effects it will experience at inference time. This produces better accuracy at very low bit widths (like INT4 or FP4) than PTQ typically achieves, but it costs substantially more compute and requires access to training infrastructure and representative data, which many teams deploying third-party models simply don't have.

Between these two extremes sit a range of lighter-weight techniques — outlier-aware quantization that keeps a small number of unusually large values at higher precision while quantizing the rest, and mixed-precision schemes that vary bit width by layer based on measured sensitivity. Most production quantization today is some blend of these ideas rather than a pure PTQ-versus-QAT choice.

Limitations and open questions

Low-precision AI is genuinely useful, but it is not a solved problem, and several open questions remain live:

• There is no universal "safe" bit width. What counts as acceptable quality loss depends entirely on the application — a customer support chatbot can tolerate more approximation error than a system generating financial calculations or medical dosage information.
• Training in FP4 is still immature. While FP4 inference is well-supported on new hardware, training directly in FP4 remains difficult because gradients are far more sensitive to precision loss than trained weights are; most FP4 deployments today are quantized after training in a higher-precision format.
• Evaluation is harder than it looks. Measuring the real-world impact of quantization requires task-specific evaluation, not just aggregate benchmark scores, and many organizations lack the evaluation infrastructure to catch subtle degradation before it reaches users.
• Hardware and software support are still catching up to each other. New precision formats often arrive in hardware before software frameworks fully support them, and vice versa, creating a lag where the theoretical benefits of a format like FP4 aren't fully realized in production toolchains yet.
• Precision reduction interacts with other optimizations. Techniques like speculative decoding, key-value cache compression, and mixture-of-experts routing all interact with quantization in ways that are not yet fully standardized, making it hard to predict combined effects without direct testing.

What to watch next

A few developments are worth tracking if you want to stay ahead of where low-precision AI is headed:

1. Whether FP4 training moves from research to practice. If reliable FP4 training methods mature, the cost of training frontier models could drop meaningfully, not just the cost of serving them.
2. Software framework support catching up to hardware. Watch whether mainstream inference frameworks make FP4 and INT4 deployment as close to a default, low-effort setting as FP16 is today, rather than requiring specialized tooling.
3. Standardization of block-scaling schemes. Different vendors currently use different block sizes and scaling conventions for FP4 and INT4, which fragments tooling; convergence on shared standards would make cross-hardware deployment easier.
4. Sub-4-bit formats. Research into 2-bit and 3-bit quantization is active, and while quality loss is currently steep at those widths, the same forces pushing FP8 to FP4 could eventually push further.
5. Edge NPU capability growth. As more devices ship hardware INT4 (and eventually FP4) support, expect more capable models to run fully on-device, shifting some workloads away from the cloud entirely.

FAQ

What is the difference between FP8 and INT4?

FP8 is a floating-point format with a sign, exponent, and mantissa, which lets it represent a wide range of magnitudes reasonably well. INT4 is a fixed-point integer format with no exponent, so it depends much more heavily on calibrated scale and zero-point values to map its limited set of integers back onto the real range of numbers a model needs.

Does quantization hurt model accuracy?

It can, and the amount of degradation depends on the bit width, the technique used, and the task. FP8 typically shows minimal accuracy loss compared to FP16, while INT4 and FP4 require more careful calibration or quantization-aware training to avoid noticeable degradation, especially on numerically sensitive tasks like math or code generation.

Why are chipmakers building hardware specifically for FP4?

Because inference at scale is largely bottlenecked by memory bandwidth and by how many low-precision operations a chip can perform per second, and FP4 lets more values be moved and computed in the same amount of memory and silicon area than higher-precision formats. Building dedicated FP4 tensor cores lets flagship hardware serve more requests per dollar of hardware and power.

Can you train a model directly in FP4?

Training directly in FP4 is still an active research area rather than standard practice, because gradients are more sensitive to precision loss than already-trained weights. Most FP4 deployments today take a model trained in a higher-precision format like BF16 or FP8 and quantize it down for inference.

What is the difference between post-training quantization and quantization-aware training?

Post-training quantization (PTQ) converts an already-trained model to lower precision using a calibration dataset, and is fast but can lose more accuracy at very low bit widths. Quantization-aware training (QAT) simulates low-precision effects during training or fine-tuning itself, producing better accuracy at formats like INT4 or FP4, at the cost of significantly more compute and data requirements.

Is INT4 only used on edge devices?

No — INT4 is common on edge NPUs because integer arithmetic is cheaper to implement in low-power silicon, but it is also used in datacenter inference when teams want maximum cost reduction and can tolerate or mitigate the accuracy trade-offs through careful calibration.

How do I know if my use case can tolerate FP4 or INT4?

The safest approach is task-specific evaluation on your actual workload rather than relying on general benchmark scores, since quality loss from low precision is highly dependent on the specific task. Numerically sensitive applications like financial calculations, code generation, or multi-step reasoning tend to be more affected than conversational or classification tasks.

Teams weighing precision trade-offs against real cost and latency budgets can get a second opinion from Woyce Technologies.