LLM推理量化:FP8 versus INT8 (English)

Generated: 2026-06-22 10:29:53

---

FP8 or INT8 for Quantization? After a Year of Trial and Error, I'm Laying It All Out Today

Let me start with a true story.

Two years ago, I was working on inference services, back when the A100 was still the hot commodity. I was messing around with Llama 2 70B, using INT8 with SmoothQuant—at the time, the papers made it sound amazing, claiming the accuracy loss was "negligible." I bought it.

The moment we went live, in code generation tasks, function calls kept breaking. Customers were furious; I was tearing my hair out.

It took me a whole week to figure out what happened—activation quantization had flattened some critical biases! Those tiny but deadly bits of information were just gone.

My heart sank.

What did I do next? I spent another week adjusting the calibration set distribution, barely managing to bring the loss back into an acceptable range. But you know the most gut-wrenching part? After switching to H100 and going straight to FP8, the same model got 2x throughput with essentially no quality loss.

That's when I truly understood—

Quantization has never been just an accuracy problem.

Hardware, algorithms, numerical formats—they're all locked together. You touch one, you have to adjust the other two.

Today, I'm going to lay out every pitfall I've hit and every conclusion I've verified over this past year.

---

1. The "Accuracy" You See Is Really Just a Different Ruler

How should I put this?

The core difference between FP8 and INT8—think of them as two different rulers.

INT8 is like a standard straight ruler: evenly spaced marks, step by step. Works fine for measuring a screw on your desk. But what if the ruler isn't long enough and you've got something really long? Snap—anything beyond the range gets chopped off.

FP8, on the other hand, is like a smart spring ruler—the markings are super dense near zero, really fine-grained; the farther out you go, the sparser the markings, but the range you can cover is much larger.

Which one is better?

It depends on the data distribution.

Let me break down the numbers:

• INT8: -128 to 127, 128 bins in total, evenly distributed.
• FP8 E4M3: Range ±448, 4-bit exponent, 3-bit mantissa. The exponent adjusts the scale automatically, making it naturally suited for distributions that are top-heavy with long tails.
• FP8 E5M2: Range ±57344! A massive range, but only 2 bits of mantissa—lower precision, better for storing gradients.

You see, each format has its own personality.

I ran an experiment, and listen to what happened—

I constructed a dataset where most values were concentrated between [-1, 1], but there were a few outliers as high as 100.

After INT8 quantization, because the scale factor got inflated by those outliers, almost all the small values turned into zeros! Information completely lost.

With the same data and FP8 E4M3 quantization, the outliers got compressed, sure, but the small values near zero still retained their relative relationships. They didn't get wiped out together.

You probably get it now: LLM activations are exactly like that—tiny scattered values carry tons of semantic information, while a few outliers can blow up the scale factor. FP8 feels custom-made for this scenario.

So FP8 is inherently less fussy than INT8.

Of course, that ease comes at a cost—the compute hardware path for FP8 is more complex, and you can't use it on A100. Only Hopper architectures from H100 onward have native FP8 Tensor Cores.

That's why anytime FP8 quantization comes up these days, it's tied to a hardware upgrade.

---

2. My "Lifesaving Recipe" from Hands-On Practice

When it comes to actual implementation, let me share what I do now.

For inference, I'm deploying a 70B model on H100 using W8A8 all-FP8 (weights and activations both FP8 E4M3), plus FP8 E5M2 for KV Cache.

I tested it on vLLM 0.6.0, and the results made me gasp—

Compared to the BF16 baseline, GPU memory was cut in half, and throughput increased by about 1.8x!

Think about it: same hardware, double the performance. Hard to resist.

And compared to using INT8 with SmoothQuant on A100 before, it's way easier—no need to prepare a calibration set, no tweaking scale factors. Just run Min-Max Calibration once. Going live feels like a rocket ride.

But—don't celebrate too soon!

Some layers are especially sensitive to quantization. I've fallen into these traps, so you need to remember them.

embedding and lm_head—I tested them twice. Switching them to FP8 dropped MMLU by 1.2 points. My heart was pounding. Later, I referred to Hugging Face's fp8 recipe and kept those two layers in BF16. The problem disappeared.

By the way, embedding layers have huge parameter counts but light computation. Keeping them high precision doesn't actually hurt performance.

I also experimented with training. Here's my recipe:

• Forward pass: Mostly BF16, using FP8 E4M3 only in the FFN and Attention computations (with delayed scaling—scale factor calculated from the max of the previous micro-batch).
• Backward pass: Gradients use FP8 E5M2. Why? Because gradient magnitudes can vary by several orders of magnitude—E5M2's wide range is better suited here.
• Optimizer state and gradient accumulation: Forced FP32. Skimp on precision here? That's a recipe for disaster—convergence becomes unstable.

With this recipe, I fine-tuned an 8B model. Compared to pure BF16, training speed improved by about 35% (on H100), and the validation loss difference was less than 0.01.

But honestly, if the model is very small—say, under 1B—FP8 might cause loss spikes due to quantization noise. I'd advise against it.

Oh, and I tried DeepSeek V3's MoE model too. The sparse routing part is more sensitive to quantization. You must keep the top-k routing layer weights in BF16, otherwise the gate scores drift.

---

3. Under Fine-Grained Quantization, Does INT8 Really Have No Chance?

Speaking of that, let me share a counterintuitive finding.

I came across a paper with a very interesting conclusion—when quantization granularity is refined to the block level (e.g., Microscaling format, block size=32), the authors found that MXINT8 could beat MXFP8 in some cases.

Wow! That doesn't sound right, does it?

Actually, the logic is simple—the smaller the block size, the more concentrated the local data distribution, reducing the crest factor (peak-to-average ratio). The disadvantage of INT's uniform scaling diminishes, and because it has higher mantissa precision, it actually gains an edge.

After reading that, I immediately replicated the experiment on my own model.

On LLaMA-3-8B, using per-block quantization with block size=32, INT8 matched FP8 in perplexity—even slightly better by 0.01 PPL.

Guess what?

But when it comes to deployment, concerns creep in. This format requires hardware and software to support blockwise scaling simultaneously. Currently, only NVIDIA's Blackwell architecture with native MX format can run it. On Hopper, purely software emulation actually slows things down instead of speeding them up.

So my assessment is:

On Hopper hardware, FP8 is the more pragmatic choice.

Once Blackwell becomes widespread, MXINT8/MXFP4 might turn the tables. But don't chase the new hotness now; wait until the framework matures before diving in.

---

4. Some Extra Lessons—All Bought with Blood and Tears

I often