An A100 does INT8 matrix math at twice the rate of FP16 and reads half the bytes to do it, so a lossless INT8 path for both operands would roughly double serving throughput for free — the arithmetic units are already sitting there.
Weight-only quantization (GPTQ, the previous page in this track) shrinks the bytes but still computes in FP16: the integers are unpacked back to floats before the matmul. W8A8 is the other prize — quantize the activations too, and the multiply itself runs in INT8. The catch is in the activations, and it is the exact tensor the foundation page raised into three axes.
Under one scale for the whole tensor, a channel whose maximum is m_i inside a tensor whose maximum is m receives only 2⁸·m_i/m effective levels — and because a few channels are tens of times louder than the rest, the ordinary ones are left quantizing into single digits.
This is the sharpest sentence in the paper (§3), and it is not a benchmark — it is arithmetic you can run on any tensor. Below it is run on a real Qwen3-8B layer, over a real calibration set. INT8 promises 256 levels; the median channel is actually working with about nine of them.
Effective levels per input channel, 2⁸·m_i/m, measured on Qwen3-8B with 2,048 tokens of WikiText-2.
Every input channel of the layer is on the horizontal axis, sorted from the most starved to the loudest. The loud channel sitting at the far right is the one setting the scale for everyone; the flat stretch on the left is what that costs. Drop to INT6 or INT4 and the whole curve slides down by the same factor — the tax is multiplicative, so it is worst exactly where you most want to go.
A scale per input channel would fix the tax outright — the paper measures it, and it recovers full accuracy — but an INT8 GEMM can only apply scales along the outer dimensions, because a per-input-channel scale would have to be applied inside the reduction, between Tensor Core instructions.
So the field went sideways instead. LLM.int8() keeps the outlier channels in FP16 and runs a second, mixed-precision matmul for them. ZeroQuant quantizes per token, which is an outer dimension and therefore legal — but the variance that hurts is across channels within a token, so per-token barely helps. Our own measurement below says the same thing the paper's Table 1 does.
Left: relative output error of the layer under each scheme, ours. Right: what the paper measured end to end, zero-shot accuracy on OPT.
Per-token is on an outer dimension and legal; it lands nearly on top of per-tensor. Per-channel is a different order of magnitude — and it is the one that cannot be expressed inside the reduction. That gap, between what works and what the kernel allows, is the entire opening SmoothQuant walks into.
One output tile of an INT8 GEMM, and the two places a scale can be applied without breaking the MMA sequence.
A scale on the token axis or the output-channel axis multiplies the accumulator after the reduction finishes — one FP16 multiply per output element. A scale on the input-channel axis would have to land between the MMAs, inside the K-loop, where the Tensor Core pipeline has no room for it. The constraint is about where a number can be applied, not how expensive it is.
No mixed precision, no retraining, no per-channel scale at runtime — the output of the quantized layer should be computed by a plain INT8 GEMM with scales only on the outer dimensions, and the accuracy should hold at 175B.
Since Y = (X·diag(s)⁻¹)(diag(s)·W) is exactly XW for any positive s, the per-channel scale can be applied before inference and folded into the previous layer's parameters — the runtime kernel never sees it.
This is the whole trick, and it is worth being clear about why it is not cheating. Nothing is thrown away and nothing is approximated: the transform is an identity on the product. What changes is only how the two operands are shaped before they meet the quantizer. Difficulty is being moved, not removed.
Four input channels of Qwen3-8B layer 23, with their real maxima and the scale the method assigns them.
The middle column is the only thing the method computes. Watch the right-hand pair: whatever α does to the activation maximum, the weight maximum moves the opposite way by exactly the reciprocal, and their product — the thing the layer actually outputs — never moves at all.
s_j = max|X_j|α / max|W_j|1−α — at α = 0 the weights are flattened and the activations absorb everything, at α = 1 it is the other way round, and both endpoints quantize badly, so the method is the interior point.
The paper reports α = 0.5 as a good default for OPT and BLOOM, and 0.75 for GLM-130B, whose activations are more extreme. On the two Qwen3-8B layers below the optimum lands at α = 0.4 and 0.6 — near the default, and different between layers, which is the honest version of the story.
The same two operands, drawn as surfaces over the same 96 input channels. One knob moves both.
α = 0 is not "leave it alone" — it is the first extreme. s_j = 1/max|W_j| flattens the weights perfectly and hands every bit of the difficulty to the activations, which is why the ridge at α = 0 is taller than the untouched number in the sub-label. Push α to the other end and the two panels trade places. Neither surface is ever improved for free: one is paid for with the other, and the figures above the plots say the exchange rate.
Relative output error of the whole layer under W8A8 per-tensor quantization, swept over α.
Both ends are bad for opposite reasons and the curve has one interior minimum — which is what makes α a real hyperparameter rather than a knob you can leave at an extreme. The two layers do not agree on where the minimum is, which is the same observation that made the authors raise α for GLM-130B.
Seven zero-shot tasks and one language-modelling benchmark, against four INT8 baselines, at three efficiency levels O1–O3, ending with a 530B model served on eight GPUs instead of sixteen.
The O1/O2/O3 ladder is worth noticing: O1 keeps per-token dynamic activation scales, O2 goes per-tensor dynamic, O3 goes per-tensor static — no calibration at runtime at all. The claim is that smoothing makes even the cheapest, least-adaptive setting work, which is a stronger statement than any single accuracy number.
On OPT-175B, plain INT8 per-tensor falls from 71.6% to 32.3% — chance-level — while per-channel would have held 71.4%; SmoothQuant-O3 recovers that without ever using a per-channel scale at runtime.
Their Table 3, charted. We did not re-run OPT-175B — re-running a headline is not understanding.
Three methods of that generation do not merely lose accuracy — they take perplexity from 10.99 into the tens of thousands, which is a model that has stopped being a model. LLM.int8() survives by keeping the outliers in FP16 and paying for a mixed-precision matmul. SmoothQuant survives at O3, the setting with no runtime calibration at all, which is the strongest form of the claim: the tensor was fixed beforehand, so the cheapest possible kernel is enough.
Their Table 8 — OPT-30B decoding on one GPU, batch 1.
Halving the bytes is the point at batch 1, where decoding is memory-bound: the step gets faster and the footprint halves at the same time. That second number is what turns a 175B model from eight GPUs into four, and it is why an accuracy result with no speed result would not have been interesting.
LLM.int8() keeps outliers in FP16 and pays with a mixed-precision matmul; ZeroQuant and Outlier Suppression stay on legal axes and therefore do not address the channel variance; SmoothQuant changes the tensor instead of the kernel.
The comparison with GPTQ is not a competition — they occupy different regimes. GPTQ is weight-only, W4A16, and computes in FP16; SmoothQuant is W8A8 and computes in INT8. The next page in this track, AWQ, goes back to weight-only, and does it with the same per-channel scale you just watched — aimed at a different target.
Before this paper, activation outliers were an obstacle to be detected and routed around; after it, they are a quantity that can be moved between the two operands by a number you choose.
α is a hyperparameter set per model family rather than derived — 0.5 for OPT and BLOOM, 0.75 for GLM-130B — and the scales come from a calibration set, so a distribution shift at serving time is not something the method can see.
Our sweep makes the same point at layer granularity: the optimum differs between two layers of the same model. A per-layer search is cheap and obvious in hindsight — and it is exactly the step AWQ takes, replacing the fixed α with a search against the layer's own output error.
Once difficulty is a movable quantity, the same transform shows up wherever two operands share a quantization budget — which is how the idea reappears, one year later, in a weight-only method aiming at four bits.
Paper: arXiv 2211.10438
(ICML 2023). Code and CUTLASS INT8 kernels:
mit-han-lab/smoothquant. Every probe number on this page
comes from one CPU run over Qwen3-8B — 8 sequences × 256 tokens of WikiText-2, forward hooks on
the inputs of layers.23.self_attn.q_proj and layers.23.mlp.down_proj,
quantizers written from the paper's own equations. Numbers badged
paper are the authors'.