SmoothQuant and AWQ both act by multiplying each input channel by a number — a diagonal transform — and a diagonal transform cannot make an outlier go away, only decide which operand carries it.
That was enough for W8A8 and for 4-bit weights with 16-bit activations. It is not enough for 4-bit both. This page is about what happens when you allow the transform to be any orthogonal matrix instead of a diagonal one — and about the theorem that says a pre-norm transformer cannot tell the difference.
On the previous page the best migration strength was interior — α = 0.4 and 0.6 — because both extremes were bad; at W4A4 the same search runs to 0.7–1.0 across six sites, four of them at 0.9 or beyond, which is the knob telling you it is nearly out of travel.
This is worth seeing before the method, because it is the honest version of "prior work fails". Nothing is broken about SmoothQuant. Its single parameter is simply being asked for more than it can express, and the search now pins it against the boundary. The paper's own table says the same thing from the other end: 4-bit SmoothQuant reaches WikiText-2 perplexity 83.12 on Llama-2-7B against a 5.47 baseline.
Where the migration-strength search lands, on the same two Qwen3-8B layers, at two bit widths.
An interior optimum means the parameter is balancing two real costs. An optimum on the boundary means one cost has swamped the other and the search is just running as far as it is allowed. At four bits the activations are so much harder than the weights that the best available move is to hand the weights nearly everything — and it is still not enough.
LLM.int8() keeps outlier channels in FP16, QUIK keeps 256 of them, Atom keeps 128 — all of them work, and all of them need a second data path for the channels they singled out.
Reading their Table 1 as a column of "#outlier features" is the clearest way to see what QuaRot is claiming. Every competitive 4-bit method at the time bought its accuracy with a set of retained channels. The claim is not "more accurate" — it is zero in that column.
Weights, activations and the KV cache all in INT4, every matmul in INT4, no retained outlier features, no re-training — and the transform that makes it possible must cost nothing at inference time.
Multiplying the hidden state by an orthogonal matrix mixes every coordinate into every other, so the energy that used to sit in one channel is distributed across all of them — and because the transform is orthogonal, the total energy is exactly conserved.
The figure below is the same tensor twice. Nothing has been clipped, scaled, or dropped. The number to check is the Frobenius norm printed under each panel: it is the same to the last digit shown, which is what makes this different from every method on the previous pages.
Qwen3-8B layer 23, the input to q_proj. 48 tokens × 96 of the 4,096 channels, drawn on one shared vertical axis.
Both panels are the same measured tensor, drawn on one axis. The right one is not a smoothed or clipped version of the left: it is the same numbers in a different coordinate system, and every channel now peaks within about a factor of two of every other. The line under the panels is what separates this from everything on the previous pages — the norm did not change, so nothing was traded away to get there.
Per-channel peak magnitude, all 4,096 channels, sorted. Before and after the same rotation.
Read this as two distributions of the same total. Before, a handful of channels hold magnitudes the rest never approach; after, the curve is nearly flat and sits where a quantizer can work. Kurtosis is the one-number version of the same picture — the paper's companion reports it going from over 200 to about 3, and on this layer it goes from 287 to 2.8.
RMSNorm(X) = RMSNorm(XQᵀ)Q for any orthogonal Q, because dividing by the norm is unaffected by a rotation — so Q can be pushed into one weight matrix and Qᵀ into the next, and the model's output is unchanged.
This is the computational-invariance theorem, borrowed from SliceGPT. It is what turns a rotation from a preprocessing trick into something you can fuse offline and never pay for again. The claim is exact, not approximate, and it is worth checking rather than believing.
‖(XQ)(QᵀW)ᵀ − XWᵀ‖ / ‖XWᵀ‖, measured at six sites in Qwen3-8B.
Every site lands at a few times 10⁻⁷, which is float32 round-off and not an approximation error — the identity holds exactly and the arithmetic is what wobbles. That single fact is what lets the rotation be fused into the weights before inference starts.
A Walsh–Hadamard matrix has entries ±1/√d, so every coordinate contributes equally to every output, and Hx costs d log d instead of d² — cheap enough to run online where fusion is impossible.
The catch is that Hadamard matrices only exist at certain sizes. Qwen3-8B's hidden size is 4,096 = 2¹², which the Sylvester construction handles directly. Its MLP width is 12,288, which is not a power of two — that is the case QuaRot handles by factorising d = 2ⁿ·m and taking a Kronecker product with a known Hadamard of size m.
Built from one entry by doubling, checked for orthogonality on two of its own rows, and then applied to a single loud coordinate.
The bottom row is the mechanism the whole method rests on. A vector with all of its energy in one coordinate comes out with that energy spread evenly over every coordinate, at ±1/√d of the original size — and the length of the vector is identical before and after, because that is what orthogonal means. Run that on a hidden state whose energy sits in a few loud channels and you get the flat surface from two figures ago. The middle row is why it is allowed: any two rows of the matrix agree in exactly half their positions and disagree in the other half, so their dot product is zero and the matrix is its own inverse up to scale.
How the matrix is built at each size, and what it costs.
Both constructions were checked before use: max|HHᵀ − I| is round-off in each case. The right-hand column is the reason this particular orthogonal matrix and not a random one — at 4,096 a dense rotation is nearly three hundred times the work of the fast transform, which is the difference between fusing it offline and being able to run it in the middle of a forward pass.
On Llama-2, W4A4KV4 QuaRot loses at most 0.63 perplexity — 0.47 at 70B — against methods that either collapse entirely or retain 128–256 channels in higher precision.
WikiText-2 perplexity, 4-bit weights, activations and KV cache. The right column is how many channels each method keeps out of 4-bit.
SmoothQuant's 83.12 is the same method that was lossless at eight bits two pages ago — the collapse is the diagonal knob running out, not a flaw in it. Note also the last two rows: QuaRot with group-128 quantization matches or beats Atom while keeping zero channels aside.
Relative output error at W4A4 on real Qwen3-8B layers, under four treatments.
Our numbers are more measured than the paper's, and the honest reading matters. At the residual sites the best smoothing and the Hadamard land close together; at the MLP input the rotation is twice as good. A single layer's error is not a perplexity — thirty-six of these compose, and small per-layer differences are what turn into 83.12 against 6.10 end to end. What one layer can show is the direction, and it agrees.
SmoothQuant and AWQ change which operand holds the outlier; LLM.int8(), QUIK and Atom keep the outlier channels in a wider type; QuaRot changes the basis so there is no outlier channel to hold or keep.
There is also a pleasant systems consequence. Because nothing is retained, there is no gather, no re-ordering, and no second matmul — the quantized network is uniformly INT4 and the kernel stays simple. Methods that retain channels pay for them twice: once in precision and once in irregularity.
Before this, quantization work chose how to round and how to scale; after it, the coordinate system itself is something you get to choose — and the network's invariances tell you exactly how much freedom you have.
Q is a randomised Hadamard picked once — the method proves the choice is free, and says nothing about whether it is good.
Which is exactly the opening the next page walks into. If every orthogonal matrix leaves the network unchanged, then the set of valid rotations is a manifold of free parameters, and drawing one at random is a decision nobody defended. SpinQuant measures how much that decision is worth and then optimises it.
Once the basis is a variable it can be trained, and once the outliers are gone the remaining error is ordinary rounding error — which is what the sub-4-bit work after this paper is aimed at.
Paper: arXiv 2404.00456 (NeurIPS 2024). Code: spcl/QuaRot. Every probe number here comes from one CPU run over Qwen3-8B — 8 sequences × 256 tokens of WikiText-2, hooks on the inputs of q_proj at four layers plus o_proj and down_proj at layer 23. The Hadamard matrices are built here (Sylvester for 2ⁿ, a Kronecker product with a Paley-12 block for 12,288) and checked orthogonal before use. Numbers badged paper are the authors'.