A pre-norm transformer is invariant to any orthogonal transform of its hidden state, so QuaRot could fuse one into the weights and pay nothing — and it picked that one at random.
Free and irrelevant are different things. If every orthogonal matrix leaves the full-precision network identical, then the set of valid rotations is a whole manifold of parameters that the network does not care about — but the quantized network does. Nobody had checked how much.
Over 100 random trials on Llama-2-7B at W4A4 the best rotation beat the worst by 13 points of zero-shot accuracy, and even restricting to random Hadamards left a spread of about 6 — so the coin flip was worth more than most papers' contributions.
That measurement is the paper. Everything after it is the corollary. Below is the same experiment at the scale we can run: 32 randomised Hadamards and 32 random orthogonal matrices on one real layer, each one a separate quantization of the same weights.
W4A4 relative output error of Qwen3-8B layer 23 q_proj, once per drawn rotation.
Two things reproduce and one does not, and the one that does not is the interesting part. Randomised Hadamards are better on average than random orthogonals, and they are tighter — both as the paper reports. But our spread is under one percent, not thirteen points, because a single layer is not a network: the paper's variance is what thirty-two independently drawn rotations do once their errors compose through the whole model. You cannot see this effect by staring at one layer, which is exactly why the search has to be driven by the task loss.
Kurtosis measures tailedness — 3 is a Gaussian, and the paper finds layers above 200 before rotation and about 3 after; on Qwen3-8B the same measurement runs from 210–472 down to 2.8–2.9.
Kurtosis across channels within a token, averaged over 512 tokens, at six sites in Qwen3-8B.
Note the one site that does not fit: the input to the output projection starts at 14, not in the hundreds, and rotating it barely helps. Attention output is already a mixture of many value vectors, so it is closer to Gaussian before anyone touches it — and a rotation has correspondingly little to do. Not every tensor in a transformer has the same problem, which is one reason the method needs more than one kind of rotation.
Find the orthogonal matrices that minimise the quantized network's loss, while keeping them exactly orthogonal so the full-precision network stays untouched and no parameters are added.
R₁ on the residual and R₂ head-wise on the value path can be folded into weights, so they are learnable and free at runtime; R₃ in the KV path and R₄ before the down-projection sit between operations that do not commute, so they stay Hadamard and run online.
The split is not a design preference, it is a consequence of where the network's invariances actually hold. Learning is only possible where fusion is possible; everywhere else the matrix has to be one you can apply quickly, which means a Hadamard.
Qwen3-8B: hidden 4,096, 32 heads of 128, MLP width 12,288.
The learnable pair is small. R₁ is one 4,096 × 4,096 matrix and R₂ is 128 × 128, which together come to a fraction of a percent of the model — the paper puts it at 0.26%. That is the whole optimisation surface: everything else stays frozen, including every weight.
A gradient step would leave the manifold, so the update is the Cayley transform of a skew-symmetric matrix, R′ = (I − α/2·Y)⁻¹(I + α/2·Y)R, which is orthogonal by construction for any Y with Yᵀ = −Y.
Below is that optimiser, running on a real layer. It learns SpinQuant's R₂ — one 128 × 128 rotation shared across the 32 heads of layer 23 — against the layer's own W4A4 output error, with the weights frozen. One detail is not optional: round() has zero gradient almost everywhere, so without a straight-through estimator the loss carries no signal back to R at all and the optimiser sits still.
Cayley SGD on a 128 × 128 head-wise rotation, 120 iterations, CPU. The horizontal lines are the 32 random Hadamards it has to beat.
The learned rotation ends below every one of the 32 draws, and by more than the draws differ from each other — which is the point: this is not a lucky sample from the same distribution, it is outside it. The orthogonality error at the end is the second number to check. If the update had merely been projected back onto the manifold each step it would drift; the Cayley form does not.
LLaMA-2, LLaMA-3 and Mistral families under W4A4KV4, W4A8KV8 and related settings, with zero-shot reasoning averages against SmoothQuant, LLM-QAT, OmniQuant and QuaRot.
On LLaMA-2-7B at 4-bit weights, activations and KV cache, SpinQuant lands 2.9 points below full precision — 19.1 points better than LLM-QAT and 25.0 better than SmoothQuant — and on LLaMA-3-8B it closes up to 45.1% of the remaining gap that QuaRot leaves.
LLaMA-2-7B, W4A4KV4, zero-shot reasoning average — distance below full precision, lower is better.
The bar to weigh is not SpinQuant against SmoothQuant — those are two generations apart, and the earlier page already showed why the diagonal knob cannot reach four bits. The one that matters is SpinQuant against QuaRot, because they differ in exactly one thing: whether the rotation was drawn or learned.
Both methods rotate, both fuse what they can and run Hadamards where they cannot; SpinQuant differs only in that R₁ and R₂ are optimised rather than sampled.
That is an unusually clean comparison for this literature, and it is worth noticing how it was made possible. QuaRot's contribution — proving the rotation is free — is what turned the rotation into a variable that could be optimised at all. The second paper could only exist because the first one established the invariance.
The transferable move is not Cayley SGD — it is noticing that an invariance in your model is a parameter you have not chosen yet, and then measuring what that choice is worth before deciding whether to optimise it.
The rotations are learned against a task loss on 800 WikiText-2 samples, so they are fitted to that distribution, and R₃ and R₄ remain unlearned because they cannot be merged.
Our own layer-level run makes the flip side visible: optimising one layer's output error is not the same objective as the network's loss, and our learned R₂ beats the best random draw by 7% on the metric it was given. Which metric to optimise is itself a design choice the paper makes deliberately — the task loss, not per-layer reconstruction.
Rotation is one symmetry of a transformer; permutations, per-head bases and scaling symmetries are others, and each is a parameter that costs nothing in full precision and something in a quantized one.
Paper: arXiv 2405.16406 (ICLR 2025). Code: facebookresearch/SpinQuant. Every probe number here comes from the same CPU run as the QuaRot page — Qwen3-8B, 8 sequences × 256 tokens of WikiText-2. The Cayley optimisation is implemented from the paper's Eq. 3–4 with a straight-through estimator through the quantizers, 120 iterations on one layer, and it reports its own orthogonality error. Numbers badged paper are the authors'.