Paper Review
Gated Delta Networks: Improving Mamba2 with Delta Rule
Unifies Mamba2-style gated decay with the delta-rule targeted memory update into a single hardware-efficient linear-attention layer that beats both parents and anchors strong hybrid stacks with sliding-window attention.
Mamba2식 gated decay와 delta rule의 targeted memory update를 하나의 hardware-efficient linear-attention 레이어로 통합해 두 부모 모델을 모두 능가하고, sliding-window attention과의 hybrid 스택의 핵심 레이어가 된 논문.
Background Linear-attention / state-space layers replace the length-proportional KV cache with a fixed-size matrix state for O(1)-memory inference, but have trailed Transformers on retrieval and long-context tasks. The state is an associative memory that superimposes key-value pairs as outer products, so the number of orthogonal associations it can hold is bounded by model dimensionality — once the sequence exceeds that, "memory collisions" corrupt exact retrieval (Schlag et al., 2021). In effect the model takes lecture notes on a single chalkboard instead of a notebook, so what gets erased, and how, determines memory quality.
배경 Linear attention / state-space 레이어는 길이에 비례해 커지는 KV cache를 고정 크기 matrix state로 대체해 O(1) 메모리 추론을 제공하지만, retrieval과 long-context 성능에서 Transformer에 밀려 왔다. 이 state는 key-value 쌍을 outer product로 겹쳐 쓰는 연상 메모리여서 저장 가능한 직교 연관 수가 모델 차원에 묶이고, 시퀀스가 그 차원을 넘으면 "memory collision"이 일어나 정확한 회수가 무너진다(Schlag et al., 2021). 말하자면 노트가 아니라 칠판 한 장에 강의 전체를 받아적는 구조라, 무엇을 어떻게 지우느냐가 곧 기억 품질을 결정한다.
Problem Memory control in fixed-state models evolved as two separate mechanisms — gating (global decay) and the delta rule (precise replacement of a specific key-value association) — so no single layer could both rapidly erase stale context and surgically overwrite targeted associations. To forget one association, Mamba2 must fade the entire state uniformly; DeltaNet, conversely, has no way to bulk-erase outdated information at context switches. One camp held only a spray that fades the whole whiteboard, the other only an eraser that rewrites one line at a time.
문제 고정 state의 메모리 관리는 gating(전역 decay)과 delta rule(특정 key-value association의 정밀 교체)이라는 별개 메커니즘으로 나뉘어 발전해, 오래된 컨텍스트를 빠르게 지우면서 동시에 특정 연관만 정확히 덮어쓰는 능력을 한 레이어가 갖지 못했다. Mamba2는 연관 하나를 잊으려 해도 state 전체를 균일하게 바래게 해야 하고, DeltaNet은 반대로 컨텍스트 전환 시 낡은 정보를 일괄 소거할 수단이 없다. 한쪽 진영은 화이트보드 전체를 옅게 만드는 스프레이만, 다른 쪽은 한 줄씩 고쳐 쓰는 지우개만 들고 있었던 셈이다.
Prior limitations Mamba2's uniform decay fades the entire state indiscriminately, hurting precise retrieval (4.6% on S-NIAH-3 at 4K), while DeltaNet performs only rank-1 replacement with no global forgetting and collapses under noisy retrieval (14.4% on S-NIAH-2 at 8K). The paper's S-NIAH case study distills this into three observations — decay hurts long-term retention (Mamba2 falls to 30.4% on pure-memorization S-NIAH-1 at 8K vs DeltaNet's 98.8%), gating helps filter irrelevant context, and the delta rule helps memorize complex values (UUIDs). DeltaNet's known mediocrity on real-world tasks is likewise attributed to the missing memory-clearing mechanism.
기존 한계 Mamba2의 uniform decay는 state 전체를 무차별하게 바래게 해 정밀 retrieval이 약하고(S-NIAH-3 4K에서 4.6%), DeltaNet은 rank-1 교체만 가능해 전역 forgetting 수단이 없어 노이즈 있는 retrieval에서 무너진다(S-NIAH-2 8K에서 14.4%). 논문의 S-NIAH 사례 연구는 이를 세 관찰로 정리한다 — decay는 장기 보존을 해치고(순수 암기형 S-NIAH-1 8K에서 Mamba2 30.4% vs DeltaNet 98.8%), gating은 무관 정보 필터링을 돕고, delta rule은 복잡한 값(UUID) 암기를 돕는다. DeltaNet의 알려진 실세계 태스크 부진 역시 memory-clearing 메커니즘의 부재로 설명된다.
Goal Unify gating and the delta rule in one recurrent update while preserving a chunkwise-parallel, tensor-core-friendly training form, so the combined layer surpasses both parents. The equation merge itself is one line; the real technical work is hardware efficiency — injecting the gating term into DeltaNet's WY-representation-based parallel algorithm (Yang et al., 2024) without breaking its matmul-dominated structure.
목표 Gating과 delta rule을 하나의 recurrent update로 통합하되, tensor-core 친화적 chunkwise-parallel 학습 형태를 유지해 두 부모 모델을 모두 능가하는 것이 목표다. 수식 결합 자체는 한 줄이고, 실질적 기술 과제는 hardware efficiency에 있다 — DeltaNet의 WY-representation 기반 병렬 알고리즘(Yang et al., 2024)에 gating 항을 주입하면서 matmul 중심 구조를 깨지 않아야 한다.
Method The gated delta rule S_t = S_{t-1} α_t (I − β_t k_t k_tᵀ) + β_t v_t k_tᵀ applies a data-dependent scalar decay α_t ∈ (0,1) (Mamba2-style parameterization) before a rank-1 delta correction with writing strength β_t, and a chunkwise-parallel algorithm built on the WY representation and UT transform keeps the whole recurrence in tensor-core-friendly matmul form. In the block design, q/k pass through linear projection, short convolution and SiLU with L2 normalization for training stability, v takes the same path without L2 norm, and α/β use plain linear projections. Viewing the state as a fast-weight matrix trained by test-time SGD — the paper's own lens — the delta rule is a gradient step with learning rate β_t and the gate is an adaptive weight decay attached to it.
방법 Gated delta rule S_t = S_{t-1} α_t (I − β_t k_t k_tᵀ) + β_t v_t k_tᵀ — data-dependent 스칼라 게이트 α_t ∈ (0,1)(Mamba2식 파라미터화)로 state를 decay시킨 뒤 writing strength β_t의 rank-1 delta 보정을 적용하며, WY representation + UT transform 기반 chunkwise-parallel 알고리즘이 recurrence 전체를 tensor-core 친화적 matmul 형태로 유지한다. 블록 설계에서 q/k는 linear projection + short convolution + SiLU에 학습 안정성을 위한 L2 normalization을 거치고, v는 L2 norm 없이 같은 경로, α/β는 linear projection만 쓴다. State를 test-time SGD로 학습되는 fast-weight 행렬로 보면 — 논문 자신의 관점 — delta rule은 learning rate β_t의 gradient step이고 게이트는 거기 붙는 adaptive weight decay다.
Key idea Gating and the delta rule are complementary rather than competing — α_t→0 gives rapid memory erasure and α_t→1 recovers the pure delta rule — so a single scalar gate grants one layer both global forgetting and local precise editing — like a mixer tap merging separate hot and cold faucets into one lever, a single continuous knob covers the entire range between the two camps' mechanisms. In the online-learning framework (Liu et al., 2024), this amounts to relaxing the regularizer that pins S_t to S_{t−1} by the factor α_t, permitting controlled deviations — i.e. selective forgetting — exactly when the state saturates.
핵심 아이디어 Gating과 delta rule은 경쟁이 아니라 상보 관계라는 관찰 — α_t→0이면 급속 memory 소거, α_t→1이면 순수 delta rule로 환원되므로, 스칼라 게이트 하나로 전역 forgetting과 국소 정밀 편집을 같은 레이어에서 얻는다 — 온수·냉수 꼭지 두 개를 레버 하나로 합친 믹서 수전처럼, 연속 손잡이 하나가 두 진영의 메커니즘 전 구간을 커버한다. 온라인 학습 프레임(Liu et al., 2024)으로 보면 S_t를 S_{t−1}에 묶어두는 regularization 항을 α_t만큼 이완하는 것으로, state가 포화되는 바로 그 시점에 통제된 이탈 — 즉 선택적 망각 — 을 허용하는 설계다.
Validation 400M/1.3B models trained on 100B FineWeb-Edu tokens (Llama2 tokenizer, 4K sequences, AdamW lr 4e-4) are evaluated on Wikitext/LAMBADA perplexity, zero-shot commonsense suites, S-NIAH, recall-intensive tasks (SWDE, SQuAD, FDA, TriviaQA, NQ, Drop), 14 LongBench tasks, length extrapolation up to 20K, and H100 training throughput — a controlled comparison with RetNet, HGRN2, Mamba, Mamba2, DeltaNet, Samba and Transformer++ retrained under identical conditions. A 400M/15B-token ablation isolates the gate itself: reverting to the naive delta rule degrades average perplexity from 27.35 to 30.87, and short convolution, output gate and L2 norm all prove essential.
검증 FineWeb-Edu 100B tokens로 400M/1.3B 모델을 학습(Llama2 tokenizer, 4K seq, AdamW lr 4e-4)해 Wikitext/LAMBADA perplexity, zero-shot commonsense, S-NIAH, recall-intensive 태스크(SWDE·SQuAD·FDA·TriviaQA·NQ·Drop), LongBench 14개 태스크, 20K까지의 length extrapolation, H100 학습 throughput으로 검증했다 — RetNet·HGRN2·Mamba·Mamba2·DeltaNet·Samba·Transformer++를 동일 조건에서 재학습한 통제 비교다. 400M/15B-token ablation은 gate 자체를 분리 검증한다: naive delta rule로 되돌리면 평균 perplexity가 27.35→30.87로 악화되고, short convolution·output gate·L2 norm 모두 필수 구성으로 확인된다.
Results At 1.3B it reaches Wikitext ppl 16.42 (vs Mamba2 16.56, DeltaNet 17.71), LAMBADA ppl 12.17 (Mamba2 12.56), and 55.32% average zero-shot accuracy — best among pure recurrent models. On retrieval it beats both parents consistently — 29.6% on S-NIAH-2 at 8K (Mamba2 17.0, DeltaNet 14.4), 27.6% on S-NIAH-3 at 4K (Mamba2 4.6), 30.6 average on real-world recall tasks (Mamba2 29.8, DeltaNet 26.2), and 16.6% LongBench average (vs 13.5/13.6). The hybrids H1/H2 push zero-shot to 56.4%/56.2% and recall averages to 39.0/40.1, surpassing even Transformer++ (52.25/37.0).
결과 1.3B에서 Wikitext ppl 16.42(Mamba2 16.56, DeltaNet 17.71), LAMBADA ppl 12.17(Mamba2 12.56), zero-shot 평균 55.32%로 순수 recurrent 모델 중 최고를 기록했다. Retrieval에서도 두 부모를 일관되게 앞선다 — S-NIAH-2 8K 29.6%(Mamba2 17.0, DeltaNet 14.4), S-NIAH-3 4K 27.6%(Mamba2 4.6), 실세계 recall 평균 30.6(Mamba2 29.8, DeltaNet 26.2), LongBench 평균 16.6%(vs 13.5/13.6). Hybrid H1/H2는 zero-shot 56.4%/56.2%, recall 평균 39.0/40.1까지 끌어올려 Transformer++(52.25/37.0)마저 넘어선다.
Comparison Against Transformer++ (FlashAttention-2), Mamba/Mamba2, DeltaNet, RetNet, HGRN2 and the Samba hybrid it wins broadly on quality, while H100 training throughput is essentially identical to DeltaNet and only 2-3K tok/s below Mamba2 (~46K vs ~48K tok/s at 1.3B) — the gate is practically free. Transformer++ is fastest at 2K context (~55K tok/s) but drops to ~27K at 16K sequences, whereas the SWA-based H1 holds ~52-55K across the whole range. The regime where hybrids dominate pure attention on both quality and speed is visible directly in the throughput plot.
비교 Transformer++(FlashAttention-2), Mamba/Mamba2, DeltaNet, RetNet, HGRN2, Samba hybrid와 비교해 품질은 전반적으로 우위이며, H100 학습 throughput은 DeltaNet과 사실상 동일하고 Mamba2보다 2-3K tok/s 낮은 수준(1.3B 기준 ~46K vs ~48K tok/s)이라 gating의 추가 비용이 사실상 공짜임을 보였다. Transformer++는 2K 컨텍스트에서 ~55K tok/s로 가장 빠르지만 16K 시퀀스에서 ~27K까지 떨어지는 반면, SWA 기반 H1은 전 구간 ~52-55K를 유지한다. 품질과 속도 양면에서 hybrid가 순수 attention을 지배하는 구간이 그래프에 그대로 드러난다.
Significance It demonstrates that the form of the state update — not just state size — determines retrieval ability in constant-memory layers: the state stays the same size and only the update rule changes — not one shelf was added to the library; changing the librarian's filing rules is what made more books findable. The gated delta rule has since been adopted as the standard linear component in production hybrid (full+linear) attention stacks such as Qwen3-Next, effectively becoming an off-the-shelf part. Concurrent convergence by RWKV-7 (diagonal-plus-low-rank transitions) and Titans (test-time SGD with weight decay) suggests this design is a general principle rather than a one-off trick (reviewer judgment).
의의 고정 state 레이어의 retrieval 능력은 state 크기가 아니라 state update의 형태가 결정한다는 것을 보인 결과다 — state 크기는 그대로 두고 update rule만 바꿔 retrieval 지표를 갈아치웠다 — 서가를 한 칸도 늘리지 않고 사서의 정리 규칙만 바꿔 찾을 수 있는 책을 늘린 셈이다. Gated delta rule은 이후 Qwen3-Next 같은 production hybrid(full+linear) attention 스택의 표준 linear 컴포넌트로 채택되며 사실상 기성 부품이 됐다. 동시기 RWKV-7(diagonal-plus-low-rank transition)과 Titans(test-time SGD + weight decay)가 같은 지점으로 수렴한 것은 이 설계가 일회성 트릭이 아니라 일반 원리임을 방증한다(리뷰어 판단).
Limitations The paper itself shows gating sacrifices perfect retention — on pure-memorization S-NIAH-1 at 8K, DeltaNet holds 98.8% while Gated DeltaNet drops slightly to 91.8%. Gains over Mamba2 also shrink on real recall tasks, which the authors attribute to repetition errors of instruction-unaligned small models dominating the error budget and diluting update-rule differences. The fixed-state capacity ceiling is only bypassed by hybrids rather than removed, and validation stops at 1.3B/100B tokens (reviewer judgment).
한계 논문 스스로 gating이 완벽한 기억 보존을 희생함을 보인다 — 순수 암기형 S-NIAH-1 8K에서 DeltaNet은 98.8%를 유지하지만 Gated DeltaNet은 91.8%로 소폭 낮다. 실세계 recall 태스크에서는 Mamba2 대비 이득이 synthetic보다 줄어드는데, 저자들은 instruction-tuning 안 된 소형 모델의 repetition error가 지배적 오류원이어서 update rule 차이가 희석되기 때문으로 분석한다. 고정 state의 용량 상한 자체는 hybrid로 우회될 뿐 해소되지 않으며, 검증 스케일도 1.3B/100B tokens에 머문다(리뷰어 판단).
Future work The authors' explicitly stated next step is evaluating these models on sequences beyond 20K tokens. The related-work section flags negative-eigenvalue transitions (Grazzi et al.) and products of Householder matrices (Siems et al.) as extensions that could be applied to Gated DeltaNet seamlessly for stronger state tracking, and notes the chunkwise algorithm adapts to RWKV-7-style diagonal-plus-low-rank transitions. More expressive online objectives — TTT/Titans-style nonlinear regression or Mesa-layer full-history regression — remain open directions, as their nonlinear recurrences require parallelization workarounds.
향후 과제 저자들이 명시한 다음 단계는 20K를 넘는 시퀀스에서의 능력 검증이다. Related work에서는 negative eigenvalue 허용(Grazzi et al.)과 Householder 행렬 곱 확장(Siems et al.)이 state-tracking 표현력 강화를 위해 Gated DeltaNet에 그대로 적용 가능하다고 지목하고, RWKV-7식 diagonal-plus-low-rank transition으로의 chunkwise 알고리즘 확장 가능성도 언급한다. TTT·Titans식 nonlinear regression이나 Mesa layer식 전체-이력 regression 같은 더 표현력 있는 online objective는 nonlinear recurrence 탓에 병렬화 우회가 필요한 열린 방향으로 남는다.
Released resources Official training code is released at the NVlabs/GatedDeltaNet GitHub repository, and kernel implementations are integrated into the flash-linear-attention (fla-org) library — the same library also ships the RWKV-7-style generalized delta rule variants. Published at ICLR 2025.
자원 공개 공식 학습 코드가 NVlabs/GatedDeltaNet GitHub에 공개되어 있고, 커널 구현은 flash-linear-attention(fla-org) 라이브러리에 통합되어 있다 — 같은 라이브러리가 RWKV-7류 generalized delta rule 변형까지 함께 제공한다. ICLR 2025 발표 논문이다.
The research thread
Where this paper sits 연구의 흐름
This paper is the confluence of two branches grown from one trunk. After Katharopoulos et al. (2020) showed that kernelized attention is secretly an RNN with a matrix-valued state, the field split over one question — how should that fixed state forget? One lineage answered with decay: from RetNet's fixed exponential discount to the data-dependent gates of GLA (Gated Linear Attention) and Mamba/Mamba2. The other reached back to a 1960s idea: Schlag, Irie and Schmidhuber (2021) recast linear attention as fast weight programmers and revived the delta rule — overwrite exactly the association being updated — which DeltaNet (Yang et al., 2024) finally parallelized over sequence length via the WY representation. The genealogy is unusually literal — Songlin Yang wrote both GLA and DeltaNet, so this paper merges her own two branches.
The conceptual shift is to drop the question "which forgetting mechanism is better" and treat global decay and targeted overwrite as orthogonal axes of one design space. The paper's own fast-weight lens makes the union feel inevitable rather than additive: if the state is a weight matrix trained by test-time SGD, the delta rule is the gradient step and the gate its adaptive weight decay — the gated delta rule is simply the optimizer written out in full. Recurrence design quietly becomes optimizer design. Layered on top is a cultural shift: a mechanism only exists once it passes the hardware test — if the recurrence does not reduce to chunkwise matmuls, it is disqualified regardless of quality.
What the paper opened first is the production hybrid. The H1/H2 recipes — gated delta layers interleaved with sliding-window attention or Mamba2 — previewed the full+linear stacks that MiniMax-01 demonstrated at frontier scale, and Qwen3-Next adopted the gated delta rule as its literal linear component. But the hybrid is a concession as much as a victory — the fixed-state capacity ceiling is bypassed, not removed, and the layers still carrying a real KV cache become the new cost center — that seam between the two attention regimes is where the next round of systems work concentrates.
The deeper opening is theoretical. That RWKV-7 (diagonal-plus-low-rank transitions) and Titans (test-time SGD with weight decay) converged on the same point suggests a unifying frame — "sequence layer = online learner" — and the paper's related work practically hands out the next moves: negative-eigenvalue (Grazzi et al.) and Householder-product (Siems et al.) transitions for state tracking, plus richer online objectives à la TTT and Mesa layers, whose nonlinear recurrences still refuse chunkwise parallelization. The remaining tension is one the paper itself exhibits: the gate that rescues noisy retrieval taxes pure retention (98.8%→91.8% on S-NIAH-1). Under a fixed memory budget, forgetting is not a free lunch but a spend — and learning where to spend it is this line's open problem.
이 논문은 같은 뿌리에서 갈라진 두 갈래가 다시 만나는 합류점에 서 있다. Katharopoulos et al.(2020)이 kernelized attention이 사실 matrix state를 가진 RNN임을 보인 뒤, 분야는 하나의 질문으로 갈라졌다 — 이 고정 state는 어떻게 잊어야 하는가. 한 갈래는 decay로 답했다: RetNet의 고정 exponential decay에서 GLA(Gated Linear Attention)와 Mamba/Mamba2의 data-dependent gate로. 다른 갈래는 1960년대 아이디어로 돌아갔다: Schlag, Irie, Schmidhuber(2021)가 linear attention을 fast weight programmer로 재해석하며 delta rule — 갱신할 association만 정확히 덮어쓰기 — 을 부활시켰고, DeltaNet(Yang et al., 2024)이 WY representation으로 이를 시퀀스 길이에 대해 병렬화했다. 이 계보는 드물게 문자 그대로다 — GLA와 DeltaNet 모두 Songlin Yang의 작업이니, Gated DeltaNet은 저자가 자기 두 갈래를 스스로 합류시킨 논문이다.
개념적 전환은 "어느 forgetting 메커니즘이 더 나은가"라는 질문을 버리고, memory control을 전역 decay와 국소 overwrite라는 직교하는 두 축의 설계 공간으로 보는 데 있다. 논문 자신의 fast-weight 관점은 이 결합을 덧셈이 아니라 필연으로 만든다: state가 test-time SGD로 학습되는 weight matrix라면 delta rule은 gradient step이고 gate는 거기 붙는 adaptive weight decay이니, gated delta rule은 그저 optimizer를 완전한 형태로 적어낸 것이다. recurrence 설계가 조용히 optimizer 설계가 된다. 여기에 문화적 전환이 하나 더 겹친다 — 메커니즘은 hardware test를 통과해야 비로소 존재한다는 것. recurrence가 chunkwise matmul로 환원되지 않으면 품질과 무관하게 실격이다.
이 논문이 먼저 연 것은 production hybrid다. gated delta 레이어를 sliding-window attention이나 Mamba2와 교차 적층한 H1/H2 레시피는 MiniMax-01이 프런티어 스케일에서 보여준 full+linear 스택을 예고했고, Qwen3-Next는 gated delta rule을 문자 그대로 linear 컴포넌트로 채택했다. 그러나 hybrid는 승리인 동시에 양보다 — 고정 state의 용량 상한은 제거된 게 아니라 우회됐고, 여전히 진짜 KV cache를 지고 있는 레이어들이 새 비용 중심이 된다. 두 attention 체제가 만나는 그 이음새가 다음 시스템 작업이 몰리는 곳이다.
더 깊은 개방은 이론 쪽이다. RWKV-7(diagonal-plus-low-rank transition)과 Titans(weight decay 딸린 test-time SGD)가 같은 지점으로 수렴했다는 사실은 "sequence layer = online learner"라는 통합 프레임을 시사하고, 논문의 related work는 다음 수를 거의 나눠준다 — state tracking을 위한 negative eigenvalue(Grazzi et al.)와 Householder 곱(Siems et al.) transition, 그리고 TTT·Mesa layer식의 더 표현력 있는 online objective. 후자의 nonlinear recurrence는 아직 chunkwise 병렬화를 거부한다. 남은 긴장은 논문이 스스로 정직하게 보인 그것이다: noisy retrieval을 구해낸 gate가 순수 retention에 세금을 물린다(S-NIAH-1에서 98.8%→91.8%). 고정된 메모리 예산 아래에서 forgetting은 공짜 점심이 아니라 지출이며, 어디에 지출할지 배우는 것이 이 계보의 열린 문제다.
Sparks — where one could take this 아이디어의 씨앗
-
Following the authors' stated next step, one could evaluate gated-delta-style models on sequences far beyond 20K tokens to see whether the gate's retention penalty on pure memorization grows or vanishes with length and scale. 저자들이 명시한 다음 단계 그대로, 20K를 훨씬 넘는 시퀀스에서 gated delta 계열을 평가해 순수 암기에서의 retention 손실이 길이와 스케일에 따라 커지는지, 사라지는지 확인해볼 수 있다.
-
As the related work itself suggests, one could graft negative-eigenvalue transitions (Grazzi et al.) or Householder-product transitions (Siems et al.) onto the gated delta rule for stronger state tracking while keeping the chunkwise matmul form. 논문 related work가 직접 지목하듯, negative eigenvalue transition(Grazzi et al.)이나 Householder 곱(Siems et al.)을 gated delta rule에 접목해 chunkwise matmul 형태를 유지한 채 state tracking 표현력을 올려볼 수 있다.
-
One could hunt for chunkwise-parallel training algorithms for the richer online objectives the paper leaves open — TTT/Titans-style nonlinear regression or Mesa-layer full-history regression. 논문이 열린 방향으로 남긴 TTT·Titans식 nonlinear regression이나 Mesa layer식 full-history regression에 대해, 그 nonlinear recurrence를 chunkwise-parallel하게 만드는 학습 알고리즘을 찾아볼 수 있다.
Seeds grounded in the paper's own stated limitations and open questions. 논문이 스스로 밝힌 limitations와 open questions에 뿌리를 둔 씨앗입니다.
Notes
Notes
Gated DeltaNet is the cleanest demonstration to date that decay and targeted replacement are complementary axes of memory control in constant-state sequence layers, and its hybrid variants (interleaving sliding-window attention or Mamba2 layers) preview the full+linear attention recipe now common in frontier models. For efficiency work, the interesting boundary is exactly where the fixed-size state fails and the hybrid must fall back to full attention — that seam is where KV-cache cost actually lives in these stacks. The chunkwise WY-representation algorithm is also a good reference point for what “hardware-efficient” must mean for any new recurrence: the update has to reduce to tensor-core matmuls, or the throughput story dies regardless of quality.
Hybrid(full+linear) attention 아키텍처에서 linear 쪽 표준 부품이 된 레이어라, KV cache 관점에서는 “남아 있는 full attention 레이어의 캐시를 어떻게 다루느냐”가 다음 병목이 된다.
Linked reviews
MiniMax-01: Scaling Foundation Models with Lightning Attention
One softmax-attention block per seven lightning (linear) attention blocks, scaled to a 456B-parameter MoE, matches GPT-4o-class quality while training at 1M-token context and extrapolating to 4M.
kv-cacheDuoAttention: Efficient Long-Context Inference with Retrieval and Streaming Heads
Learns which attention heads actually retrieve from long context and gives only those a full KV cache, cutting long-context memory up to 2.55x with retrieval accuracy intact.
linear-attentionThe Key to Going Linear: Analysis-Driven Transformer Linearization
By freezing the entire pretrained backbone and training only a replacement attention mechanism, this paper derives a first-order Taylor approximation showing that softmax attention intrinsically applies a key-dependent rank-1 correction, which delta-rule updates (Gated DeltaNet) implement natively and pure gated-decay updates (Gated Linear Attention) structurally cannot, then adds sink tokens and short convolutions to patch exactly where that approximation still fails, beating prior post hoc linearization baselines on MMLU while using orders of magnitude fewer training tokens.
long-contextJet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE
A tuning-free method extends an LLM's context window beyond its pretrained length by combining a local RoPE-faithful attention window with a long-range window whose rescaling factor adapts dynamically to the current sequence length, fused into a single GPU kernel.
hybrid-architectureKernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones
KATA derives linear-attention feature maps from symmetric-cone geometry, names a Welch-bound interference floor that governs associative-recall capacity, and shows several variants beating Gated DeltaNet on long-range recall while running up to ~11x FlashAttention-2's throughput at 131k tokens.
architectureKimi K3: Open Frontier Intelligence
A 2.8T-parameter (104B active) native multimodal MoE model with a 1M-token context, built on Kimi Delta Attention, Attention Residuals, and Stable LatentMoE — architecture co-design that yields a 2.5x scaling-efficiency gain over Kimi K2 and full open weights.
on-deviceMobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
A systematic sub-billion design-space study showing that at 125M-350M scale deep-thin layouts, embedding sharing, GQA, and block-wise layer sharing buy accuracy per byte, yielding state-of-the-art on-device LMs.
hybrid-architectureMOSAIC: Adaptive Inter-layer Composition for Efficient Heterogeneous Vision-Language Models
MOSAIC automatically searches, per transformer layer, over five attention variants and seven FFN widths to turn a homogeneous VLM into a hardware-latency-constrained heterogeneous one, using cheap blockwise distillation plus a multi-objective MIP solver instead of hand-designed periodic mixing patterns, and its resulting MOSAIC-4B matches Qwen3-VL-4B-Instruct across 19 benchmarks at 2.54x decoding speedup and under 2% of the original training cost.
compressionNemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs
An iterative pruning-plus-distillation pipeline compresses a hybrid Mamba/MoE model from 120.7B to 75.3B total parameters, roughly doubling server throughput at matched user-throughput constraints while retaining most downstream accuracy.
ssmRaven: High-Recall Sequence Modeling with Sparse Memory Routing
Raven unifies SSMs and sliding-window attention under one framework (Routing Slot Memories) by giving a fixed set of memory slots a learned, input-dependent router that selects a Top-K subset to update each step and decays only those slots, matching or beating linear-time baselines on recall benchmarks and extrapolating to 16x its training context length.
speculative-decodingTreeWY: Speculative Verification for Gated DeltaNet Hybrids
TreeWY removes the per-draft-position recurrent-state snapshot that makes wide speculative-decoding draft trees memory-infeasible for hybrid softmax+Gated-DeltaNet models, by rewriting the gated delta rule as a tree-structured WY transform that verifies every draft node with one triangular solve and reconstructs only the accepted state on commit.