Paper Review
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Pure RL with rule-verifiable rewards (GRPO) incentivizes o1-level reasoning to emerge from a base model without supervised CoT, and the capability transfers to small dense models via distillation.
Rule-verifiable reward 기반 pure RL(GRPO)만으로 base model에서 o1급 reasoning이 emergent하게 나타남을 보이고, 그 능력을 distillation으로 small dense model까지 이전한 open recipe.
Background OpenAI o1 showed that scaling inference-time chain-of-thought yields large reasoning gains, but its training recipe stayed closed and the open community was still relying on SFT over human-annotated CoT. The paper opens by noting that prior open explorations — process-based reward models, RL, and search algorithms such as MCTS and Beam Search — had not reached o1-level general reasoning. That left the source of the capability an open question: does it require supervised reasoning traces as a seed, or can it be incentivized directly?
배경 OpenAI o1이 inference-time long chain-of-thought 스케일링으로 큰 reasoning 도약을 보여줬지만 training recipe는 비공개였고, open 진영은 human-annotated CoT에 대한 SFT에 의존하고 있었다. 논문 서두는 공개 진영의 선행 탐색 — process-based reward model, RL, MCTS·Beam Search 같은 search algorithm — 이 어느 것도 o1급 general reasoning에 도달하지 못했다고 정리한다. 그래서 이 능력의 원천이 supervised reasoning trace라는 씨앗인지, 아니면 incentive만으로 직접 유도 가능한 것인지가 열린 질문으로 남아 있었다.
Problem It was unresolved whether strong reasoning can be incentivized by pure RL on a base model — with no supervised reasoning demonstrations at all — and how to turn that into a usable general-purpose model. At bottom this is a question about where reasoning comes from: is human-annotated CoT a necessary seed, or will the right incentive alone make the model evolve its own strategies (self-evolution)? A practical gap rides along: raw pure-RL output is hard to read and mixes languages, so it cannot ship as-is.
문제 Supervised reasoning demonstration 없이 base model에 pure RL만으로 강한 reasoning을 유도할 수 있는지, 그리고 그것을 실사용 가능한 general model로 만드는 방법이 미해결이었다. 본질적으로 reasoning이 어디서 오는가에 대한 질문이다 — human-annotated CoT가 반드시 필요한 씨앗인가, 아니면 올바른 incentive만 주면 모델이 스스로 전략을 진화시키는가(self-evolution). 실용 격차도 함께 걸려 있다: pure RL의 raw output은 읽기 어렵고 language mixing이 있어 그대로는 쓸 수 없다.
Prior limitations Prior open attempts — process reward models, MCTS-style search, and large-scale human CoT annotation — ran into reward hacking, search-space explosion, and annotation cost, and none reached o1-level general reasoning. The paper's own Unsuccessful Attempts section backs this with first-hand evidence: for PRM, fine-grained steps are hard to define and grade in general reasoning and a model-based PRM inevitably invites reward hacking, while for MCTS, token generation's search space is exponentially larger than chess, so capping per-node expansion traps the search in local optima and training a fine-grained value model is inherently difficult. This failure list is precisely why the recipe collapses to outcome-verifiable rewards.
기존 한계 기존 공개 시도들 — process reward model, MCTS 계열 search, 대규모 human CoT annotation — 은 reward hacking, search space 폭발, annotation 비용에 막혀 o1급 general reasoning에 도달하지 못했다. 논문 자체의 Unsuccessful Attempts 절이 이를 실측으로 뒷받침한다: PRM은 general reasoning에서 fine-grained step의 정의와 중간 단계 채점이 어렵고 model-based PRM은 필연적으로 reward hacking을 부르며, MCTS는 체스와 달리 token generation의 search space가 지수적으로 커서 노드당 확장 상한을 걸면 local optima에 갇히고 fine-grained value model 학습 자체가 어렵다. 이 실패 목록이 곧 recipe가 outcome-verifiable reward로 수렴한 이유다.
Goal Reproduce o1-class reasoning with an open recipe: prove the mechanism with pure-RL R1-Zero, make it usable via the multi-stage R1 pipeline, and transfer the capability to small dense models through distillation. The concrete deliverables are three: R1-Zero as an existence proof (RL applied directly to the base model with no SFT), R1 with cold-start data and multi-stage training for usability, and six 1.5B–70B dense models distilled from R1's outputs. Releasing weights for all three tracks — so the recipe can be reproduced and extended outside closed labs — is an explicit goal.
목표 Open recipe로 o1급 reasoning을 재현한다 — pure RL의 R1-Zero로 메커니즘을 입증하고, multi-stage pipeline의 R1로 실사용성을 확보하고, distillation으로 small dense model까지 능력을 이전한다. 구체적 산출물은 셋이다: SFT 없이 base model에 RL을 직접 적용한 R1-Zero(existence proof), cold-start 데이터와 multi-stage 훈련으로 usability를 갖춘 R1, 그리고 R1의 출력으로 distill한 1.5B–70B dense 모델 6종. 세 갈래 모두 weight를 공개해 closed lab 밖에서 재현·확장이 가능하게 하는 것이 명시적 목표다.
Method Starting from DeepSeek-V3-Base (671B MoE, 37B activated), GRPO — a critic-free policy gradient that baselines each sample against its group's mean reward — is trained with rule-based accuracy and think-tag format rewards to produce R1-Zero, and R1 adds a four-stage pipeline: long-CoT cold-start SFT on thousands of samples, reasoning-oriented RL with a language-consistency reward, rejection-sampling SFT on ~800k samples (600k reasoning + 200k non-reasoning), and a final all-scenario RL stage. GRPO drops the critic model (normally as large as the policy itself) and instead samples G outputs per question, scoring each against the group mean — grading sibling attempts at the same problem on a curve instead of keeping a separate judge model. Rewards are verified by rules — boxed final answers for math, compiler test cases for code — plus a format reward that forces the thinking into think tags; neural reward models were deliberately excluded because they suffer reward hacking at RL scale. The six distilled models get SFT only, on the same 800k samples, with no RL stage.
방법 DeepSeek-V3-Base(671B MoE, 37B activated)에 critic 없이 group 평균 reward를 baseline으로 쓰는 GRPO를 rule-based accuracy·format(think 태그) reward로 학습해 R1-Zero를 얻고, R1은 수천 개 long-CoT cold-start SFT → language-consistency reward를 더한 reasoning-oriented RL → rejection sampling 기반 약 800k 샘플 SFT(reasoning 600k + non-reasoning 200k) → all-scenario RL의 4단 pipeline로 훈련한다. GRPO는 policy와 같은 크기의 critic 모델을 없애고 질문마다 G개의 출력을 뽑아 그룹 평균 대비로 각 답을 채점한다 — 별도 심판 모델을 두는 대신 같은 문제에 대한 형제 답안들끼리 상대평가(curve)를 매기는 셈이다. Reward는 규칙으로 검증한다 — 수학은 boxed 최종 답, 코드는 compiler와 test case — 여기에 thinking을 think 태그 안에 쓰도록 강제하는 format reward를 더하며, neural reward model은 대규모 RL에서 reward hacking을 일으켜 의도적으로 배제했다. Distilled 모델 6종은 같은 800k 샘플로 SFT만 수행하고 RL 단계는 생략한다.
Key idea Rule-verifiable rewards (answer correctness + output format) alone are sufficient training signal — self-verification, reflection, and long deliberation emerge from RL incentives without any neural reward model or reasoning SFT. The emblematic case is the "aha moment": an intermediate checkpoint interrupts itself ("Wait, wait.") to re-evaluate its initial approach, a behavior nobody taught it. It is like grading a student only on whether the final answer is right, never showing worked solutions — and watching them spontaneously develop the habit of checking their own work. Figure 3's steadily growing response length is the quantitative trace: the model learns by itself to spend more test-time compute per problem.
핵심 아이디어 정답을 rule로 검증 가능한 reward(accuracy + format)만으로도 충분한 training signal이 되어, neural reward model도 reasoning SFT도 없이 self-verification·reflection 같은 reasoning 행동이 RL incentive에서 emergent하게 나타난다. 대표 사례가 "aha moment"다 — 훈련 중간 checkpoint가 스스로 "Wait, wait." 하며 초기 접근을 재평가하는데, 누구도 가르친 적 없는 행동이다. 학생에게 풀이 과정은 한 번도 보여주지 않고 최종 답의 정오만 채점했는데 스스로 검산하는 습관이 생긴 것과 같다. Figure 3의 response length가 훈련 내내 꾸준히 길어지는 것이 그 정량적 흔적이다 — 모델이 문제당 test-time compute를 늘리는 법을 자발적으로 학습한다.
Validation Evaluated on AIME 2024, MATH-500, CNMO 2024, GPQA Diamond, MMLU/-Redux/-Pro, Codeforces, LiveCodeBench, SWE-bench Verified, Aider, and open-ended judges (AlpacaEval 2.0, ArenaHard), with six distilled 1.5B–70B Qwen2.5/Llama3 checkpoints tested on the reasoning subset. The protocol caps generation at 32,768 tokens and reports pass@1 from temperature-0.6/top-p-0.95 sampling over 4–64 responses per question, since greedy decoding is unstable (repetition) for long-output reasoning models, and keeps prompts zero-shot because few-shot consistently degrades R1. The distilled six are checked on AIME 2024, MATH-500, GPQA Diamond, Codeforces, and LiveCodeBench.
검증 AIME 2024, MATH-500, CNMO 2024, GPQA Diamond, MMLU/-Redux/-Pro, Codeforces, LiveCodeBench, SWE-bench Verified, Aider와 open-ended judge(AlpacaEval 2.0, ArenaHard)에서 평가하고, Qwen2.5/Llama3 기반 1.5B–70B distilled 모델 6종은 reasoning 벤치 부분집합으로 검증했다. 프로토콜은 생성을 32,768 token까지 허용하고, long-output reasoning 모델에서 greedy decoding이 repetition으로 불안정해 temperature 0.6·top-p 0.95 sampling으로 질문당 4–64개 응답을 뽑아 pass@1을 보고하며, few-shot이 성능을 일관되게 떨어뜨려 프롬프트는 zero-shot으로 통일했다. Distilled 6종은 AIME 2024·MATH-500·GPQA Diamond·Codeforces·LiveCodeBench 5개 벤치로 확인한다.
Results R1-Zero lifts AIME 2024 pass@1 from 15.6% to 71.0% (86.7% cons@64) by RL alone, and R1 reaches 79.8% on AIME, 97.3% on MATH-500, 71.5% on GPQA Diamond, and a 2,029 Codeforces rating (96.3 percentile), on par with OpenAI-o1-1217. The gains generalize beyond math and code — 90.8% MMLU, 84.0% MMLU-Pro, an 87.6% length-controlled win-rate on AlpacaEval 2.0 and 92.3% on ArenaHard, all well above DeepSeek-V3. On the distillation side, the 7B distill (55.5% AIME) beats QwQ-32B-Preview, even the 1.5B distill outscores GPT-4o and Claude-3.5-Sonnet on math (28.9% AIME, 83.9% MATH), and the 32B/70B distills exceed o1-mini on most benchmarks.
결과 R1-Zero는 RL만으로 AIME 2024 pass@1을 15.6%→71.0%(cons@64 86.7%)로 끌어올리고, R1은 AIME 79.8%, MATH-500 97.3%, GPQA Diamond 71.5%, Codeforces rating 2,029(96.3 percentile)로 OpenAI-o1-1217과 동급이다. 이득은 수학·코드 밖으로도 일반화된다 — MMLU 90.8%, MMLU-Pro 84.0%, AlpacaEval 2.0 LC win-rate 87.6%, ArenaHard 92.3%로 DeepSeek-V3를 크게 상회한다. Distillation 쪽은 7B distill이 AIME 55.5%로 QwQ-32B-Preview를 넘고, 1.5B distill조차 math 벤치에서 GPT-4o·Claude-3.5-Sonnet을 앞서며(AIME 28.9%, MATH 83.9%), 32B·70B distill은 대부분 벤치에서 o1-mini를 상회한다.
Comparison Benchmarked head-to-head against OpenAI-o1-1217, o1-mini, GPT-4o-0513, Claude-3.5-Sonnet-1022, DeepSeek-V3, and QwQ-32B-Preview; notably, distilled R1-Qwen-32B (72.6% AIME) far exceeds direct RL on the same 32B base (47.0%), showing distillation from a strong teacher beats small-scale RL. That contrast is close to a controlled experiment — 10K+ steps of large-scale RL on Qwen-32B-Base with math/code/STEM data only reaches QwQ-32B-Preview level, while plain SFT on the teacher's 800k samples adds another +25.6pp. Learning from a master's worked solutions beats rediscovering everything alone at small scale — though the paper is explicit that pushing the frontier of intelligence still needs stronger base models and larger-scale RL.
비교 OpenAI-o1-1217, o1-mini, GPT-4o-0513, Claude-3.5-Sonnet-1022, DeepSeek-V3, QwQ-32B-Preview와 직접 비교했으며, 특히 distilled R1-Qwen-32B(AIME 72.6%)가 같은 32B base에 직접 RL을 적용한 경우(47.0%)를 크게 앞서 대형 teacher distillation이 소형 모델 직접 RL보다 낫다는 것을 보였다. 이 대조는 통제 실험에 가깝다 — Qwen-32B-Base에 math·code·STEM 데이터로 10K+ step 대규모 RL을 돌려도 QwQ-32B-Preview 수준에 그치는 반면, teacher가 만든 800k 샘플 SFT만으로 +25.6pp가 더해진다. 소형 모델이 혼자 RL로 모든 것을 재발견하는 것보다 거장의 풀이를 보고 배우는 편이 낫다는 뜻이다 — 단, 논문은 지능의 경계를 넓히려면 여전히 더 강한 base model과 더 큰 규모의 RL이 필요하다고 못박는다.
Significance It open-sourced the first o1-class reasoning recipe, establishing verifiable-reward RL (RLVR) with GRPO as the default post-training paradigm and proving via 800k-sample distillation that small dense models can inherit reasoning. If o1 was a dish everyone could taste but nobody could cook, R1 is the paper that published the recipe — most open reasoning models since are variations on it (reviewer judgment). The documented negative results — why PRM and MCTS fail at RL scale — are a contribution of their own, steering the field away from expensive dead ends.
의의 o1급 reasoning recipe를 처음으로 open-source화해 verifiable-reward RL(RLVR)+GRPO를 post-training의 표준 패러다임으로 만들었고, 800k 샘플 distillation으로 small dense model도 reasoning을 물려받을 수 있음을 증명했다. o1이 모두가 맛볼 수는 있지만 아무도 만들 수 없는 요리였다면, R1은 그 레시피를 공개한 논문이다 — 이후 open reasoning 모델 대부분이 이 recipe의 변형을 쓴다(리뷰어 판단). PRM과 MCTS가 대규모 RL에서 왜 실패하는지 기록한 negative result 역시 필드가 비싼 막다른 길을 피하게 한 독자적 기여다.
Limitations The paper states R1 trails DeepSeek-V3 on function calling, multi-turn, complex role-playing, and JSON output, mixes languages outside Chinese/English, and is consistently sensitive to few-shot prompting, so zero-shot use is recommended. Software engineering saw little gain over V3 because long evaluation times kept large-scale RL from being applied there. R1-Zero itself suffers poor readability and language mixing — the very reason the cold-start stage exists. And the thousands-of-tokens long-CoT decode is itself a real serving cost in latency and KV-cache pressure — like deliberating over the route while the taxi meter runs, the better answer comes with a longer fare (reviewer judgment).
한계 논문 명시 한계로 function calling·multi-turn·복잡한 role-playing·JSON output에서 DeepSeek-V3보다 약하고, 중국어·영어 외 언어에서 language mixing이 발생하며, few-shot prompt에 일관되게 민감해 zero-shot 사용이 권장된다. Software engineering은 RL 평가 지연 때문에 대규모 RL을 적용하지 못해 V3 대비 개선이 크지 않다. R1-Zero 단계 자체도 poor readability와 language mixing을 겪는데, 이것이 cold-start 단계가 존재하는 이유다. 또한 수천 token의 long-CoT decode는 latency와 KV cache 압박이라는 실질적 serving 비용을 만든다 — 미터기가 도는 택시 안에서 경로를 고민하는 격이라, 더 나은 답에는 더 길게 돈 미터기가 따라온다(리뷰어 판단).
Future work Stated directions are leveraging long CoT for general capabilities (function calling, multi-turn, role-playing, JSON output), fixing multilingual mixing, and introducing rejection sampling or asynchronous evaluation to make software-engineering RL efficient. The authors also report that applying RL on top of the distilled models yields significant further gains, but deliberately leave that exploration to the community. The broader open problem is reward design beyond rule-verifiable domains — open-ended tasks that resist verification: a sprint is settled by a stopwatch but figure skating needs judges, and installing a judge (a learned reward model) reopens the door to reward hacking (reviewer judgment).
향후 과제 논문이 제시하는 방향은 long CoT를 function calling·multi-turn·role-playing·JSON output 같은 general capability로 확장하는 것, 다국어 mixing 해결, 그리고 software engineering RL을 위한 rejection sampling 또는 asynchronous evaluation 도입이다. 저자들은 distilled 모델 위에 RL을 얹으면 상당한 추가 이득이 있음을 확인했지만 그 탐구는 커뮤니티에 남겨 뒀다. 더 넓게는 rule-verifiable domain 밖 — 검증이 어려운 open-ended task — 의 reward 설계가 열린 문제로 남아 있다: 100m 달리기는 스톱워치가 판정하지만 피겨 스케이팅은 심판이 필요하고, 심판(learned reward model)을 세우는 순간 reward hacking의 문이 다시 열리기 때문이다(리뷰어 판단).
Released resources Weights for R1-Zero, R1, and the six distilled models (based on Qwen2.5-Math-1.5B/7B, Qwen2.5-14B/32B, Llama-3.1-8B, Llama-3.3-70B-Instruct) are on HuggingFace, with the repository and weights under the MIT license. The distills additionally inherit their base-model licenses (Apache 2.0 for Qwen, the Llama community licenses). The RL training code and training data are not public, so full reproduction of the recipe is possible only at the level of the paper's description.
자원 공개 R1-Zero, R1과 distilled 모델 6종(Qwen2.5-Math-1.5B/7B·Qwen2.5-14B/32B·Llama-3.1-8B·Llama-3.3-70B-Instruct 기반)의 weight가 HuggingFace에 공개됐고, repo와 weight는 MIT license다. 단 distill 모델은 base model의 라이선스(Qwen Apache 2.0, Llama community license)를 함께 따른다. RL training code와 학습 데이터는 비공개라, recipe의 완전한 재현은 논문 서술 수준에서만 가능하다.
The research thread
Where this paper sits 연구의 흐름
DeepSeek-R1 stands where three research lines meet. The alignment line, from InstructGPT (Ouyang et al., 2022), made RL standard for LLM post-training but built everything on a reward model trained from human preference. The verification line pushed toward denser judgment — "Let's Verify Step by Step" (Lightman et al., 2023) argued that process reward models grading every step beat outcome-only checks. The test-time-compute line ran from chain-of-thought prompting (Wei et al., 2022) and STaR-style self-bootstrapping (Zelikman et al., 2022) to OpenAI's o1, which showed longer deliberation buys accuracy but kept the recipe closed. R1's direct ancestor is in-house: DeepSeekMath (Shao et al., 2024) introduced GRPO and rule-checked math rewards at 7B scale. R1 carries that machinery to a 671B base and asks the question all three lines were circling — is the supervised reasoning trace a necessary seed, or just scaffolding?
The shift is a double deletion. RLHF's crown jewel was the reward model, and the process-supervision line wanted an even finer judge; R1 removes both learned arbiters — the neural reward model because it invites reward hacking at RL scale, the critic replaced by GRPO's group-relative baseline. What remains are checks that cannot be gamed: the boxed answer matches or it doesn't, the tests pass or they don't. The deeper inversion is in what reasoning is. Earlier recipes treated chain-of-thought as data to collect and clone; R1-Zero treats it as behavior that emerges when incentive makes it useful — nobody labels the "aha moment" or the lengthening responses, they are selected for. Supervision density is traded for incorruptibility, and the Unsuccessful Attempts section records why the denser alternatives collapsed.
What it opened is by now the default vocabulary of post-training. Tülu 3 (Lambert et al., 2024) had just coined RLVR for RL with verifiable rewards; R1 became its existence proof at frontier scale, and open reproductions — Hugging Face's Open-R1, SimpleRL-style zero-RL runs on smaller bases — followed within weeks. The distillation result quietly created a two-tier economy: reasoning is discovered once by large-scale RL on a strong base, then propagated cheaply by SFT — the paper's own 32B comparison is the receipt.
The unresolved tension is the boundary of verifiability itself. The recipe works exactly where a stopwatch exists — math with a boxed answer, code with a test suite — and the paper is candid that software engineering (slow evaluation) and open-ended tasks (no rule to check) resist it. Expect three pushes next: asynchronous, sandboxed evaluation to make agentic and SWE tasks verifiable; rubric-based and generative verifiers that re-admit learned judgment while containing the reward hacking it brings back; and taming the cost of ever-longer thinking traces, the price the recipe pays for accuracy. Whoever widens the verifiable frontier gets to run this recipe on new domains.
DeepSeek-R1은 세 연구 계보가 만나는 지점에 서 있다. Alignment 계보는 InstructGPT(Ouyang et al., 2022)에서 시작해 RL을 LLM post-training의 표준으로 만들었지만, 모든 것을 human preference를 학습한 reward model 위에 세웠다. Verification 계보는 더 촘촘한 심판 쪽으로 밀고 갔다 — "Let's Verify Step by Step"(Lightman et al., 2023)은 모든 step을 채점하는 process reward model이 outcome-only 검증을 이긴다고 주장했고, 한동안 field도 동의했다. Test-time-compute 계보는 chain-of-thought prompting(Wei et al., 2022)과 STaR류 self-bootstrapping(Zelikman et al., 2022)을 거쳐 OpenAI o1에 이르러 더 긴 숙고가 정확도를 산다는 것을 입증했지만 recipe는 닫아뒀다. 정작 R1의 직계 조상은 사내에 있다: DeepSeekMath(Shao et al., 2024)가 이미 GRPO와 rule-checked math reward를 7B 규모에서 선보였다. R1은 그 기계장치를 671B base로 끌고 가서, 세 계보가 모두 맴돌던 질문을 정면으로 던진다 — supervised reasoning trace는 필수적인 씨앗인가, 아니면 비계(scaffolding)일 뿐인가?
전환의 핵심은 이중 삭제다. RLHF의 왕관 보석은 reward model이었고 process-supervision 계보는 더 정밀한 심판을 원했는데, R1은 학습된 심판 둘을 모두 걷어낸다 — neural reward model은 대규모 RL에서 reward hacking을 부르기에 배제하고, critic은 GRPO의 group-relative baseline으로 대체한다. 남는 것은 조작할 수 없는 검증뿐이다: boxed 답이 맞거나 틀리거나, test가 통과하거나 아니거나. 더 깊은 역전은 reasoning이 무엇인가에 있다. 이전 recipe들은 chain-of-thought를 수집하고 복제할 데이터로 다뤘지만, R1-Zero는 incentive가 유용하게 만들 때 나타나는 행동으로 다룬다 — "aha moment"도 갈수록 길어지는 response도 누가 라벨링한 것이 아니라 선택된 것이다. Supervision의 밀도를 내주고 supervision의 비부패성(incorruptibility)을 얻는 거래이며, PRM·MCTS를 다룬 Unsuccessful Attempts 절이 더 촘촘한 대안들이 왜 무너졌는지를 기록해 뒀다.
이 논문이 연 것은 이제 post-training의 기본 어휘다. Tülu 3(Lambert et al., 2024)가 verifiable reward 기반 RL에 RLVR이라는 이름을 막 붙인 참이었고, R1은 frontier 규모에서 그 existence proof가 되었으며, 몇 주 안에 open 재현 — Hugging Face의 Open-R1, 소형 base 위의 SimpleRL류 zero-RL 재현 — 이 뒤따랐다. Distillation 결과는 조용히 2층 경제를 만들었다: reasoning은 강한 base 위의 대규모 RL로 한 번 발견되고, 그 다음은 SFT로 값싸게 전파된다 — 같은 32B에 직접 RL을 돌리면 따라오지 못한다는 논문 자신의 비교가 그 증거다.
풀리지 않은 긴장은 verifiability의 경계 그 자체다. 이 recipe는 정확히 스톱워치가 존재하는 곳 — boxed 답이 있는 수학, test suite가 있는 코드 — 에서 작동하고, 논문은 software engineering(느린 평가)과 open-ended task(검증할 rule의 부재)가 저항한다고 솔직하게 적는다. Field는 세 방향으로 밀 것이다: agentic·SWE task를 검증 가능하게 만드는 asynchronous·sandboxed evaluation, 학습된 판단을 다시 들여오되 그것이 되불러오는 reward hacking을 가두려는 rubric·generative verifier, 그리고 비용 길들이기 — 갈수록 길어지는 thinking trace는 이 recipe가 정확도에 지불하는 가격이기 때문이다. Verifiable frontier를 넓히는 쪽이 이 recipe를 새 도메인에서 돌릴 권리를 얻는다.
Sparks — where one could take this 아이디어의 씨앗
-
The authors report RL on top of the distilled checkpoints yields significant further gains but leave it to the community — one could try sweeping distill-then-RL across model scales to find where small-scale RL starts exceeding its teacher. 저자들이 distilled checkpoint 위에 RL을 얹으면 상당한 추가 이득이 있다고 보고하면서도 탐구를 커뮤니티에 남겨둔 만큼, distill-then-RL 조합을 model scale별로 체계적으로 sweep해 small-scale RL이 어디서부터 teacher를 넘어서기 시작하는지 시험해볼 수 있다.
-
Since the paper states long evaluation times blocked large-scale software-engineering RL, one could try asynchronous-evaluation or rejection-sampling reward pipelines that keep policy updates flowing while slow test suites run. 논문이 명시한 대로 software engineering RL이 긴 evaluation time에 막혀 있으니, 느린 test suite가 도는 동안에도 policy 업데이트가 멈추지 않는 asynchronous evaluation이나 rejection sampling 기반 reward pipeline을 설계해볼 수 있다.
-
Given the reported performance cost of the language-consistency reward that fixes mixing, one could try quantifying that alignment tax per language and searching for reward weightings that recover both readability and accuracy. Language mixing을 잡는 language-consistency reward가 성능을 소폭 희생시킨다고 보고된 만큼, 그 alignment tax를 언어별로 정량화하고 readability와 정확도를 모두 회복하는 reward weighting을 탐색해볼 수 있다.
Seeds grounded in the paper's own stated limitations and open questions. 논문이 스스로 밝힌 limitations와 open questions에 뿌리를 둔 씨앗입니다.
Notes
Notes
DeepSeek-R1 is the report that moved reasoning post-training from closed labs into the open: R1-Zero is the cleanest existence proof that rule-verifiable rewards alone can elicit long-horizon reasoning from a base model, and the four-stage R1 pipeline (cold-start SFT → RLVR → rejection-sampling SFT → alignment RL) became the template most open reasoning models now follow. The negative results are as instructive as the headline — neural PRMs collapsed to reward hacking and MCTS to search-space explosion, which is precisely why the recipe reduces to outcome-verifiable rewards. From an efficiency standpoint its two lasting artifacts are the distilled 1.5B–70B dense models, now a standard testbed for compression and serving work, and the shift of inference cost into long thinking traces — the very regime that makes long-generation KV-cache research matter. 이후 공개된 reasoning 모델 대부분이 이 recipe를 변형해 쓰고 있고, GRPO/RLVR 계열 후속 연구의 출발점이 된 논문이다.
Linked reviews
MiniMax-01: Scaling Foundation Models with Lightning Attention
One softmax-attention block per seven lightning (linear) attention blocks, scaled to a 456B-parameter MoE, matches GPT-4o-class quality while training at 1M-token context and extrapolating to 4M.
on-deviceMobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
A systematic sub-billion design-space study showing that at 125M-350M scale deep-thin layouts, embedding sharing, GQA, and block-wise layer sharing buy accuracy per byte, yielding state-of-the-art on-device LMs.
speculative-decodingDSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
A semi-autoregressive drafter combined with confidence-scheduled verification fixes the intra-block acceptance decay of parallel speculative-decoding drafters, delivering 51-85% throughput and latency gains in DeepSeek-V4 production serving.
speculative-decodingLess Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts (EcoSpec)
Speculative decoding for Mixture-of-Experts models optimizes for token-acceptance likelihood alone, ignoring that each draft token also determines which experts get loaded from HBM during verification; EcoSpec adds a lightweight expert-activation predictor and a dynamic expert buffer to draft-tree selection, cutting the number of distinct experts touched per verification step and reaching up to 1.62x speedup on DeepSeek-V3.1, Qwen3-235B-A22B, and GPT-OSS-120B without touching the verification rule itself.
architectureKimi K3: Open Frontier Intelligence
A 2.8T-parameter (104B active) native multimodal MoE model with a 1M-token context, built on Kimi Delta Attention, Attention Residuals, and Stable LatentMoE — architecture co-design that yields a 2.5x scaling-efficiency gain over Kimi K2 and full open weights.
llm-reasoningParason: Revealing Subtask and Trial Parallelism in LLM Reasoning
Parason shows that most of the hidden parallelism in LLM reasoning traces is Trial Parallelism (parallel speculative attempts), not the Subtask Parallelism prior systems target, and trains models via PA-GRPO plus tool-call execution for a real 1.7x wall-clock speedup.
kv-cacheTriAttention: Efficient Long Reasoning with Trigonometric KV Compression
Observes that pre-RoPE Q/K vectors concentrate around fixed centers, so attention can be predicted as a trigonometric series in token distance — a query-free KV eviction score that matches full-attention AIME25 accuracy at 2.5x throughput or 10.7x less KV memory.
kv-cacheWindowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
Windowed-MTP names the long-context draft-attention tax in built-in Multi-Token-Prediction speculative-decoding heads -- the draft re-reads the entire KV cache every step, so at million-token context it can flip speculation net-negative -- and fixes it losslessly with a training-free sink+window applied only to the draft's attention, cutting per-decode-step cost 28-44% across three architecture families at 1M tokens.