SAGE: self-play adversarial games enhance LLM reasoning

§Abstract

We introduce Sage (Self-play Adversarial Games for Enhancement), a framework for improving LLM reasoning capabilities through adversarial self-play without human-curated data. Sage places two model instances in an asymmetric game: a Setter generates a problem and predicts its solution, while an Opponent attempts to solve the problem independently. The Setter receives positive reward only when it answers correctly and the Opponent fails, incentivizing the generation of problems that are solvable yet challenging and naturally targeting the frontier of model capabilities. We instantiate Sage in two domains: Code-Game, where problems are Python programs verified by execution, and Math-Game, where math problems are graded by an external LLM judge, as a proxy for a verifiable environment. Training models from 1B to 4B parameters across two architectures (Qwen3, Llama), Sage consistently outperforms baselines: up to +10% on MATH, +8% on MBPP, and +6% on ARC-Challenge. Notably, we find cross-domain transfer: Code-Game training improves mathematical reasoning and vice versa, suggesting Sage strengthens domain-general reasoning skills. Ablations confirm that adversarial pressure, rather than verified rewards alone, drives these gains: removing the opponent while retaining execution-verified rewards decreases the improvement by 40–70%. Sage offers a scalable path to reasoning improvement that requires only a verifier, not human supervision.

§1Introduction

Large language models (LLMs) have achieved remarkable performance on code generation and mathematical reasoning benchmarks, yet their capabilities remain bounded by the quality and diversity of training data. As models approach or exceed human-level performance on existing benchmarks, the scarcity of novel, challenging problems becomes a critical bottleneck for further improvement¹. We draw on work that has achieved superhuman performance through self-generated experience: self-play in game-playing agents² and adversarial training in generative models³. AlphaZero demonstrated that an agent can rapidly improve without human supervision by playing against itself, while GANs formalized model improvement as a minimax game between generator and discriminator.

Can a language model improve its own reasoning by writing problems that defeat a copy of itself?

We propose Sage (Self-play Adversarial Games for Enhancement), a framework that combines these insights for LLM improvement. In Sage, two LLM instances engage in an asymmetric game:

• The Setter (\(\pi_\theta\)) generates a problem \(p\) and a predicted answer \(\hat a\).
• The Opponent (\(\pi_{\mathrm{opp}}\)) observes only \(p\) and produces its own prediction \(\tilde a\).
• The Verifier (\(\mathcal V\)) computes the ground-truth answer \(a^{*}\).
• The Setter receives reward based on whether it “fools” the Opponent, that is, whether \(\hat a = a^{*}\) while \(\tilde a \neq a^{*}\).

Here, the Setter is incentivized to generate problems that are (i) verifiable (correctness can be determined by comparing solutions against ground truth), (ii) solvable (\(\hat a\) must equal \(a^{*}\)), and (iii) challenging (\(\tilde a \neq a^{*}\)). During training, this reward drives the Setter toward increasingly sophisticated problems that probe the frontier of its current capabilities. The opening figure follows one such round.

We propose Sage for two domains:

• Code-Game. The Setter writes a deterministic Python program that prints exactly one value (numeric, string, or boolean), and then predicts that value. The Opponent sees the code and attempts the same prediction. Rewards are verified by program execution.
• Math-Game. The Setter constructs a math problem of its choice, then predicts its answer (accurate to six decimal digits, or a symbolic answer). The Opponent attempts to compute the same value. Rewards are computed using LLM-judged correctness (GPT-4.1-mini), serving as a proxy reward.

§2Method

§2.1Problem Formulation

We frame adversarial self-play as a two-player asymmetric game. Let \(\mathcal P\) denote the space of problems (programs or mathematical expressions), \(\mathcal A\) the space of answers, and \(\mathcal V : \mathcal P \to \mathcal A \cup \{\bot\}\) a verifier that computes ground-truth answers (returning \(\bot\) for invalid problems). An episode of the adversarial prediction game proceeds as follows:

• Setter move. Given context \(c\), the Setter \(\pi_\theta\) generates \((p, \hat a) \sim \pi_\theta(\cdot \mid c)\).
• Verification. Compute \(a^{*} = \mathcal V(p)\).
• Solver move. The Solver \(\pi_\phi\) observes \(p\) and generates \(\tilde a \sim \pi_\phi(\cdot \mid p)\).
• Outcome. Compare the predictions to the ground truth and assign rewards.

We consider two training regimes: frozen-opponent (\(\phi\) fixed throughout training) and co-evolving (\(\phi\) updated alongside \(\theta\)). We use “Opponent” for the frozen second player and “Solver” for the co-evolving one, to emphasize the distinct training dynamics.

§2.2Reward Structure

The Setter’s reward captures the adversarial objective. Writing \(\mathrm{correct}_G\) for the Setter (generator) and \(\mathrm{correct}_S\) for the second player:

When training the Solver concurrently, we use a simple correctness reward:

§2.3Training Algorithm

We train the Setter using Group Relative Policy Optimization (GRPO)⁴, which estimates advantages from groups of samples without requiring a separate value network.

§2.4Game Variants

Code-Game. The Setter generates a deterministic Python program \(p\) that prints exactly one value (boolean, string, number), along with a predicted output \(\hat a\). The verifier \(\mathcal V\) executes \(p\) in a sandboxed subprocess with resource constraints: a timeout of 4 seconds, a separate Python process with captured stdout and stderr, and the standard library only, with no file I/O, network, or randomness. The ground-truth answer \(a^{*}\) is extracted from the program’s stdout. This execution-based verification provides deterministic reward signals.

Math-Game. The Setter constructs a math problem of its choice, then predicts its answer (either as a numeric value or a symbolic expression such as a LaTeX fraction). The Opponent attempts to compute the same answer. Rewards are computed using LLM-judged correctness (GPT-4.1-mini)⁵, which handles both numeric comparison (with tolerance \(\epsilon = 10^{-6}\)) and symbolic equivalence, serving as a proxy reward. This LLM-as-judge approach extends Sage to domains where symbolic evaluation is insufficient, such as word problems requiring multi-step reasoning or problems with multiple valid answer formats, at the cost of introducing a learned verifier. Figure 2 shows a round of each game.

During training, we periodically audited grading reliability on a fixed subset of 20 held-out problems and found that our GPT-4.1-mini–based grader matched the ground-truth labels on 19/20 cases (95% accuracy). We also manually inspected generated problems throughout training for signs of reward hacking (e.g., malformed problems that exploit judge parsing, adversarial phrasings). We observed no systematic exploitation: generated problems remained mathematically coherent, and errors were typical calculation mistakes rather than judge-specific exploits.

§2.5Opponent Variants: Frozen vs. Co-Evolving

We consider two training regimes that differ in whether the second player’s parameters are updated. Both share the same game mechanics.

Frozen Opponent. The Opponent is initialized from the same base model as the Setter but remains fixed throughout training (\(\phi\) fixed at initialization). This provides stable reward signals and isolates the Setter’s learning dynamics.

Co-Evolving. Both the Setter and the second player (called the Solver in this setting) update via GRPO, alternating training phases. The Solver receives reward based on correctness alone (Eq. 2). This tests whether co-evolution produces complementary improvements, and can be used to prevent a potential saturation of the reasoning gains against a frozen opponent. Training details are in Appendix B.

§2.6Ablation Conditions

To isolate the contribution of adversarial pressure from other aspects of our training setup, we design two ablation conditions that systematically vary the reward structure and prompt framing:

• SPR (Same Prompt, Different Reward) uses the identical game-framed prompt as Sage, but rewards are based solely on execution correctness: \(+1\) if the Setter’s prediction matches the verified output, \(-1\) otherwise. The Opponent is never queried. This isolates whether gains come from verified execution rewards alone.
• DPDR (Different Prompt, Different Reward) uses a standard code or math generation prompt without game framing, with the same correctness-only reward as SPR. This tests whether the competitive prompt framing itself provides learning signal independent of adversarial rewards.

If adversarial pressure is essential, we expect the ordering Sage > SPR > DPDR ≥ Baseline: removing the Opponent (SPR) should reduce gains, and further removing the game framing (DPDR) should reduce them further. If verified rewards alone suffice, SPR should match Sage.

Relationship to RLVR baselines. Our DPDR ablation is structurally equivalent to standard RL with verified rewards (RLVR): the model generates code or math solutions and receives binary rewards based on verified correctness. Our ablations control for general RLVR effects: any improvement of Sage over DPDR cannot be attributed to verified rewards or GRPO optimization alone, since both conditions share these components.

§3Experiments

To evaluate the generality of Sage across architectures and scales, we conduct experiments on two model families at multiple parameter counts. We use Qwen3-1.7B and Qwen3-4B⁶, representing small and medium-scale models within Alibaba’s Qwen3 series. We additionally evaluate on Llama-3.2-1B-Instruct and Llama-3.2-3B-Instruct from Meta’s Llama series⁷.

We evaluate on three benchmarks spanning code generation, mathematical reasoning, and general reasoning:

• MBPP (Mostly Basic Python Programming)⁸: Python programming tasks with test cases.
• MATH⁹: problems covering algebra, geometry, number theory, and calculus.
• ARC-Challenge: the challenge split of the AI2 Reasoning Challenge, testing commonsense and scientific reasoning in multiple-choice format.

All evaluations use temperature \(\tau = 0.01\) for generation. We evaluate both MATH and ARC-Challenge on a fixed, pre-specified subset of 200 problems (selected prior to experimentation) used for all model variants.

§4Results

We evaluate Sage under two configurations, frozen Opponent and co-evolving Opponent, and compare against three conditions: the untrained baseline, DPDR (standard generation with correctness rewards), and SPR (game-framed prompt with correctness rewards but no opponent). Figure 3 shows every result; Table 1 and Table 2 list the numbers. “Frozen-Opp Gen” reports the Setter’s downstream performance under the frozen-opponent regime, while “Co-Evolve Gen” and “Co-Evolve Solver” report the Setter’s and the Solver’s performance under the co-evolving regime.

Frozen-opponent Sage is at or above the baseline and both ablations in 23 of the 24 model–benchmark cells (the exception is Llama-3.2-1B on ARC after Math-Game training). The gap between Sage (full adversarial objective) and SPR (same prompt, correctness-only reward) isolates the contribution of the opponent: if verified rewards alone sufficed, SPR would match Sage. Where SPR exceeds DPDR, the game-framed prompt provides some benefit even without adversarial rewards, likely by encouraging varied problem difficulty.

The co-evolving Setter matches or exceeds the frozen-opponent Setter on several benchmarks, most clearly for the Llama models. We hypothesize that an improving opponent provides a richer curriculum than a frozen one, preventing the Setter from overfitting to a fixed adversary’s weaknesses. One plausible explanation for the remaining cases is that the frozen opponent has not saturated within our training budget; under this hypothesis, longer training could amplify the observed gap.

Cross-domain transfer. Code-Game training improves mathematical reasoning (MATH: up to +7 points), while Math-Game training improves code generation (MBPP: up to +5 points). We hypothesize this reflects shared underlying reasoning capabilities: both domains require understanding and solving problems and careful attention to edge cases. The adversarial objective may be particularly effective at strengthening these domain-general skills because it forces the model to identify and exploit subtle reasoning gaps.

§4.1Accuracy over Training

Figure 4 shows benchmark accuracy over training steps for frozen-opponent Sage across model families. Accuracy rises through training with small dips and no collapse, and the curves are not saturated: both MBPP curves are still improving at step 750, suggesting additional training could yield further gains.

§4.2Training Dynamics

Figure 5 shows the training reward of every Sage variant (frozen-opponent and co-evolving) along with the ablations. The co-evolving curves show the two players adapting to each other: the Setter’s reward starts near \(+1\) and falls as the Solver climbs from \(-1\), with reversals as each side overcomes the other’s improving strategy. DPDR stays at consistently high reward and SPR is more volatile. In the Code-Game, the frozen-opponent Setter’s reward trends upward over training: the Setter learns to generate increasingly valid problems that it can solve but the frozen Opponent cannot.

§4.3How the Problems Change

Figure 6 gives problems the Setter generated early and late in training, for code and for math, to illustrate the progression in difficulty. Early problems are routine. Late problems are built around a trap: a dictionary whose keys silently collide, or an expression whose intimidating first term is numerically zero.

§5Discussion

§5.1The Reward Paradox: Lower Training Rewards, Better Performance

A seemingly counterintuitive finding emerges from comparing training dynamics to downstream performance: for much of training, frozen-opponent Sage earns lower rewards than DPDR (Figure 5), yet it produces better benchmark results (Table 1, Table 2). This apparent paradox resolves when we consider what each reward signal measures. A correctness-only reward saturates near \(+1.0\) once the model learns reliable generation patterns, because its objective (generate correct problems) becomes trivial. High reward reflects task mastery, but the task itself provides diminishing learning signal. The model is no longer being challenged. Moderate rewards under the adversarial objective indicate that the Setter is operating at the edge of its capabilities: generating problems hard enough to sometimes fool the Opponent, but not so hard that the Setter itself fails.

An adversarial objective that maintains moderate rewards throughout training may induce more robust learning than a simpler objective that quickly saturates. The key is not maximizing reward but maximizing the information content of the reward signal.

§5.2Why Does Adversarial Self-Play Work?

The central question is why Sage outperforms simpler alternatives that also use verified rewards. In standard supervised fine-tuning, or even RLVR without an adversarial component, the training distribution is fixed: problems are drawn from a static dataset or generated according to a fixed sampling procedure. The model may waste capacity on problems it already solves easily or fail to make progress on problems far beyond its current abilities.

Sage induces a fundamentally different dynamic. The Setter receives maximum reward (\(+1\)) only when it generates problems that are simultaneously (i) within its own capability (it must predict correctly) and (ii) beyond the Opponent’s capability (the Opponent must fail). Problems that are too easy (both succeed, reward \(0\)) or too hard (both fail, reward \(-0.5\)) provide weaker learning signal. This concentrates gradient updates precisely at the boundary where the Setter slightly exceeds the Opponent.

Standard fine-tuning with data augmentation generates training examples according to fixed transformations (e.g., paraphrasing, back-translation, synthetic problem generation). While this increases data diversity, it does not adapt to the model’s evolving capabilities. Self-play, by contrast, implements a form of active learning where the training distribution is determined by model capabilities. Each training iteration generates problems specifically calibrated to the current capability frontier. This is qualitatively different from passive consumption of fixed data: it represents a shift from “learning from data” to “learning from self-generated experience.”

§6Related Work

RLVR (RL with verified rewards) has emerged as a promising direction for LLM improvement¹. While RLVR has been applied to code generation^10,11,12, these approaches typically rely on curated problem sets. Sage’s adversarial formulation provides a natural incentive for generating diverse, high-quality problems: the Setter is rewarded only when problems are verifiable, solvable, and challenging enough to defeat the Opponent, eliminating the need for external dataset curation. Recent work has advanced RL-based training for LLMs: CodeRL+ trains models to infer variable-level execution trajectories beyond binary rewards¹³, while Wen et al.¹⁴ show that RLVR extends reasoning boundaries by implicitly incentivizing correct chain-of-thought. GRPO estimates advantages from sample groups without a separate value network⁴, and DeepSeek-R1¹⁵ demonstrated that GRPO with rule-based rewards can elicit sophisticated reasoning through pure RL. SeRL¹⁶ combines self-instruction generation with online filtering for quality and diversity.

Self-play has been applied to negotiation¹⁷, non-zero-sum games¹⁸, debate^19,20, and adversarial language games²¹. SPIRAL²² showed zero-sum games transfer to improved reasoning with 8.6% math gains. For alignment, self-play preference methods^23,24 use game-theoretic frameworks where models compete against prior versions, as in SPIN²⁵. Multi-agent approaches further enhance this: execution feedback in self-play improves instruction-following²⁶, while co-evolution among diverse agents yields complementary gains beyond single-agent improvement^27,28. AlphaCode²⁹ achieved human-competitive programming performance through massive sampling and execution-based filtering, while ACES³⁰ generates progressively harder puzzles using solve-rate heuristics. Self-improvement frameworks leverage model-generated feedback^31,32,33 and iterative refinement for reasoning^34,35,36,37. Recent work extends these ideas to code: a coding agent that edits its own codebase through reflection³⁸, self-generated experiences stratified by difficulty for curriculum learning³⁹, and GRPO applied to software evolution with iterative bug injection and repair^12,40.

Absolute Zero¹ and R-Zero⁴¹ also target reasoning improvement from zero external data through self-evolving task generation. Sage differs in key respects: (i) it uses an explicit two-player adversarial game with separate Setter and Solver roles rather than a single model in both roles; (ii) its reward is directly adversarial (the Setter is rewarded for fooling the Solver) rather than based on task learnability; and (iii) it extends to domains with proxy verification (Math-Game with an LLM judge), beyond execution-verified coding tasks.

While RLVR and value-based approaches like B-Coder⁴² excel with abundant high-quality data, Sage is complementary: it requires only a verifier and generates its own training problems in data-scarce settings, enables continuous curriculum generation through adversarial self-play that concentrates learning at the capability frontier, and comes with controlled ablations (SPR and DPDR, Section 2.6) isolating the opponent’s contribution beyond verified rewards alone.

§7Future Work & Conclusion

Sage demonstrates strong results across two verification regimes, deterministic execution for Code-Game and LLM-as-judge for Math-Game, but several directions remain for extending adversarial self-play to broader and more challenging domains. Proxy verification through an external LLM judge introduces an implicit capability ceiling: the judge must be sufficiently capable to provide accurate rewards, and the Setter cannot learn to generate problems beyond the judge’s grading ability. In our setup, GPT-4.1-mini suffices because the trained models’ capabilities remain below it. Two observations suggest scalable alternatives for frontier models. First, inference-time compute can boost verification capability, allowing the same base model to serve as a stronger judge with additional reasoning budget. Second, verification is fundamentally easier than generation, a principle formalized in the P vs. NP distinction, so problems can be structured to be self-verifiable through logical consistency, embedded constraints, or executable sub-components. Our results show that learning occurs even under imperfect reward signals, suggesting that with careful problem design, Sage can scale beyond the capability ceiling imposed by external graders.

Moving beyond mathematics to creative domains (story generation, poetry, persuasive writing, open-ended dialogue) presents a more interesting challenge than grading accuracy: the absence of ground-truth answers entirely. What does it mean for the Setter to fool the Opponent when quality is subjective? Domains with partial verifiability offer natural intermediate test cases. Theorem proving provides machine-checkable proofs but requires creative lemma discovery that benefits from LLM judgment; scientific hypothesis generation can be partially verified through simulation or literature consistency; code optimization admits correctness verification via execution, but quality assessment (efficiency, readability) requires judgment.

Conclusion. We introduced Sage, a framework for improving LLM capabilities through adversarial self-play games. By competing against an opponent to generate “fooling” problems with verifiable solutions, an LLM learns to probe the boundaries of reasoning capabilities, discovering edge cases and subtle patterns. Our framework provides a scalable, data-efficient path toward LLM self-improvement that requires only a verifier rather than human supervision.

§References

1. 1Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute Zero: Reinforced self-play reasoning with zero data. 2025. arXiv:2505.03335
2. 2David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. 2017. arXiv:1712.01815
3. 3Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, pp. 2672–2680, 2014.
4. 4Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. 2024. arXiv:2402.03300
5. 5OpenAI. GPT-4.1. API release, April 14, 2025. openai.com
6. 6Qwen Team. Qwen3 technical report. 2025. arXiv:2505.09388
7. 7Llama Team. The Llama 3 herd of models. 2024. arXiv:2407.21783
8. 8Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. 2021. arXiv:2108.07732
9. 9Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. NeurIPS, 2021.
10. 10Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C.H. Hoi. CodeRL: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35, 2022.
11. 11Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. Execution-based code generation using deep reinforcement learning. 2023. arXiv:2301.13816
12. 12Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. SWE-RL: Advancing LLM reasoning via reinforcement learning on open software evolution. 2025. arXiv:2502.18449
13. 13Xue Jiang, Yihong Dong, Mengyang Liu, Hongyi Deng, Tian Wang, Yongding Tao, Rongyu Cao, Binhua Li, Zhi Jin, Wenpin Jiao, Fei Huang, Yongbin Li, and Ge Li. CodeRL+: Improving code generation via reinforcement with execution semantics alignment. 2025. arXiv:2510.18471
14. 14Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Yang Mao. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs. 2025. arXiv:2506.14245
15. 15DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. 2025. arXiv:2501.12948
16. 16Wenkai Fang, Shunyu Liu, Yang Zhou, Kongcheng Zhang, Tongya Zheng, Kaixuan Chen, Mingli Song, and Dacheng Tao. SeRL: Self-play reinforcement learning for large language models with limited data. 2025. arXiv:2505.20347
17. 17Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from AI feedback. 2023. arXiv:2305.10142
18. 18Austen Liao, Nicholas Tomlin, and Dan Klein. Efficacy of language model self-play in non-zero-sum games. 2024. arXiv:2406.18872
19. 19Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. pp. 17889–17904, 2024.
20. 20Samuel Arnesen, David Rein, and Julian Michael. Training language models to win debates with self-play. 2024. arXiv:2409.16636
21. 21Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Zheng Yuan, Yong Dai, Lei Han, Nan Du, and Xiaolong Li. Self-playing adversarial language game enhances LLM reasoning. In Advances in Neural Information Processing Systems, volume 37, pp. 126515–126543, 2024.
22. 22Bo Liu, Simon Yu, Zichen Liu, Leon Guertler, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, and Natasha Jaques. SPIRAL: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. 2025. arXiv:2506.24119
23. 23Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. 2024. arXiv:2405.00675
24. 24Xiaohang Tang, Sangwoong Yoon, Seongho Son, Huizhuo Yuan, Quanquan Gu, and Ilija Bogunovic. Game-theoretic regularized self-play alignment of large language models. 2025.
25. 25Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. 2024. arXiv:2401.01335
26. 26Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving instruction-following capabilities of large language models. 2024. arXiv:2406.13542
27. 27Yixing Chen, Yiding Wang, Siqi Zhu, Haofei Yu, Tao Feng, Muhan Zhang, Mostofa Patwary, and Jiaxuan You. Multi-agent evolve: LLM self-improve through co-evolution. 2025. arXiv:2510.23595
28. 28Vighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains. 2025. arXiv:2501.05707
29. 29Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with AlphaCode. Science, 378(6624):1092–1097, 2022.
30. 30Julien Pourcel, Cédric Colas, Gaia Molinaro, Pierre-Yves Oudeyer, and Laetitia Teodorescu. ACES: Generating diverse programming puzzles with autotelic language models and semantic descriptors. 2024.
31. 31Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. RLAIF: Scaling reinforcement learning from human feedback with AI feedback. 2023.
32. 32Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. 2022. arXiv:2212.08073
33. 33Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E. Weston. Self-rewarding language models. pp. 57905–57923, 2024.
34. 34Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-Refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594, 2023.
35. 35Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652, 2023.
36. 36Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STaR: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
37. 37Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. 2022. arXiv:2203.11171
38. 38Maxime Robeyns, Martin Szummer, and Laurence Aitchison. A self-improving coding agent. 2025. arXiv:2504.15228
39. 39Genghan Zhang, Weixin Liang, Olivia Hsu, and Kunle Olukotun. Adaptive self-improvement LLM agentic system for ML library development. 2025. arXiv:2502.02534
40. 40Yuxiang Wei, Zhiqing Sun, Emily McMilin, Jonas Gehring, David Zhang, Gabriel Synnaeve, Daniel Fried, Lingming Zhang, and Sida Wang. Toward training superintelligent software agents through self-play SWE-RL. 2025. arXiv:2512.18552
41. 41Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-Zero: Self-evolving reasoning LLM from zero data. In the 5th Workshop on Mathematical Reasoning and AI at NeurIPS 2025.
42. 42Zishun Yu, Yunzhe Tao, Liyu Chen, Tao Sun, and Hongxia Yang. B-Coder: Value-based deep reinforcement learning for program synthesis. 2023. arXiv:2310.03173

§ACorrectness

We define correctness for numeric answers as \(\mathrm{correct}_{G,S} := |\hat a - a^{*}| < \epsilon\), where \(\epsilon = 10^{-6}\) provides numerical tolerance for floating-point comparisons. For non-numeric answers (booleans, strings, etc.), we evaluate correctness using direct equality comparison.

§BCo-Evolving Opponent Training Details

Unlike the frozen opponent setting, concurrent training alternates between updating the Generator (the Setter) and the Solver. At each phase \(t\):

• Both models play \(N\) games in evaluation mode.
• If \(t \bmod 2 = 0\): train the Generator via GRPO for \(k\) steps.
• If \(t \bmod 2 = 1\): train the Solver via GRPO for \(k\) steps.

This alternating schedule prevents the destabilizing effect of simultaneous updates while allowing both players to improve. In our experiments, we use \(N = 25\) games per phase and \(k = 25\) GRPO steps.

Solver training. The Solver trains on problems generated during gameplay, stored in a rolling buffer of up to 1000 games, with reward \(+1.0\) if \(|\tilde a - a^{*}| < \epsilon\) and \(-1.0\) otherwise. Unlike the Generator, which trains on self-generated prompts, the Solver trains on code extracted from successful Generator outputs, ensuring it faces problems at the current difficulty frontier.

Experience replay. Training samples are drawn from games where (i) the code parsed successfully, (ii) the code executed without error, and (iii) the Generator predicted correctly. This filtering ensures the Solver only trains on valid, solvable problems.

§CCompute Resources

All experiments were conducted on a single NVIDIA A100-40GB GPU. Training used mixed precision (bfloat16 for Code-Game, float16 for Math-Game) with gradient checkpointing enabled for full fine-tuning runs. Training is computationally accessible: a single Sage run fits on one A100-40GB.

§DReward Function Ablations

We ablated several reward functions for training the Generator, and found that simpler binary objectives consistently produced the most stable learning dynamics. Below we summarize the variants we explored and why they underperformed.

Binary “fool the opponent” reward. We first used a purely adversarial objective: \(+1\) if the opponent is fooled (fails to predict the output), \(-1\) otherwise. In practice, this produced poor learning signals early in training: episodes were dominated by negative rewards, yielding highly skewed feedback and unstable updates. This led to noisy training dynamics and frequent collapse rather than gradual curriculum formation.

Graded correctness reward. We next experimented with a graded reward (\(+1\) if the opponent predicts the exact output, \(+0.5\) if it is partially correct, \(-1\) otherwise). This objective incentivized the generator to produce overly easy instances. Because the opponent could obtain nontrivial reward through weak heuristics or shallow guessing, the generator received limited pressure to increase difficulty, resulting in a degenerate equilibrium of low-complexity problems.

Rewards with compilability bonuses. Finally, we explored more heavily shaped rewards that separately incentivized compilability and correctness, e.g. \(r = \mathbb 1[\text{compiles}] + \mathbb 1[\text{correct output}]\), with additional intermediate bonuses for partial progress. While this design was intended to improve credit assignment, it proved unnecessary: compilation was already strongly correlated with solvability under our binary objective. Moreover, the dense shaping introduced substantial extraneous signal and degraded performance relative to the simpler reward.

Takeaway. Additional reward shaping either destabilized early training (overly negative feedback), collapsed to trivial instance generation (graded correctness), or injected unnecessary intermediate incentives. The best outcomes were achieved using a minimal reward aligned directly with the desired adversarial outcome.

§EPrompt Templates

§E.1Code-Game Setter Prompt

You are Player in a head-to-head game against another AI called the Opponent.

GAME RULES
- Only YOU write the Python code.
- The Opponent will see your code and try to guess its output number.
- Your goal: generate code whose output you can predict exactly while the Opponent cannot.
- Scoring: +1 if you are correct and Opponent is wrong; -1 if Opponent is correct.

REQUIREMENTS
1. Deterministic Python, standard library only (no randomness, time/date, files, or network).
2. Print exactly one numeric, string, or boolean value and nothing else.
3. On the very next line, output your predicted output in angle brackets, e.g., <42>

EXAMPLE
###
code
###
<output>

NOW PRODUCE YOUR OUTPUT:
###

Your turn (generate a DIFFERENT program):

§E.2Code-Game Opponent Prompt

You are the Opponent in a prediction game.
I will show you a Python script.
Predict *exactly* the value it prints.
Output *only* that value, enclosed in angle brackets.
Examples: <42>, <True>, <False>, <hello>, <3.14>, <None>

```python
{code}
```

§E.3Math-Game Setter Prompt

You are competing against a rival AI. Your goal: craft a math problem that YOU can solve correctly but your OPPONENT cannot.
- Scoring: +1 if you are correct and Opponent is wrong;
-1 if Opponent is correct.
Create a tricky math problem. Output ONLY in this format:
PROBLEM: [your math problem]
ANSWER: [the correct answer]

Examples:

PROBLEM: A bat and ball cost $1.10 together. The bat costs $1 more than the ball.
How much does the ball cost in cents?
ANSWER: 5
Your turn (different problem):

§E.4Math-Game Opponent Prompt

Solve this math problem. Give only the final answer, nothing else.

PROBLEM: {problem}

ANSWER:

§E.5Math-Game Grader Prompt

You are a math grader. Is this answer correct?

PROBLEM: {problem}

CLAIMED ANSWER: {claimed_answer}

Respond in JSON format only:
{"correct": true/false, "true_answer": "...", "explanation": "..."}

§E.6Benchmark Prompts and Answer Extraction

MATH
System: You are a math expert. Solve the problem step by step.
Provide your final answer in \boxed{answer} format.
User: {problem}

ARC-Challenge
System: Answer the multiple choice question by selecting the
correct option. Provide your answer as \boxed{letter}.
User: Question: {question}
A. {choice_A}
B. {choice_B}
C. {choice_C}
D. {choice_D}

MBPP
System: Write a Python function to solve the following task.
User: {task_description}
Your code should pass these test cases:
{test_cases}

MATH. We extract answers in this order of priority: a \boxed{...} pattern, then a “the answer is: ...” pattern, then the last numerical value as a fallback. Answers are normalized by removing whitespace, dollar signs, and commas before comparison. Numeric answers are compared with tolerance \(\epsilon = 10^{-6}\).

ARC-Challenge. We extract the selected option letter (A–D) from a \boxed{[A-D]} pattern, then an “answer is [A-D]” pattern, then the first standalone letter A–D as a fallback.

§FHyperparameters

§BibTeX

@misc{amjith2026sage,
  title  = {SAGE: Self-play Adversarial Games Enhance Large Language Model Reasoning Capabilities},
  author = {Amjith, Saraswathy and Wang, Michael and Lynch, Jayson and Gundlach, Hans and Thompson, Neil},
  year   = {2026}
}