Can LLMs get better at math by training on flawed thinking?

§Abstract

Chain-of-thought (CoT) prompting has become central to mathematical reasoning in large language models, yet models remain brittle to early errors: a single arithmetic slip or unjustified inference typically propagates uncorrected to an incorrect final answer. We investigate whether training on intentionally flawed reasoning traces can teach models to detect and recover from such errors without degrading standard problem-solving ability. Using competition-level problems from MATH-lighteval, we generate CoT prefixes containing exactly one controlled error, either a calculation error (sign flips, dropped terms) or a reasoning error (misapplied rules, unjustified logical steps), and fine-tune Qwen3-4B with GRPO using a binary final-answer reward. Our Mixed-CoT-RL model matches standard RL on clean problems (41% vs. 41%) while substantially outperforming it on problems prefilled with flawed reasoning (24% vs. 19%). Notably, clean-only RL fine-tuning degrades robustness below the untuned baseline (19% vs. 20%), indicating that conventional training increases susceptibility to misleading prefills. Among error types, training on reasoning errors yields greater robustness gains than calculation errors alone, with mixed training performing best. These findings demonstrate that exposure to flawed traces during training can improve error-recovery behavior without sacrificing accuracy, suggesting a path toward more robust mathematical reasoning in LLMs.

§1Introduction

Recently, large language models (LLMs) have achieved strong performance across a wide range of reasoning tasks, most visibly on math and coding benchmarks. Much of this progress can be attributed to post-training methods, which fine-tune a base model after it has already been trained. Well-known examples include Reinforcement Learning from Human Feedback (RLHF)¹, which has humans rate responses generated by the model, and Direct Preference Optimization (DPO)², which aligns model outputs with human preferences using a simpler objective and no separate reward model.

A major limitation of existing post-training methods is that they primarily optimize LLMs to produce a correct final answer rather than the correct intermediate chain-of-thought used to derive it. As a result, even reasoning models can be brittle: if an early error is introduced, for example an arithmetic slip or an unjustified logical step, the model typically propagates the mistake rather than correcting it (Figure 1). This is especially problematic in mathematics, where a single local error can invalidate an entire derivation. Ensuring that LLMs can identify and recover from their own mistakes is vital for users who rely on step-by-step solutions for both correctness and understanding.

Can training on flawed thinking teach a model to recover from it?

Reliable step-by-step reasoning has benefits beyond producing correct answers. For mathematics researchers, LLMs can assist by verifying intermediate calculations, exploring alternative solution strategies, and creating derivations more quickly than manual methods. For learners, LLMs can function as interactive study tools: students can follow detailed problem-solving steps, receive immediate explanations for each step, and gain an intuitive understanding of complex concepts.

In this paper, we introduce a simple classification of flawed reasoning types and study how different error families affect a model’s ability to recover from misleading CoTs. For each math problem, we prefill the chain-of-thought with one of two error types and evaluate how the model generates a corrected reasoning sequence and final answer. Calculation errors include sign flips, dropped terms, or incorrect simplifications. Reasoning errors include unjustified inferences, misapplied rules, or broken invariants. We then train models under mixtures of clean and flawed trajectories and evaluate how exposure to each error type influences robustness to misleading prefills.

Our primary contributions are:

• We formulate a process-level training setup for mathematical reasoning that explicitly injects flawed CoTs and encourages the model to recover from them.
• We explore two major families of errors, calculation errors and reasoning errors, and measure how training on each type affects robustness.
• We outline a reinforcement learning pipeline based on a binary reward and GRPO that can be run on modest hardware.

§2Background

Post-training methods such as RLHF¹ and DPO² have significantly improved LLM performance across reasoning tasks. These approaches optimize models to produce preferred outputs but typically supervise only the final answer, leaving intermediate reasoning steps unsupervised. As a result, LLMs can propagate early mistakes in multi-step problems, particularly in mathematics, where local errors can invalidate entire derivations³. Recent efforts, including Outcome Reward Models (ORMs) and Process Reward Models (PRMs), attempt to score reasoning steps sequentially, showing that step-level supervision improves faithfulness and interpretability^4,5. However, these methods generally rely on clean reasoning traces and do not explore how exposing models to flawed or misleading steps affects learning.

Peng et al.⁶ introduce the RECAP framework, which strengthens models’ ability to withstand missteps in reasoning. RECAP trains models to recover from flawed reasoning prefixes: short, intentionally incorrect CoTs that precede correct demonstrations. In safety-alignment settings, this process-level supervision reduces vulnerability to jailbreaks, refusal attacks, and adversarial prompts, outperforming standard RLHF- and DPO-style post-training. However, RECAP treats all flawed traces uniformly and focuses on safety and moral reasoning, where failures are discrete and easy to label. This is fundamentally different from mathematical or scientific reasoning, where multiple distinct trajectories can lead to correct or incorrect outcomes, and it is unclear which kinds of flawed traces are most useful for training. The authors explicitly highlight mathematical reasoning as an open challenge.

We take on the challenge of applying the RECAP idea to mathematical reasoning. Our work extends the flawed-prefix framework by separating injected errors into calculation and reasoning types and empirically studying how training on each affects robustness. This connects to a broader line of work on how models detect, correct, or learn from mistakes in mathematical reasoning. S³c-Math encourages models to spontaneously detect and correct step-level errors, improving correctness on arithmetic and algebra problems⁷. LEMA uses incorrect reasoning traces paired with corrected solutions for fine-tuning, mimicking human learning from mistakes⁸. Stepwise verification methods such as StepCo and pedagogically inspired frameworks like PedCoT enable models to evaluate and correct intermediate steps⁹.

Mathematical reasoning in machine learning has also been shaped by new, increasingly complex datasets. RMath¹⁰, MuMath¹¹, and HighMATH¹² reflect the community’s growing interest in complex, multi-step mathematical reasoning beyond arithmetic. None of these datasets evaluates whether models can recover from misleading or subtly incorrect starts, a gap we address with a perturbed benchmark that tests robustness under intentionally corrupted first steps.

Another line of work, orthogonal to ours, enhances mathematical reasoning through hybrid symbolic and neuro-symbolic frameworks. SymbCoT, CoMAT, and PoT integrate symbolic representations or programmatic computation with LLM reasoning, improving logical fidelity and reducing arithmetic errors^13,14,15. Systems such as Logic-LM and Lean-STaR translate natural-language problems into symbolic form, iteratively refining solutions with solver verification^16,17. Beyond LLM-specific methods, research in cognitive science and curriculum learning supports the idea that controlled exposure to errors can enhance learning, which motivates our systematic study of flawed reasoning types in training.

§3Methods

§3.1Problem Setup

Given a query \(q\), the policy model produces a multi-step chain-of-thought \(\mathbf x = (x_1, x_2, \dots, x_T)\), where \(x_t\) is the \(t\)-th reasoning step and \(T\) is the trajectory length. We write \(\mathbf x_{1:t}\) for the prefix up to step \(t\).

Given a dataset of math queries \(\mathcal Q = \{q_1, \dots, q_N\}\), we construct flawed trajectories \(\tilde{\mathbf x}^{(i)}\) by injecting exactly one controlled error at the first step. Errors are categorized as either calculation errors or reasoning errors. A mixture hyperparameter \(\alpha \in [0, 1]\) controls the fraction of training examples that contain flawed prefixes:

where each \(\mathbf x^{(i)}_{\mathrm{train}}\) is clean with probability \(1 - \alpha\) and flawed with probability \(\alpha\). In our runs \(\alpha = 0.5\). This mixture allows the model to learn from both correct reasoning and examples containing subtle errors (Figure 2).

During fine-tuning, the model parameters \(\theta\) are optimized using a binary final-answer reward:

The training objective is thus

where the presence of flawed trajectories encourages the model to learn corrective behavior following corrupted prefixes \(\tilde{\mathbf x}_{1:t}\). Figure 3 shows what the model sees.

The flawed prefixes are generated with this prompt:

Task. Generate a FLAWED mathematical solution for this problem.
Problem. [problem]
Correct Answer (do NOT arrive at this). [correct answer]
Requirements.
  1. Make ONE subtle error (reasoning error, calculation mistake, sign error, or wrong formula).
  2. Continue reasoning logically from that error.
  3. Arrive at a WRONG answer.
  4. Keep it concise (only a single step).
  5. Do not mention there is an error.
  6. Do not include \boxed{} — leave the solution incomplete.
Generate the flawed reasoning (no final boxed answer).

§3.2Models and Training

Our model is Qwen3-4B¹⁸. We fine-tune it using reinforcement learning with GRPO, a policy-gradient method that maximizes expected reward while being more resource-efficient than classical PPO-style RL. We apply Low-Rank Adaptation (LoRA) to the base model with \(r = 128\), \(\alpha_{\mathrm{LoRA}} = 32\) and dropout \(0.05\).

We configure GRPO with a per-device batch size of 2 and 2 gradient accumulation steps, an effective batch size of 4, and sample 2 generations per prompt. The learning rate is \(10^{-5}\), and training runs for 150 steps. We use a temperature of 0.7 and top-\(p\) sampling at 0.9 to encourage diverse yet coherent generations. The KL penalty coefficient is set to 0, allowing the model to deviate freely from the reference policy, a choice motivated by our goal of teaching substantively new error-recovery behavior rather than staying close to the base model’s distribution. Maximum completion and prompt lengths are 1024 and 2048 tokens.

Reward design. Learning is driven solely by the correctness of the final answer (Eq. 2). We initially considered an LLM-as-a-judge that gave partial rewards for preliminary steps. In two early runs comparing the judge with the binary reward, accuracies were 31% and 34% respectively, so we kept the binary reward.

§3.3Verification and Evaluation

During training we record the model’s intermediate reasoning for logging and analysis, but these traces are not used for computing reward. The model is required to mark its final answer in a canonical format (\boxed{...}) to enable reliable extraction and scoring.

Benchmarks. We train on MATH-lighteval¹⁹, a benchmark of 12,500 high-school competition problems (AMC 10, AMC 12, AIME, and more) that stress long-horizon, multi-step symbolic reasoning, selecting a randomized training set of 500 samples. MATH-500 is a subset of 500 problems from MATH-lighteval, which we hold out as our test set. We evaluate with temperature \(T = 0.6\).

We evaluate five models: the pretrained baseline, the untuned Qwen3-4B; Ablation-RL, GRPO on the same problems without flawed CoTs; Mixed-CoT-RL, GRPO on mixtures of clean and flawed trajectories with both error types; Calc-RL, identical but restricted to calculation-error prefills; and Reas-RL, restricted to reasoning-error prefills. All use the same binary reward and the same GRPO hyperparameters.

§3.4Perturbed-Math-100

To evaluate robustness to misleading reasoning, we constructed the Perturbed-Math-100 dataset by generating flawed chain-of-thought prefixes for 100 held-out problems from MATH-500. For each problem, we prompted GPT-4o-mini to produce a single reasoning step containing exactly one subtle error, either a calculation error or a reasoning error, while continuing logically from that mistake without revealing the error’s presence. These flawed prefixes are prepended to the model’s context as an assistant prefill, simulating scenarios where the model must continue from corrupted intermediate reasoning (Figure 4). Models that blindly propagate the injected error will arrive at incorrect answers, while models with robust self-correction will detect the inconsistency and recover toward the correct solution.

§4Results

We report accuracy as the fraction of correctly solved problems out of 100 held-out examples, on standard MATH-500 problems and on Perturbed-Math-100. Treating each problem as an independent Bernoulli trial, we compute standard errors as \(\mathrm{SE} = \sqrt{\hat p (1 - \hat p) / n}\) with \(n = 100\); all error bars represent \(\pm 1\) standard error.

Our primary ablation compares the three Flawed-CoT RL runs with Ablation-RL to isolate the effect of training on flawed reasoning prefixes. If improvements were driven solely by “more RL on math,” Ablation-RL should match or exceed Flawed-CoT RL. A consistent advantage otherwise would indicate that exposure to flawed prefixes improves error-recovery reasoning.

Clean problems. Ablation-RL and Mixed-CoT-RL both reach 41%, ten points above the baseline (31%), ahead of Reas-RL (38%) and Calc-RL (37%). Training exclusively on one error type may slightly impair standard problem-solving, perhaps because the model overfits to detecting that specific error family.

Perturbed problems. Mixed-CoT-RL reaches 24%, against 19% for Ablation-RL. All three Flawed-CoT RL variants are above both the baseline and Ablation-RL, and Reas-RL (23%) is above Calc-RL (21%). Ablation-RL, despite the highest clean accuracy, falls just below the untuned baseline (19% vs. 20%).

§4.1Training Rewards

Figure 6 shows the rolling mean of the training reward for the four RL runs. Since the reward is \(\pm 1\), a rolling mean of 0 means half of the sampled answers were correct. All four runs move within the same band. Over the whole run the clean run averages a higher reward (0.24, against 0.02 to 0.07 for the runs with flawed prefills), as expected: a prompt that opens with a wrong step is harder to answer correctly.

§5Discussion

Our experiments point to one result: training on intentionally flawed chain-of-thought trajectories improves robustness to misleading CoT prefills without sacrificing standard accuracy. On MATH-500, Mixed-CoT-RL achieved 41%, matching Ablation-RL exactly, while on Perturbed-Math-100 it achieved 24% compared to 19% for Ablation-RL. This 5-point improvement in robustness, which comes at no cost to standard performance, suggests that Flawed-CoT training teaches error-recovery behavior that complements rather than competes with general mathematical reasoning ability.

One might expect that training on corrupted reasoning traces would degrade the model’s ability to solve normal problems, either by introducing noise into learned solution strategies or by teaching the model to “expect” errors where none exist. Instead, the model does both: it learns to detect and correct errors when they are present while maintaining its problem-solving ability on unperturbed inputs.

Ablation-RL performed worse than the untuned baseline on perturbed problems (19% vs. 20%). This suggests that standard RL fine-tuning may inadvertently increase susceptibility to misleading prefills. We hypothesize that optimizing purely for final-answer correctness on clean trajectories reinforces the model’s tendency to continue confidently from any given prefix, rather than critically evaluating whether that prefix contains errors.

On perturbed problems, Reas-RL (23%) outperformed Calc-RL (21%), with Mixed-CoT-RL (24%) achieving the best robustness. Training on reasoning errors, such as misapplied rules or unjustified logical jumps, may better prepare models to handle diverse perturbations than training on arithmetic mistakes alone. Reasoning errors require the model to evaluate logical validity rather than simply recompute numerical results, potentially fostering more general critical-evaluation skills.

The combination of maintained standard accuracy and improved robustness has practical implications. Users frequently provide partial or incorrect solutions and ask models to continue or debug them. Students may supply work containing errors and expect the model to identify mistakes rather than propagate them. As LLMs are deployed in agentic pipelines where they iteratively refine their own outputs, the ability to recover from self-generated errors becomes critical: models trained on flawed trajectories may be better equipped to catch and correct their own mistakes during multi-turn reasoning, reducing error accumulation across extended problem-solving sessions.

§5.1Limitations

Our evaluation is limited to 100 problems for both standard and perturbed accuracy, due to compute constraints, so the standard errors are large (about 4 to 5 points). On perturbed problems the error bars of the Flawed-CoT RL variants overlap substantially with those of Ablation-RL, which prevents strong conclusions about the size of the effect. Larger evaluation sets are needed to establish whether the robustness advantage of Flawed-CoT training over standard RL fine-tuning is statistically significant.

The flawed prefixes are generated by GPT-4o-mini and may not fully capture the natural distribution of errors that arise during model generation or in human work. The single-error constraint also limits ecological validity, as real-world mistakes can compound across multiple steps.

Finally, we evaluate on one use case, high-school competition mathematics, and on a single model (Qwen3-4B). How our results generalize to other mathematical domains, reasoning tasks, and model types and scales remains to be established.

§5.2Future Work

Several directions emerge from this work. Scaling the experiments to larger models and datasets could establish whether the benefits of Flawed-CoT training persist, and possibly improve, at scale. Learning strategies that progressively increase the difficulty of the error or introduce multiple errors per trajectory may further improve robustness. Extending the framework to other reasoning domains, including code generation, scientific problem-solving, and formal theorem proving, would test the generality of our findings.

§5.3Conclusion

Training a language model on intentionally flawed mathematical reasoning traces improved its ability to recover from misleading CoT prefills without degrading standard problem-solving performance. Our Mixed-CoT-RL run matched Ablation-RL in MATH-500 accuracy (41%) while outperforming it on perturbed problems (24% vs. 19%). This improvement does not arise from standard RL fine-tuning on clean data alone; indeed, Ablation-RL slightly increases susceptibility to perturbations, suggesting that explicit exposure to errors teaches corrective behavior. These findings motivate further research into Flawed-CoT training for reasoning in other domains.

§References

1. 1Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017. arXiv:1706.03741
2. 2Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, 2023. arXiv:2305.18290
3. 3Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Luyu Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. 2022. arXiv:2211.14275
4. 4Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations (ICLR), 2024. arXiv:2305.20050
5. 5Zhuoyan Wang et al. Beyond the first error: Process reward models for reflective mathematical reasoning. 2025. arXiv:2505.14391
6. 6ShengYun Peng, Eric Smith, Ivan Evtimov, Song Jiang, Pin-Yu Chen, Hongyuan Zhan, Haozhu Wang, Duen Horng Chau, Mahesh Pasupuleti, and Jianfeng Chi. Large reasoning models learn better alignment from flawed thinking. 2025. arXiv:2510.00938
7. 7Yuchen Yan, Jin Jiang, Yang Liu, Yixin Cao, Xin Xu, Mengdi Zhang, Xunliang Cai, and Jian Shao. S3c-Math: Spontaneous step-level self-correction makes large language models better mathematical reasoners. 2024. arXiv:2409.01524
8. 8LeMa Authors. Learning from mistakes makes LLM better reasoner. 2023. semanticscholar.org
9. 9Zhuoxuan Jiang, Haoyuan Peng, Shanshan Feng, Fan Li, and Dongsheng Li. LLMs can find mathematical reasoning mistakes by pedagogical chain-of-thought. 2024. arXiv:2405.06705
10. 10Ziyi Hu, Jun Liu, Zhongzhi Liu, Yuzhong Liu, Zheng Xie, and Yiping Song. RMath: A logic reasoning-focused dataset toward mathematical multistep reasoning tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 34585–34593, 2025. doi.org
11. 11Weilin You et al. MuMath: Multi-perspective data augmentation for mathematical reasoning. In Findings of the Association for Computational Linguistics: NAACL 2024, 2024. aclanthology.org
12. 12Yan Liu, Minghui Zhang, Bojian Xiong, Yifan Xiao, Yinong Sun, Yating Mei, Longyu Zeng, Jingchao Yang, Yang Wang, and Deyi Xiong. HighMATH: Evaluating math reasoning of large language models in breadth and depth. In Findings of the Association for Computational Linguistics: EMNLP 2025, 2025. aclanthology.org
13. 13Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. Faithful logical reasoning via symbolic chain-of-thought. 2024. arXiv:2405.18357
14. 14Joshua Ong Jun Leang et al. CoMAT: Chain of mathematically annotated thought improves mathematical reasoning. 2024. arXiv:2410.10336
15. 15Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. 2023. arXiv:2211.12588
16. 16Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. 2023. arXiv:2305.12295
17. 17Haohan Lin, Zhiqing Sun, Sean Welleck, and Yiming Yang. Lean-STaR: Learning to interleave thinking and proving. 2025. arXiv:2407.10040
18. 18An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. 2025. arXiv:2505.09388
19. 19Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. 2021. arXiv:2103.03874

§BibTeX

@article{amjith2025flawed,
  title   = {Can Large Language Models Improve Accuracy on Mathematical Tasks Using Flawed Thinking?},
  author  = {Amjith, Saraswathy and Dusad, Mihika and Muramalla, Neha and Shah, Shweta},
  journal = {arXiv preprint arXiv:2512.17079},
  year    = {2025}
}