Reasoning Models: How They Are Trained

⏱️ Reading time: 16 min

Almost every language model you use in 2026 already thinks before answering. GPT, Gemini, Qwen, and Anthropic’s models offer a reasoning mode that can be turned on or off, and that extra deliberation is what separates a chatbot that completes text from a system capable of solving a complex math problem step by step.

📑 En este artículo

1. TL;DR
2. What Are Reasoning Models?
3. Why It Matters Now
4. How a Reasoning Model Is Evaluated
5. Inference-Time Scaling
6. Reinforcement Learning with Verifiable Rewards
7. Distillation: Compressing Reasoning into a Smaller Model
8. How to Start Experimenting
9. Real-World Use Cases
10. Common Mistakes and Best Practices
11. Comparison: Inference-Time Scaling, RL, and Distillation
12. Going Deeper: GRPO, PRM, and the Cost of Deliberation
13. Frequently Asked Questions
    1. What’s the difference between a reasoning LLM and a regular LLM?
    2. Do reasoning models always need RL?
    3. How do you evaluate whether a reasoning system actually solves the problem?
    4. What is distillation applied to a reasoning engine?
    5. Do you need a large reasoning model to start experimenting?
14. References

Sebastian Raschka, a leading author on building LLMs from scratch, published in October 2026 the book and course that explain how one of these reasoning models is trained: evaluation, inference-time scaling, reinforcement learning, and distillation. This article uses that work as a technical map to explain the full process, with original Python examples.

TL;DR

• Building a reasoner combines evaluation, inference-time scaling, verifiable RL, and distillation into smaller models.
• Inference-time scaling spends more compute when answering, through sampling and majority voting.
• RLVR rewards only the verifiable final answer, the method DeepSeek-R1 used without labeling intermediate steps.
• Distillation transfers the behavior of a large model to a smaller one using a KL loss between their outputs.
• GRPO computes the advantage by comparing answers against each other, without the separate value network that PPO uses.

What Are Reasoning Models?

Reasoning models are LLMs trained, beyond standard pretraining, to generate a chain of intermediate steps before giving the final answer, and to review or correct those steps when they detect an error, instead of directly predicting the most likely next token without deliberating.

The idea isn’t new: chain-of-thought already existed as a prompting technique. What changed is that this deliberation is now trained with reinforcement learning, instead of just being requested through a clever prompt. Through practice, the model learns when to think more and when to answer directly.

Why It Matters Now

The source behind this article sums it up in one line: almost every current-generation LLM is now a reasoning model or offers a reasoning mode that can be toggled on. What started as a specialty for math and code has generalized to other domains: legal drafting, technical diagnosis, long task planning.

Raschka offers a useful analogy for placing these models in the AI agent landscape: the reasoning model is the engine, and the agent harness (the software that manages context and gives access to tools) is the vehicle around it. An agent can use a conventional LLM, but it almost always chooses a reasoner because it handles multi-step tasks better, like booking a full trip or debugging a real bug in a large repository.

That’s why understanding how one of these models is trained isn’t an academic exercise. It’s the piece that determines whether the agent you’re coding this week can follow through on a fifteen-step plan, or gets lost at the third one.

How a Reasoning Model Is Evaluated

Before training anything, you need a way to measure whether the model is improving. Evaluating a reasoner relies on benchmarks with verifiable answers: math problems like GSM8K or AIME, coding exercises with automated tests like LiveCodeBench, logic puzzles with a single correct solution.

The simplest metric is pass@1. The model answers once and is checked against the correct answer. Pass@k is more demanding. The model generates k independent attempts, and it counts as a success if at least one is correct. The gap between the two metrics reveals whether the model fails due to lack of capability or lack of consistency.

The verifier that decides whether an answer is correct is, at bottom, a simple function. For math, it’s enough to extract the final number and compare it against the reference:

import re

def grade_answer(prediction: str, reference: str) -> bool:
    def normalize(text):
        match = re.search(r"-?\d+(?:\.\d+)?", text)
        return match.group(0) if match else text.strip()
    return normalize(prediction) == normalize(reference)

print(grade_answer("The answer is 42.", "42"))

This function prints True: it extracts the first number from each string and compares them. It’s rudimentary on purpose. The same verifier you use to measure pass@1 is, as you’ll see in the next section, the piece that becomes the reward signal for RL training.

A constant risk is data contamination. If the evaluation benchmark (or something very similar) was in the pretraining corpus, the model might recall the answer instead of reasoning through it. That’s why serious benchmarks rotate versions and publish a cutoff date for their questions.

Inference-Time Scaling

The cheapest way to improve a reasoner without retraining it is to let it think more. Instead of adding parameters, inference-time scaling spends more compute at the moment of answering: longer reasoning chains, several attempts in parallel, or a search that explores different paths before deciding.

The simplest technique is multiple sampling with majority voting, known as self-consistency since the 2022 paper by Wang et al. The model generates several independent reasoning chains for the same question and keeps the answer that repeats most often.

from collections import Counter

def self_consistency(answers):
    votes = Counter(answers)
    winner, count = votes.most_common(1)[0]
    return winner

samples = ["42", "41", "42", "42", "40"]
print(self_consistency(samples))

The result is 42: it appears three times out of five samples, so it wins the vote. A more sophisticated variant is best-of-n with a separate verifier or reward model, which doesn’t just count votes but scores each candidate and picks the highest-scoring one, even if it’s a minority.

Other ways to scale at inference time include tree search (like Monte Carlo Tree Search applied to reasoning steps) and budget forcing: forcing the model to extend or cut its chain of thought according to a fixed token budget. None of these techniques change the model’s weights, so they can be applied to any already-trained reasoner, even a closed one you can only access through an API.

flowchart TD
A["User prompt"] --> B["The model generates N reasoning chains"]
B --> C["Verifier or majority vote"]
C --> D["Final answer"]

The limit of this approach is economic, not technical: each additional sample costs tokens, and past a certain point the improvement curve flattens out. If the question is too hard for the base model, no number of attempts will solve it. That’s where RL training comes in.

Reinforcement Learning with Verifiable Rewards

Inference-time scaling doesn’t improve the model, it only makes better use of what it already knows. For a reasoner to gain real capability, its weights have to be adjusted. That’s where reinforcement learning with verifiable rewards (RLVR) comes in, where the same verifier you used for evaluation becomes the reward function.

The core idea is simple: the model (the policy) generates a complete answer, the verifier scores it 1 if correct and 0 if not, and that score pushes the model to increase the probability of generating that kind of answer in the future. No human labels like “this intermediate step is correct” are needed, only the verifiable final result.

DeepSeek-R1 showed in 2025 that this recipe works even without a prior stage of supervised fine-tuning. The version called R1-Zero was trained with direct RLVR on a base model, and reasoning behavior emerged on its own, though with messy writing that mixed languages. The final version, R1, added a fine-tuning stage before RL so the output would be readable for humans.

The classic algorithm for this type of training is PPO (Proximal Policy Optimization), but PPO needs a separate value network that estimates how good each situation is, which doubles the training memory footprint. The DeepSeekMath paper introduced GRPO (Group Relative Policy Optimization). Instead of a value network, GRPO generates a group of answers for the same question and computes each one’s advantage by comparing it against the group average.

import numpy as np

def grpo_advantages(rewards):
    rewards = np.array(rewards, dtype=float)
    return (rewards - rewards.mean()) / (rewards.std() + 1e-8)

group_rewards = [1.0, 0.0, 1.0, 1.0, 0.0]
print(np.round(grpo_advantages(group_rewards), 3))

The output is [ 0.816 -1.225 0.816 0.816 -1.225]. The group’s three correct answers end up with positive advantage and the two incorrect ones with negative advantage, normalized by the group’s own spread. That advantage is the signal used to update the model’s weights, with no separate value network.

sequenceDiagram
participant P as Model (policy)
participant V as Verifier
P->>V: generates candidate answer
V-->>P: reward (0 or 1)
Note over P,V: the gradient adjusts the model's weights

⚠️ Watch out: a poorly designed verifier can be exploited. If the checker only looks for a text pattern, the model learns to produce that pattern without solving the problem: this is reward hacking, one of the most cited risks of RLVR training.

Distillation: Compressing Reasoning into a Smaller Model

Training with RL is expensive: it requires generating thousands of answers, verifying them, and updating weights in a loop. Distillation offers a shortcut: take an already-trained large model (the teacher) and transfer its behavior to a small model (the student), without repeating the entire RL process.

There are two typical variants. The first is pure supervised fine-tuning: you generate thousands of reasoning chains with the teacher, filter them for correctness, and train the student to reproduce them as plain text. This is how several of the open distilled models from DeepSeek-R1 were built, using a small Qwen or Llama as the base.

The second variant is more refined and comes from the original 2015 paper by Hinton et al. Instead of just copying the final text, the student learns to imitate the teacher’s full probability distribution over each token, using a KL divergence loss.

import torch
import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, temperature=2.0):
    student_log_probs = F.log_softmax(student_logits / temperature, dim=-1)
    teacher_probs = F.softmax(teacher_logits / temperature, dim=-1)
    return F.kl_div(student_log_probs, teacher_probs, reduction="batchmean") * (temperature ** 2)

student_logits = torch.tensor([[2.0, 1.0, 0.1]])
teacher_logits = torch.tensor([[3.0, 1.0, 0.1]])
print(round(distillation_loss(student_logits, teacher_logits).item(), 4))

The result is close to 0.1195. The more similar the two distributions are, the smaller this number gets, and training pushes the student to reduce it. The temperature (2.0 in the example) smooths both distributions before comparing them, so the student also learns from the low probabilities the teacher assigns to near-correct answers, not just from the winning answer.

Distillation doesn’t invent new capability, it imitates what already exists in the teacher, so the student never surpasses its teacher in the distilled domain. In exchange, it runs at a fraction of the inference and training cost, which makes it attractive for deploying a reasoner on a phone or on a server without a dedicated GPU.

flowchart LR
A["Large model (teacher)"] --> B["Generates reasoning chains"]
B --> C["Small model (student) imitates those chains"]
C --> D["Student retains most of the capability with fewer parameters"]

How to Start Experimenting

The three examples in this article (the verifier, majority voting, and the distillation loss) run in any Python notebook, without needing a GPU or a large model loaded in memory.

On Linux or macOS, with Python 3.10 or newer installed:

python3 -m venv venv
source venv/bin/activate
pip install torch numpy

On Windows, the only change is activating the environment with venv\Scripts\activate instead of source venv/bin/activate; the rest of the commands are the same.

Save any of the code snippets above into a .py file and run it with python filename.py. Each one prints exactly the output shown in its section.

To go beyond these standalone snippets and build a complete reasoner, using a small pretrained Qwen3 as a starting point, Raschka’s original publication is the most detailed reference: it covers everything from evaluation to RL training in PyTorch, step by step.

Real-World Use Cases

DeepSeek-R1 is the most cited case of RLVR applied at scale, with open weights and a public technical report detailing its training. OpenAI’s o1 and o3 models popularized the toggleable reasoning mode in a commercial product, though without publishing the details of their training method.

Alibaba’s Qwen3 lets you switch between fast mode and reasoning mode depending on task complexity, a product decision rather than a technical limitation: the same model knows when it’s worth thinking more. Anthropic’s Claude offers something similar with its extended thinking, visible as intermediate steps before the final answer.

These cases show that reasoning models are already built with the pieces described above: verifiable evaluation, inference-time scaling, and RL, with distillation to cut deployment costs on small models that run on a phone or on a server without an expensive GPU.

Common Mistakes and Best Practices

Reward hacking is the most frequent risk. If the verifier has a loophole, for example it accepts a format instead of validating the content, the model finds it and exploits it during RL training, instead of learning to solve the underlying problem.

Overthinking is the symmetric mistake: a poorly calibrated reasoner spends thousands of tokens deliberating over trivial questions, which inflates cost and latency without improving the answer. Part of the fine-tuning after RL is specifically aimed at teaching the model when to cut off its chain of thought.

Benchmark contamination, mentioned above, also affects training. If the RL dataset includes questions the model already memorized during pretraining, the measured improvement is illusory. That’s why it’s always worth keeping training data separate from evaluation data and rotating test questions over time.

One last gotcha with distillation: if the teacher’s reasoning chains have silent errors, a poorly justified intermediate step that happens to land on the correct answer, the student learns to imitate those errors too. Filtering traces by final correctness isn’t enough. It’s worth manually reviewing a sample of the intermediate steps before training the student on them.

Comparison: Inference-Time Scaling, RL, and Distillation

Going Deeper: GRPO, PRM, and the Cost of Deliberation

GRPO doesn’t just save on PPO’s value network. It also typically includes a KL penalty term against a reference policy, usually the model itself before RL started, so the trained policy doesn’t drift so far that it starts producing incoherent text just to maximize the reward.

Another important distinction is between outcome reward models (ORM) and process reward models (PRM). An ORM only looks at whether the final answer is correct, like the verifier in this article. A PRM scores each intermediate step of the reasoning chain separately, which gives a richer signal, but requires labeling individual steps, something far more costly than labeling just the final answer.

The contrast between R1-Zero and R1 illustrates the trade-off between purity and usability. R1-Zero, trained with pure RLVR from the base model, developed effective reasoning but with writing that mixed languages and was hard to read. R1 added a fine-tuning stage before RL specifically to fix that readability problem, without losing the capability gained from RLVR.

💭 Key takeaway: the ability to reason didn’t come from adding parameters or new pretraining data. It came from a relatively small post-training stage, but one guided by a verifiable reward signal.

Your next step: take the self-consistency example from this article, replace the list of answers with ones from your own problem, and test how many samples it takes for majority voting to start getting it right consistently.

Frequently Asked Questions

What’s the difference between a reasoning LLM and a regular LLM?

A reasoning LLM generates a chain of intermediate steps before the final answer and was trained with an extra stage, typically RL with verifiable rewards, to improve that deliberation. A regular LLM directly predicts the next token without that explicit step.

Do reasoning models always need RL?

Not necessarily. You can reach similar behavior using only distillation from another model’s chains, but the capability ceiling stays tied to the teacher. Full RL (RLVR) is the path that allows surpassing any available teacher.

How do you evaluate whether a reasoning system actually solves the problem?

With benchmarks that have verifiable answers and metrics like pass@k, which require getting it right in at least one of k independent attempts, not just the first one. This distinguishes real capability from a lucky guess.

What is distillation applied to a reasoning engine?

It’s training a small model (student) to imitate the behavior of a large one (teacher), either by copying its reasoning chains as text (SFT) or by imitating its full probability distribution with a KL divergence loss.

Do you need a large reasoning model to start experimenting?

No. The evaluation and self-consistency examples in this article run on CPU with any small model, or even without a model, as pure verification functions. Only large-scale RL or distillation training needs a GPU.

References

• Sebastian Raschka, Build Your Own Reasoning Model: A Full Video Course: the article and lessons that inspired this piece, with the full training code.
• Sebastian Raschka on GitHub: repositories with the code from his books on LLMs and reasoning from scratch.
• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning: the paper documenting the RLVR training of R1-Zero and R1.
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models: the paper that introduced GRPO.
• Hinton, Vinyals, and Dean, Distilling the Knowledge in a Neural Network: the original 2015 paper on distillation with KL divergence.
• Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models: the 2022 paper that formalized majority voting across reasoning chains.

📱 Enjoy this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Featured image: Foto de Markus Kammermann en Unsplash