Qwen3.5 Reasons Almost as Well as GLM-5.2 but Hallucinates 82% of Facts

⏱️ Lectura: 13 min

GLM-5.2 solves 99.2% of AIME 2026 problems with just 40 billion active parameters per token, a fraction of the roughly 280 billion that GPT-4 was rumored to use in 2023 to barely solve a single problem on the same test. The trick isn’t magic: labs are emptying models’ factual knowledge to make room for pure reasoning.

📑 En este artículo

1. TL;DR
2. Introduction
3. What happened to factual knowledge
4. Context and history
5. Technical details and performance
6. How to test it
7. Impact and analysis
8. What’s next
9. Frequently Asked Questions
   1. Why do AI models memorize fewer facts now?
   2. What is the hallucination rate and how is it measured?
   3. Is a small model useful for coding if it doesn’t know facts?
   4. What is the mixture of experts (MoE) architecture?
   5. Will I be able to run a frontier model on my own GPU?
   6. How do I check if a model’s answer is a real fact or made up?
10. References

Ask those same compact models a hard fact without giving them external tools, and the illusion breaks. Qwen3.5, at 9 billion parameters, hallucinates up to 82% of the time when it doesn’t know the answer, according to Artificial Analysis. The factual knowledge didn’t disappear: it moved outside the weights.

TL;DR

• GLM-5.2 scores 99.2% on AIME 2026 with about 40 billion active parameters per token.
• Qwen3.5 hits 91.3% on AIME 2026 using only 17 billion active parameters.
• DeepSeek V4-Flash reasons with about 13 billion active parameters per token.
• GPT-4 (2023) used an estimated 280 billion active parameters and barely solved AIME.
• On SimpleQA, the leader, Gemini 2.5 Pro, gets only 53% of factual questions right without tools.
• Qwen3.5’s 4B and 9B versions hallucinate between 80% and 82% of the time when they don’t know a fact, according to Artificial Analysis.
• The Physics of Language Models series measures factual capacity at about 2 bits of knowledge per parameter.
• Microsoft’s Phi-4 (14 billion parameters) is strong at math and weak at trivia due to its synthetic training data.

Introduction

Through 2023 and 2024, the AI frontier advanced by brute force: more parameters, better model. That assumption broke. Today labs train smaller models that are more capable at verifiable reasoning, math, and code, at the cost of deliberately sacrificing how much they know by heart about the world. It’s not a side effect or a technical limitation: it’s a design decision.

What happened to factual knowledge

Developer Walter van der Giessen documented the pattern in an analysis published on w4g1.dev: as reasoning scores go up, compute per token goes down. GLM-5.2 reaches 99.2% on AIME 2026 with about 40 billion active parameters per token. Qwen3.5 scores 91.3% with only 17 billion active. DeepSeek V4-Flash reasons with just 13 billion active parameters per token. To put the jump in perspective: GPT-4 ran, according to estimates that circulated in 2023, on roughly 280 billion active parameters and still barely solved one AIME problem.

At the other end of the scale, a quantized 9 billion parameter Qwen3.5 fits in 6 GB of VRAM (a gaming laptop GPU) and doubles the score of the next best model under 10 billion parameters on Artificial Analysis’s intelligence index. If math and code benchmarks were the only criteria, the conclusion would be that models got radically smarter per parameter.

The picture changes completely with a simple factual question and no search tools involved. On SimpleQA, the benchmark for tool-free factual recall, the current leader is Gemini 2.5 Pro with just 53% accuracy. Small models don’t come close: the 4 billion and 9 billion parameter versions of Qwen3.5 hallucinate between 80% and 82% of the time they don’t know the answer, according to Artificial Analysis’s knowledge benchmark. When they don’t know something, which is most of the time, they make up an answer with total confidence.

Context and history

Hard facts take up physical space inside a neural network. The Physics of Language Models research series measured that capacity and arrived at a memorable figure: a model stores, on average, about 2 bits of factual knowledge per parameter. If you want a model that knows the birth year of every minor 19th century mathematician, the population of every Dutch municipality, and the exact argument order of every function in every npm package, you pay for that knowledge in weights. That’s a big part of why frontier models grew to trillions of parameters.

Reasoning compresses much better than facts because it’s, in essence, a small set of procedures that repeat: break a problem into parts, track intermediate state, check your own work, backtrack when a step fails. Distillation and reinforcement learning on verifiable tasks transfer those procedures to small models surprisingly well. Microsoft’s Phi-4 has 14 billion parameters, was trained heavily on synthetic textbook-style data, and is good at math and bad at trivia: the exact outcome of what was, and wasn’t, in its training data. What used to be seen as a limitation of the synthetic data approach is now seen as the design goal.

The knowledge that survives the cut has a particular shape. These models are generalists: they know a little about almost everything and almost nothing in depth. Ask one about PostgreSQL and it’ll tell you what it is, what it’s for, and roughly how MVCC works, but ask which version added a specific planner feature, and it goes back to making things up. That’s the right layer to store in the weights: breadth is what lets the model understand what a question is about, know what to look up, and judge whether a source is credible. Depth is cheap to look up and expensive to store, so it’s the part that gets cut.

Technical details and performance

Model	Active parameters per token	AIME 2026 accuracy	Detail
GLM-5.2	~40 billion	99.2%	MoE architecture, most parameters in knowledge experts
Qwen3.5	17 billion	91.3%	Runs on mid-range hardware
DeepSeek V4-Flash	~13 billion	Not reported in the source	~284 billion total parameters, most in experts
GPT-4 (2023, reference)	~280 billion (rumored)	Low, barely solved AIME	No dedicated optimization for verifiable reasoning

The architecture that makes this cut possible is mixture of experts (MoE): the model has far more total parameters than active ones, and only a subset of experts fires per token. DeepSeek V4-Flash reasons with about 13 billion active parameters, but its experts add up to roughly 284 billion in total. Most of those experts aren’t reasoning engines: they’re fact storage. It’s precisely that part, the knowledge experts, that this design shift makes optional.

There’s an immediate practical consequence: if knowledge lives mostly in the experts and reasoning fits in 13 to 40 billion active parameters, trimming the fact experts brings the model’s total size closer to its active size. A 20 to 40 billion parameter model at 4-bit quantization fits on a 24 GB card, the kind that’s shipped in gaming PCs since 2022. The trade-off is that a compact model like that won’t know, say, a recent fact unless someone tells it.

How to test it

The pattern can be reproduced today with real models running on your own machine, without depending on a frontier lab. The simplest way is Ollama, which packages the model and exposes a local API.

Windows (PowerShell):

winget install Ollama.Ollama
ollama pull qwen2.5:7b
ollama run qwen2.5:7b "Solve step by step: a train leaves at 14:00 at 80 km/h and another leaves 30 minutes later at 100 km/h on the same route, at what time do they meet?"

macOS:

brew install ollama
ollama pull qwen2.5:7b
ollama run qwen2.5:7b "Solve the same train problem step by step"

Linux:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:7b
ollama run qwen2.5:7b "Solve the same train problem step by step"

That same small model, with no external context, will fail if you ask it a specific factual detail it never memorized. The pattern w4g1.dev documents with frontier models reproduces at small scale: good reasoning, poor factual knowledge.

The complement is a harness that hands the model the fact at query time, instead of expecting it to have it memorized:

// harness.js: the model reasons, the search provides the up-to-date fact
import ollama from 'ollama'

async function answerWithFacts(question, factualContext) {
  const prompt = `Verified context:\n${factualContext}\n\nQuestion: ${question}\nAnswer using only the context above.`
  const response = await ollama.chat({
    model: 'qwen2.5:7b',
    messages: [{ role: 'user', content: prompt }]
  })
  return response.message.content
}

const release = await fetch('https://api.github.com/repos/ollama/ollama/releases/latest').then(r => r.json())
const text = await answerWithFacts('What is the latest version of Ollama?', release.tag_name)
console.log(text)

The model never memorized the Ollama version: it receives it in the prompt at query time, so the fact doesn’t go stale even if the model is months old.

To confirm which model and quantization you actually have loaded, ollama show qwen2.5:7b prints the parameter count and quantization detail. To measure the hallucination rate for your own use case, ask the model a verifiable fact that isn’t in the context you gave it (a date, a version, a specific number) and check the answer against the real source. A model used well in this setup should say it doesn’t know or ask for more context, not make up a number with the same confidence as if it actually knew it.

Impact and analysis

This shift has a practical reading for anyone building with coding agents. An agent editing a repository doesn’t need to have memorized a dependency’s API surface: it’s enough to read node_modules or the documentation before calling any function, and its answer stays anchored to the installed version, not to whatever the training corpus knew. Recall that used to be a fixed cost on every forward pass has become an on-demand query.

💭 Key point: the technical frontier is no longer just training bigger models, but deciding what knowledge lives in the weights and what gets resolved at query time against an external source.

This also changes the economics of a model’s aging. A frontier training run takes months and costs hundreds of millions of dollars, and the moment it finishes, the facts it memorized start going stale: APIs change, prices change, people change jobs. A model loaded with facts ages along with its corpus. One loaded mostly with procedures depends far less on its cutoff date, because the current state of the world was never meant to live in the weights.

⚠️ Careful: a model optimized for reasoning answers with the same confidence whether the fact is correct or made up. The confidence of generated text isn’t a signal of factual accuracy, and trusting a small model’s output without external context is the fastest way to publish a false claim.

The real limitation is that this approach shifts the burden of correctness onto the harness. If the search returns a stale source or a retrieval system indexes a document poorly, a model that reasons very well will build an equally convincing answer on equally wrong data. For tasks where there’s no reliable external source at hand, like opinion or synthesizing scattered literature, a small model without deep factual memory is still worse than a frontier model with more knowledge of its own.

What’s next

The logic of this trend, extrapolated a couple of years out, points to a frontier-level reasoning model running on a single consumer GPU. Half the compute is nearly there already: DeepSeek V4-Flash reasons with about 13 billion active parameters per token, within consumer GPU range. What doesn’t fit yet is the other roughly 271 billion parameters living in its experts, and those experts are mostly fact storage.

flowchart TD
A["Model: weights with reasoning"] --> B{"Needs a current fact?"}
B -- "Yes" --> C["Harness: search or tool"]
C --> D["Verified, up-to-date fact"]
D --> E["Final answer"]
B -- "No, pure logic" --> E

That diagram sums up the underlying shift: the model contributes the reasoning, and everything it reasons about is handed to it at query time. The cost of keeping a system like this current no longer lies in retraining a trillion-parameter model every few months, but in keeping the harness (the search index, the documents, the tools) that feeds it up to date.

Try it yourself: run ollama pull qwen2.5:7b and compare it in your own terminal on a math question and a trivia question with no context, you’ll see the same pattern w4g1.dev documents in much larger models.

📖 Summary on Telegram: View summary

Frequently Asked Questions

Why do AI models memorize fewer facts now?

Because reasoning compresses better than factual knowledge: a few general procedures pay off across many problems, while each new fact requires parameters dedicated solely to storing it.

What is the hallucination rate and how is it measured?

It’s the percentage of incorrect answers a model delivers with apparent certainty when it doesn’t know the real fact. Artificial Analysis measures it in its knowledge benchmark, and it reaches 80-82% in the 4 billion and 9 billion parameter versions of Qwen3.5.

Is a small model useful for coding if it doesn’t know facts?

Yes, if it works with a harness: documentation, the repository itself, or search tools. Reasoning is the part a small model retains well; the specific details of an API are looked up in real time.

What is the mixture of experts (MoE) architecture?

It’s a design where the model has far more total parameters than it activates per token: only a subset of experts turns on per query, which allows storing a lot of knowledge in the total without paying that cost on every computation.

Will I be able to run a frontier model on my own GPU?

The trend points that way: if factual knowledge keeps getting trimmed and total size approaches active size, a 20 to 40 billion parameter model at 4-bit already fits on a 24 GB card, something many gaming PCs have had since 2022.

How do I check if a model’s answer is a real fact or made up?

Ask it for the source and check it against the harness: official documentation, an API, or a search. If the model can’t point to where it got the fact, or the context you gave it doesn’t contain it, treat the answer as unverified.

References

• Models Are Getting Dumber on Purpose, Walter van der Giessen: the original analysis with the GLM-5.2, Qwen3.5, DeepSeek V4-Flash, and SimpleQA data.
• Physics of Language Models: the research series measuring factual capacity in bits per parameter.
• Phi-4 on Hugging Face: model card for Microsoft’s model trained on synthetic data.
• Ollama: tool for running language models locally on Windows, macOS, and Linux.
• Mixture of experts, Wikipedia: explanation of the MoE architecture used by the models mentioned.

📱 Enjoy this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day. @programacion

Imagen destacada: Foto de Zach M en Unsplash