KL Divergence Explains Why Your Local LLM Performs Worse Than the Original

⏱️ Lectura: 12 min

You ran the same Qwen3.6-27B checkpoint on your machine and on the lab’s official demo, and the answers don’t even come close to matching. It’s not your imagination or a broken model: it’s the numerical signature of the inference stack you built, and it has a technical name, KL divergence.

📑 En este artículo

1. TL;DR
2. Introduction
3. What Happened
4. Background and History
5. Technical Details and Performance: Measuring KL Divergence
6. How to Start Measuring It
   1. Linux (Recommended, Native CUDA Support)
   2. macOS (Apple Silicon, No CUDA Acceleration)
   3. Windows (via WSL2)
7. Impact and Analysis
8. What’s Next
9. Frequently Asked Questions
   1. What is KL divergence applied to an LLM?
   2. Why does my quantized model in Ollama feel worse than the lab’s official demo?
   3. Does a low KLD guarantee the quantized model is just as smart?
   4. Which attention backend should I use on my GPU?
   5. Why does my model get stuck inside a think tag without finishing?
   6. Are three-prompt, zero-temperature benchmarks useful for evaluating a local LLM?
10. References

An analysis published on August 16, 2026 on the Level1Techs forums breaks down, layer by layer, why two instances of the same model (same architecture, same weights) produce different tokens depending on the attention backend, quantization, and sampler configuration.

TL;DR

• The Level1Techs forum published an analysis on August 16, 2026 about why a local LLM performs differently than the original.
• KL divergence measures how far the token probability distribution of your inference deviates from a baseline.
• The experiment used Qwen3.6-27B in BF16 on an RTX PRO 6000 Blackwell with tensor parallelism 1 and no KV cache quantization.
• The nightly vLLM container tested included 734 packages, 252 of them installed via uv/pip in Python.
• Each attention backend uses different CUDA kernels depending on the GPU’s compute capability, which changes prefill precision.
• A sampler temperature that’s too low can leave models like Qwen stuck in loops inside the think tag.
• Comparing with three prompts at zero temperature doesn’t represent agentic tasks: long-context and tool-calling evaluations are needed.

Introduction

KL divergence (Kullback-Leibler divergence) is the technical metric behind this phenomenon: it measures how far your local inference’s probability distribution strays from a reference baseline. It doesn’t measure intelligence directly, it measures distance between token distributions.

The analysis starts from a simple premise: every hardware and software combination running an LLM today is different, and that difference begins long before you see the first token. GPUs from mixed generations, different instruction sets, CUDA kernels compiled for specific compute capabilities: all of that changes how the next token gets computed mathematically, even when running the exact same weights.

What Happened

User thr3e posted a series of technical experiments in the Machine Learning, LLMs, & AI category of the Level1Techs forum, designed to isolate, one by one, the points where a local implementation diverges from what the lab that trained the model (what the author calls the reference implementation) reports in its original benchmarks.

The base experiment ran the official BF16 checkpoint of Qwen3.6-27B on an RTX PRO 6000 Blackwell GPU, with tensor parallelism at 1 and no quantization whatsoever of weights, activations, or KV cache. To eliminate variables, the author disabled CUDA graphs, prefix caching, and MTP (multi-token prediction), and forced execution in eager mode.

The software stack used was a nightly build of vLLM, pinned to a specific version so the results would be reproducible. That detail matters: the author himself measured that the nightly vLLM container image he downloaded shipped with 734 packages, of which 252 are Python packages installed via uv/pip. That’s 734 distinct codebases, each with its own bugs and undocumented behaviors, sitting between your prompt and the text you see on screen.

Background and History

To understand what KL divergence measures, you first need to understand what a logit is. A logit is the raw score the model assigns to each possible token as a candidate for the next one. Those scores get normalized into probabilities with a softmax function, pass through a configured sampler (temperature, top-p, top-k), and the detokenizer converts them back into readable text, token by token.

Kullback-Leibler divergence is a 1951 information theory concept that quantifies how far one probability distribution strays from another taken as a reference. Applied to an LLM: the output logits get converted into a probability distribution, and the measurement is how far that distribution is from the one produced by the reference checkpoint at the same position in the text.

A detail the analysis highlights: KL divergence is directional. The order of the two distributions you’re comparing matters, reversing it doesn’t give you the same number. And a lower KLD means closer to the chosen baseline, not smarter: those are two different claims that often get conflated.

⚠️ Heads up: don’t confuse a low KLD published on a quantized model’s card with the same quality as the original. Without knowing the exact reference checkpoint, the full execution environment, the evaluation text, the calibration data, the context length, the sampled positions, the direction of the calculation, and how the measurements were aggregated, that number can’t be interpreted.

Technical Details and Performance: Measuring KL Divergence

During prefill (the initial processing of the full prompt), the inference engine chooses among several available attention backends. That choice isn’t cosmetic: each backend uses different CUDA kernels, compiled for specific architectures and compute capabilities, and that affects both the speed and the numerical precision of the result.

Backend	When to Use It	Advantage	Limitation
FlashAttention-2	Ampere, Hopper, or Blackwell GPUs with native support	Faster prefill with good numerical precision	GPU-family-specific kernels, doesn’t cover all hardware
FlashInfer	Decode with paged KV cache	Good balance between speed and precision during generation	Must be compiled against your exact CUDA version
xFormers	Older GPUs without FlashAttention support	Broad compatibility	Slower prefill and higher memory usage
Eager (native PyTorch)	Debugging and precision validation	Maximum numerical fidelity as a reference	Too slow for production

The Level1Techs analysis ran its first experiment (comparing attention backends by precision) on that unquantized BF16 base, to isolate the backend’s effect from the quantization’s effect. It’s the only honest way to know which of the two variables introduces more noise into your setup.

flowchart TD
A["Input prompt"] --> B["Tokenizer"]
B --> C["Prefill: attention backend"]
C --> D["Raw logits"]
D --> E["Sampler: temperature and top-p"]
E --> F["Detokenizer"]
F --> G["Output text"]

To measure your own divergence against a baseline, you need the raw logits of both models at the same position in the text. A minimal script using the transformers library looks like this:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

modelo_id = "Qwen/Qwen3.6-27B"
tokenizer = AutoTokenizer.from_pretrained(modelo_id)
modelo = AutoModelForCausalLM.from_pretrained(
    modelo_id, torch_dtype=torch.bfloat16, device_map="cuda"
)

entrada = tokenizer("What is a B+ index in a database?", return_tensors="pt").to("cuda")

with torch.no_grad():
    salida = modelo(**entrada)
    logits = salida.logits[:, -1, :]

print(logits.shape)

That snippet loads the reference checkpoint and extracts the logits of the last token. The expected result is a tensor with shape [1, vocab_size]: one score for each possible token in the vocabulary. Repeat the same process with your local model (quantized, with a different backend) and compare both distributions with KL divergence:

import torch.nn.functional as F

log_p_referencia = F.log_softmax(logits_referencia, dim=-1)
p_local = F.softmax(logits_local, dim=-1)

kld = F.kl_div(log_p_referencia, p_local, reduction="batchmean")
print(f"KL divergence at this position: {kld.item():.6f}")

The closer to zero, the more your local distribution resembles the reference one at that specific position. For the number to mean anything, you need to average it over many positions and many prompts representative of your actual use case, not over three random prompts at zero temperature.

How to Start Measuring It

Before running anything, check the model card of the checkpoint you’re going to use on Hugging Face: it usually specifies the recommended sampler settings (temperature, top-p) and the exact chat template. Using values from another model is a common cause of weird behavior.

💡 Tip: if your Qwen keeps spinning inside a think tag and never exits, check the temperature first: a temperature that’s too low can leave the model stuck repeating the same reasoning without converging on an answer.

Linux (Recommended, Native CUDA Support)

python3 -m venv .venv
source .venv/bin/activate
pip install vllm
VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve Qwen/Qwen3.6-27B --tensor-parallel-size 1 --dtype bfloat16

macOS (Apple Silicon, No CUDA Acceleration)

python3 -m venv .venv
source .venv/bin/activate
pip install vllm --extra-index-url https://download.pytorch.org/whl/cpu

Windows (via WSL2)

wsl --install -d Ubuntu
wsl
python3 -m venv .venv
source .venv/bin/activate
pip install vllm

To confirm which attention backend actually ended up active in your process, you don’t need to guess: vLLM prints it explicitly in the server startup log, in a line like Using FlashInfer backend or Using FlashAttention backend. If that line doesn’t match what you set in VLLM_ATTENTION_BACKEND, your GPU doesn’t support that backend and the engine silently fell back to another one.

Impact and Analysis

The practical conclusion of the analysis is uncomfortable but useful: your local implementation will never match the lab’s published benchmark bit for bit, but that’s also true for anyone else who isn’t using the exact same hardware, the exact same software build, and the exact same configuration as that lab. The relevant question isn’t whether it matches the original, but how far off it is and in which direction.

This has direct implications for how quantized model cards on Hugging Face should be read. When a quantization claims an extremely low KLD without publishing the reference checkpoint, the execution environment, the calibration text, and the direction of the calculation, that number can’t be audited. The original analysis says it plainly: methodology matters as much as the number, and plenty of people ignore it.

It also questions a widespread habit: evaluating a model with three prompts at zero temperature and drawing a conclusion about whether it’s good or not. That kind of zero-shot test doesn’t represent a real agentic task, which needs long context, tool-calling, and domain-specific knowledge to expose where your setup actually fails compared to another one running the same weights.

What’s Next

The analysis itself leaves the door open for a second half of the investigation: comparing GGUF quantizations at different bit depths against the already-characterized BF16 baseline, using the same per-position KL divergence methodology. That would make it possible to separate, with your own reproducible data, how much noise the attention backend introduces versus how much noise the quantization itself introduces.

At the community level, the expectation is that more model cards on Hugging Face will start publishing the full methodology behind their KLD measurements (reference checkpoint, environment, calibration data) instead of an isolated number. Without that, any comparison between quantizations from different authors remains, in practice, unauditable.

📖 Summary on Telegram: View summary

Try it yourself: install vLLM with pip install vllm, spin up your favorite checkpoint by setting VLLM_ATTENTION_BACKEND to an explicit value, and compare the logits against the official demo of the same model to see your own KL divergence.

Frequently Asked Questions

What is KL divergence applied to an LLM?

It’s a metric that compares the token probability distribution generated by your local inference, at a given position in the text, against the distribution produced by a reference checkpoint. The closer to zero, the more similar both distributions are at that point.

Why does my quantized model in Ollama feel worse than the lab’s official demo?

Because quantization isn’t the only thing that changes: the attention backend, KV cache precision, default sampler, and chat template all change too. Every layer of that stack diverges a bit from the environment the lab used to publish its original benchmarks, and those small differences add up.

Does a low KLD guarantee the quantized model is just as smart?

No. A low KLD only says the token distribution is close to the chosen baseline. Without knowing the full methodology (reference checkpoint, environment, calibration data, direction of the calculation), the number isn’t interpretable or comparable to another author’s.

Which attention backend should I use on my GPU?

It depends on your architecture and use case: FlashAttention-2 or FlashInfer for recent GPUs with good support, xFormers as an alternative on older hardware, and eager only for debugging because of how slow it is. Always confirm in the startup log which one ended up active.

Why does my model get stuck inside a think tag without finishing?

It’s almost always the sampler temperature set too low for that specific model. Check the model card on Hugging Face and use the temperature and top-p values the model’s own author recommends.

Are three-prompt, zero-temperature benchmarks useful for evaluating a local LLM?

Not in a representative way. Zero-shot benchmarks with few prompts don’t capture real agentic tasks involving long context and tool-calling. It’s recommended to use suites like terminal-bench, HLE, or SWE-bench geared toward your specific use case.

References

• Level1Techs Forums: original analysis Why your local LLM feels dumber than it is, posted by user thr3e on August 16, 2026.
• vLLM Repository on GitHub: source code of the inference engine used in the experiments.
• Kullback-Leibler divergence on Wikipedia: formal mathematical definition of the metric.
• NVIDIA CUDA Programming Guide: documentation on compute capability and per-GPU-architecture kernels.
• Hugging Face Transformers Documentation: reference for the library used to load models and extract logits.

📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Imagen destacada: Foto de Sandip Kalal en Unsplash