⏱️ Lectura: 10 min
A file compressor and a language model like GPT solve, at their core, the same math problem: predicting which symbol comes next. That’s the central thesis of a post published on the technical blog of ngrok, the company behind the HTTP tunnel of the same name, which rebuilds data compression from scratch to reach that conclusion.
📑 En este artículo
The finding isn’t new among information theory researchers, but the post explains it with examples accessible to any developer: from the most basic run-length encoding to the arithmetic coding used by compressors like Brotli and zstd.
TL;DR
- Ngrok published a post explaining that data compression and LLMs solve the same problem: predicting the next symbol.
- The post’s run-length encoding example reduces a 224-bit string to 96 bits, a 57% reduction.
- Minifying a 156-character script down to 62 characters (a 60% reduction) doesn’t count as real compression, according to the post.
- Every modern compressor combines three stages: transform, probability model, and entropy encoder.
- Arithmetic coding represents an entire file as a single number within the interval [0, 1).
- The Hutter Prize has, since 2006, rewarded the best compression of a plain-text Wikipedia snapshot.
- gzip, Brotli, and zstd already come installed or natively available on Linux, macOS, and Windows.
What happened
The article, published on the ngrok.com blog, opens with an example of JavaScript code minification: a function that adds up the numbers in a list. The source code takes up 156 characters; after removing comments, whitespace, and shortening variable names to a single letter, the result takes up 62 characters, a 60% reduction. The post clarifies something important right there: minification is almost never mentioned in the field of real data compression, because it only strips syntax that machines don’t need, without exploiting the redundancy in the content.
To show the difference, the post uses the string AAAAAAAAABBBBCCDAAADDDDDDDDD: nine “A”s, four “B”s, two “C”s, one “D”, three more “A”s, and nine more “D”s. Encoded character by character in 8-bit ASCII, that 28-character string weighs 224 bits. If instead each run of symbols is encoded as (character, count), the result is A9B4C2D1A3D9: 12 characters, 96 bits, 57% smaller. That technique is called run-length encoding (RLE), and it’s just the starting point: real compressors like gzip, Brotli, or zstd combine several techniques at once.
Context and history
The idea that compressing is predicting didn’t originate in 2026: it has roots in the information theory that Claude Shannon formalized in 1948. Shannon showed that the minimum number of bits needed to represent a symbol depends on its probability: frequent symbols need few bits, rare ones need more. That relationship is the mathematical foundation of the entire family of modern compressors, from Huffman coding (1952) to the transformers that generate text today.
The explicit connection between compression and language models also has a well-known precedent in artificial intelligence research: the Hutter Prize, created in 2006 by Marcus Hutter, rewards whoever achieves the best compression of a 1 GB plain-text Wikipedia snapshot (enwik9). The premise of the contest is the same one the ngrok post makes: compressing human text near the theoretical limit requires, in practice, modeling language almost as well as a system that understands what it reads.
Technical details: the anatomy of a compressor
According to the post, a modern compressor has three pieces: transforms (preprocessing transformations, like the RLE in the earlier example), models (which assign a probability to each possible symbol), and entropy encoders (which convert those probabilities into the final bitstream). Transforms don’t always reduce the size of the data: sometimes they reorganize the information to create more redundancy, which the model and entropy encoder then exploit.
The model is the piece that decides how much can be compressed. For the example string, each symbol has this probability, calculated by dividing its frequency by the total number of characters:
| Symbol | Occurrences | Probability |
|---|---|---|
| A | 12 | 0.429 |
| D | 10 | 0.357 |
| B | 4 | 0.143 |
| C | 2 | 0.071 |
The entropy encoder takes those probabilities and produces the final bitstream: the more confident the model’s prediction, the fewer bits it needs to represent each symbol. That’s the key that connects compression with prediction: a good predictive model compresses better, whether it’s predicting the next character in a text file or the next token in a conversation.
Arithmetic coding: one file, one number
The method that best illustrates this idea is arithmetic coding. Instead of assigning a fixed or variable-length code to each symbol (as Huffman does), it represents the entire file as a single number within the interval [0, 1). Each new symbol shrinks that interval in proportion to its probability: frequent symbols shrink it a little (they need few extra bits), rare symbols shrink it a lot (they need more bits).
A simplified Python example, calculating the cumulative range for the string ABABAAC with probabilities A=4/7, B=2/7, C=1/7:
simbolos = ["A", "B", "C"]
probabilidades = {"A": 4/7, "B": 2/7, "C": 1/7}
def construir_rangos(probabilidades):
rangos = {}
acumulado = 0.0
for simbolo, prob in probabilidades.items():
rangos[simbolo] = (acumulado, acumulado + prob)
acumulado += prob
return rangos
rangos = construir_rangos(probabilidades)
for simbolo, (inicio, fin) in rangos.items():
print(f"{simbolo}: [{inicio:.3f}, {fin:.3f})")
That snippet prints the initial subinterval for each symbol: A occupies [0.000, 0.571), B occupies [0.571, 0.857), and C occupies [0.857, 1.000). Encoding the entire string means repeating the process symbol by symbol, shrinking the interval each time, until you end up with a single number that represents the whole sequence.
flowchart LR
A["Original data"] --> B["Transform (RLE, BWT, etc.)"]
B --> C["Probability model"]
C --> D["Entropy encoder"]
D --> E["Compressed bitstream"]Comparing compression techniques
| Technique | When to use it | Advantage | Limitation |
|---|---|---|---|
| Run-length encoding (RLE) | Data with long runs of the same symbol (simple bitmaps, faxes) | Very simple and fast | Doesn’t compress data without repeated runs |
| Huffman coding | When each symbol has a fixed, known frequency | Optimal for integer-length bit codes | Can’t use fractional bits per symbol |
| Arithmetic coding | When you need to squeeze out every possible fraction of a bit | Gets close to Shannon’s theoretical limit | More computationally expensive than Huffman |
| Modern compressors (gzip, Brotli, zstd) | General use: logs, HTTP, files, backups | Automatically combine transform, model, and entropy encoder | Gain almost nothing on data that’s already compressed or encrypted |
How to try it yourself
No need to install anything unusual: gzip already lives on most operating systems. You can compare, in minutes, the relationship between compression and how predictable a file is.
Linux:
# Generate a repetitive text file (highly predictable)
yes "the same text over and over" | head -n 5000 > repetitivo.txt
# Generate a random file (not predictable at all)
head -c 200000 /dev/urandom > aleatorio.bin
gzip -k -9 repetitivo.txt
gzip -k -9 aleatorio.bin
ls -la repetitivo.txt.gz aleatorio.bin.gz
macOS: the same commands work as-is, because macOS ships with gzip and /dev/urandom by default in its terminal.
Windows: with PowerShell there’s no need to install anything extra; you can use the system’s native compression:
# PowerShell
"the same text over and over " * 5000 | Out-File repetitivo.txt
Compress-Archive -Path repetitivo.txt -DestinationPath repetitivo.zip
Get-Item repetitivo.zip | Select-Object Length
When you compare the final size, you’ll see that the repetitive file compresses far more than the random one, even though both are the same size uncompressed. The reason is exactly what the ngrok post explains: the repetitive file is easy to predict symbol by symbol, while the random one isn’t predictable at all.
Impact and analysis
The relationship between compression and prediction isn’t just a theoretical curiosity: it explains why language models trained with more data and more parameters tend to compress text better than traditional compressors. An LLM that predicts the next token with high confidence is doing, essentially, the same work as the model inside a classic compressor, just with vastly more context variables.
This has a practical consequence for anyone working with data every day: choosing the right compressor depends on how predictable the content is. Logs with a repetitive format, JSON with fixed keys, or source code with consistent patterns lend themselves much better to data compression than files that are already compressed (JPEG images, video, or the output of encryption), which are already nearly indistinguishable from random noise.
💭 Key point: trying to compress a file that’s already compressed (a .zip inside another .zip) almost never saves space, because there’s no redundancy left for a model to predict.
What’s next
The ngrok post teases a second installment focused on quantization, the technique that applies this same compression logic to shrink large language models without losing too much predictive capacity. It’s the same principle applied in the opposite direction: instead of compressing a file with the help of a model, you compress the model itself.
💡 Tip: if you code in Node.js or Python, you can inspect how much each algorithm shrinks one of your own files using thezlibmodule (Node) orgzip/lzma(Python), without installing anything extra.
⚠️ Heads up: none of these techniques improve compression on data that’s already compressed or encrypted: there, the model has nothing to predict.
Try it yourself: run gzip -9 on a real log from your project and compare the resulting size against any binary file, to see in practice the difference between predictable data and data that isn’t.
📖 Summary on Telegram: View summary
Frequently Asked Questions
What is arithmetic coding?
It’s a compression method that represents an entire file as a single number within the interval [0, 1), instead of assigning a separate code to each symbol. The better the model predicts each symbol, the fewer bits the final number needs.
Why doesn’t minification count as real compression?
Because it only removes syntax that machines don’t need (whitespace, comments, long names), without exploiting the statistical redundancy in the data. Real data compression uses a probability model to predict symbols.
What’s the relationship between compressing data and training an LLM?
Both solve the same math problem: predicting the next symbol (character, byte, or token) from the preceding context. A model that predicts better compresses better, and vice versa.
What is the Hutter Prize?
It’s a contest created in 2006 by Marcus Hutter that rewards whoever achieves the best compression of a plain-text Wikipedia snapshot, under the premise that compressing human text near the limit requires modeling language.
Is it worth compressing the same file twice?
Almost never. A file that’s already compressed statistically resembles random noise: it has none of the redundancy a model needs to predict symbols, so a second compression pass doesn’t save space.
References
- ngrok.com: the original post, “Compression is prediction,” which lays out the relationship between compressors and language models.
- en.wikipedia.org: article on arithmetic coding, the method that represents an entire file as a single number.
- en.wikipedia.org: description of the Hutter Prize, the contest that has rewarded Wikipedia text compression since 2006.
- en.wikipedia.org: biography of Claude Shannon, creator of the information theory that underlies all modern compression.
📱 Like this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.
Imagen destacada: Foto de Markus Spiske en Unsplash
0 Comments