⏱️ Lectura: 12 min
On git.kernel.org there are now fourteen CPU cores, spread across five geodistributed nodes, dedicated full-time to a single task: converting Linux kernel commits into HTML that no human will ever read. This was revealed by Konstantin Ryabitsev, the administrator of the kernel’s git infrastructure, on August 29, 2026, with a figure that sums up the underlying problem: the platform spends more processor cycles serving AI crawlers than on all the rest of the legitimate traffic combined, git clones included.
📑 En este artículo
- TL;DR
- What happened
- Why git.kernel.org is a perfect target for AI crawlers
- Context and history: from banning user-agents to pretending to be your TV
- Technical details and performance: Anubis and the difficulty race
- How to try Anubis on your own server
- Impact and analysis
- What’s next
- Frequently Asked Questions
- What is Anubis and why does it use proof-of-work?
- Why don’t crawlers just clone the repository with git clone?
- What is the ‘proxy SDK monetization’ the article mentions?
- How many commits and forks does git.kernel.org have to handle?
- Can I install Anubis on my own Git server or wiki?
- What other open source projects faced the same problem?
- References
This isn’t an isolated anecdote. It’s the latest chapter in a war of attrition that has been going on for over a year between open source infrastructure administrators and the AI crawlers that feed language models.
TL;DR
- On August 29, 2026, Konstantin Ryabitsev published hard numbers on the impact of AI crawlers on git.kernel.org.
- git.kernel.org dedicates 14 CPU cores across 5 geodistributed nodes solely to rendering commits into HTML for scrapers.
- linux.git has close to 1.48 million commits and 922 forks on git.kernel.org, amounting to billions of scannable URLs.
- Bots went from lying about their user-agent to using residential and mobile IPs via proxy SDK monetization.
- Anubis, the SHA-256 proof-of-work proxy, held bots off for months until they started solving difficulty 5.
- cgit allows generating up to 1.2 trillion valid URLs per fork, with diffs and patches between arbitrary commits.
- GNOME and VideoLAN also adopted Anubis to curb the same kind of aggressive AI crawler scraping.
What happened
Ryabitsev published a technical breakdown on his personal blog of what he calls the ‘background radiation’: a constant load of automated requests that consumes compute capacity with no benefit whatsoever to the project. The core figure is stark: git.kernel.org burns more CPU rendering commits into HTML for bots than it does serving all legitimate human and automated traffic combined.
The Linux repository (linux.git) has accumulated close to 1.48 million commits. Add to that 922 forks hosted on git.kernel.org, which on the backend share almost all the same git objects (storage is efficient), but which for a scraper represent 922 separate copies of the same history to crawl from scratch.
The result, according to Ryabitsev, is that a single fork of linux.git can generate on the order of 1.2 trillion valid URLs, because cgit, the git web viewer used by kernel.org, doesn’t just let you view each commit individually: it also generates patches, diffs between any arbitrary pair of commits, and flat renders on demand.
Why git.kernel.org is a perfect target for AI crawlers
For a language model, the Linux history is a goldmine of training data for a very specific reason: it is guaranteed to be free of AI-generated content. All kernel development happens in the open, from cloneable repositories to mailing list archives, and much of that history predates the LLM explosion.
💭 Key point: training a model on text generated by another model produces cumulative degradation, which some researchers compare to a digital prion disease. That’s why a guaranteed ‘pre-AI’ source, like the kernel’s complete commit history, is worth its weight in gold to any lab training new models.
What’s striking is that the most efficient way to make use of that data already exists: clone the repositories with git clone and walk through the history commit by commit, locally, without touching the project’s servers more than necessary. That’s exactly what Ryabitsev suggests in his post: you can clone the entirety of LKML and then do whatever you want with it. But crawlers, in their vast majority, choose the most expensive route: asking cgit to render every commit as an HTML page, one by one, and then parsing that HTML.
Context and history: from banning user-agents to pretending to be your TV
Kernel.org’s initial response was the simplest one: check the logs, identify which IPs behaved like obvious bots, and ban them with fail2ban. At first it worked because the bots identified themselves with their own user-agent in the HTTP header.
Then the bots started lying, posing as common browsers. The team then switched to banning by IP: it was easy to detect that an address requesting every possible commit of a fork abandoned eight years ago wasn’t a Chrome user clicking around manually. When the bots spread across entire subnets rented from cloud providers, kernel.org started banning the whole ASN, even knowing that would occasionally catch a legitimate user automating link verification in commits.
The breaking point came when the crawlers started operating from millions of distinct residential and mobile IPs, each one simulating a modern browser, making just 4 or 5 requests before disappearing from the logs forever. Banning those addresses was pointless: by the time the team detected the pattern, the bot was already gone and wasn’t coming back from that same IP. Ryabitsev describes it as a swarm of locusts: they hit hard and fast until they bring the system down, move on to the next target, and come back later to repeat the cycle.
That traffic comes from what the industry calls ‘proxy SDK monetization’: companies that pay app developers, including smart TV app developers, to integrate an SDK that turns the end user’s device into a node in a residential proxy network, without that end user knowing their TV or phone is being used to crawl git.kernel.org at three in the morning.
Technical details and performance: Anubis and the difficulty race
About a year ago, kernel.org tried a different strategy: instead of identifying and blocking, change the economics of the attack. The idea is simple: force every client to solve a disposable math problem before letting it through, something cheap for an occasional human and expensive for a scraper that needs to complete billions of requests.
That’s the logic behind Anubis: a reverse proxy that puts a proof-of-work challenge in front of every new visitor. The challenge consists of finding a string that, combined with the client’s IP and a server secret, produces a SHA-256 hash with a certain number of leading zeros. The more zeros the server requires (the ‘difficulty’), the more brute-force attempts the client needs to find a valid solution.
Once deployed, the effect was immediate: the bots simply gave up and migrated to easier targets. For several months there was relative peace: legitimate users noticed a minimal, tolerable delay, and the Anubis stack was easy to deploy in front of any service behind a reverse proxy.
The peace didn’t last. Months later, the bots came back solving difficulty 4. Kernel.org bumped it to difficulty 5, which on a mobile device means several seconds of intensive computation and, as Ryabitsev reports, noticeably heats up the phone while it solves the challenge. It worked again, for a while. But the crawlers ended up solving difficulty 5 too.
| Strategy | When to use it | Advantage | Limitation |
|---|---|---|---|
| User-agent blocking | Bots that identify themselves honestly | Trivial to implement | Useless as soon as the bot spoofs the header |
| IP blocking | Bots from fixed, repeat-offender addresses | Effective against unsophisticated scrapers | Doesn’t work if the bot never reuses the same IP |
| ASN blocking | Massive traffic from a single cloud provider | Cuts off entire ranges at the root | Can also block legitimate users on the same provider |
| Anubis (proof-of-work) | Bots with rotating residential proxies, impossible to list | Raises the cost of every request without relying on blocklists | Every difficulty increase also penalizes users on modest mobile devices |
flowchart TD
A["Client (browser or bot)"] --> B["Anubis: proof-of-work challenge"]
B -->|"solves sha256 with N zeros"| C["cgit / git.kernel.org"]
B -->|"doesn't solve or gives up"| D["Blocked, no access"]
C --> E[("linux.git commits")]How to try Anubis on your own server
If you administer a git server, a wiki, or any service with content of interest to an LLM, you can deploy Anubis today in front of your backend, without touching the protected application’s code. Anubis runs as a standalone reverse proxy, so you just need to redirect traffic to it instead of directly to the service.
docker run -d \
--name anubis \
-p 8923:8923 \
-e BIND=:8923 \
-e TARGET=http://localhost:80 \
-e DIFFICULTY=5 \
ghcr.io/techarohq/anubis:latestThat command starts Anubis listening on port 8923, forwarding already-filtered traffic to your real backend on port 80, at difficulty 5. Adjust DIFFICULTY based on how much automated traffic you’re seeing: raising it too fast punishes your legitimate mobile users before it slows down the most persistent bots.
Then, in your edge proxy (nginx, Caddy, or whatever you use), point the server_name at the Anubis port instead of the original backend:
server {
listen 443 ssl;
server_name git.mydomain.org;
location / {
proxy_pass http://127.0.0.1:8923;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}To confirm Anubis is actually intermediating the traffic, request the page with curl and check the response headers: a new visitor without a valid cookie should receive the embedded JavaScript challenge, not the site’s real content.
curl -sI https://git.mydomain.org | grep -i set-cookieIf you see an Anubis-related cookie in the response, the proxy is active and filtering before traffic reaches the backend.
Impact and analysis
Kernel.org’s case isn’t unique. GNOME and VideoLAN went through the same problem and also ended up adopting Anubis to curb aggressive AI bot scraping on their infrastructure, a pattern that confirms the phenomenon doesn’t depend on the tech stack or the project’s size: any open source repository with an extensive history is a target.
The real cost isn’t just the compute bill. It’s the engineering time that a project maintained largely by volunteers has to spend on an arms race against proxy SDK monetization companies that profit from the bandwidth of other people’s smart TVs. That time isn’t spent reviewing kernel patches or improving infrastructure for real developers.
⚠️ Heads up: raising Anubis’s difficulty improves blocking, but it also worsens the experience for legitimate users on mobile devices, whose phone noticeably heats up while solving the challenge.
There’s also an underlying economic asymmetry: for the crawler operator, each request costs practically nothing because it runs on someone else’s residential device, paid for by a third party. For kernel.org, each request costs real CPU, on real servers, with a real bill. Anubis’s proof-of-work tries to reverse that asymmetry by shifting the cost back to the client, but it only works as long as it’s cheaper for the bot to give up than to solve the challenge.
What’s next
The next obvious move, raising to difficulty 6, is already on the table, but each additional step punishes legitimate users on modest hardware more before it deters the better-funded scraper. Ryabitsev doesn’t offer a definitive solution in his post: he documents a problem that the open source infrastructure community is still solving in real time.
It’s likely we’ll see more projects migrate from reactive blocklists to computational-cost mechanisms like Anubis, and that the proxy SDK monetization industry itself becomes a target of pressure among infrastructure operators, similar to how malicious IP lists are already shared today.
📖 Summary on Telegram: See summary
Try it yourself: clone linux.git with git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git and compare how long that single operation takes against the load it would impose on a server to crawl every commit one by one over HTTP.
Frequently Asked Questions
What is Anubis and why does it use proof-of-work?
It’s an open source reverse proxy that puts a cryptographic challenge in front of every new visitor before letting them reach the real service. Solving the challenge is cheap for an occasional user, but expensive at scale for a scraper that needs to complete millions of requests.
Why don’t crawlers just clone the repository with git clone?
That would be the most efficient method, but most of the bots Ryabitsev analyzed prefer to ask cgit to render every commit as an HTML page and then parse that HTML, a much more expensive method for both the bot and the server.
What is the ‘proxy SDK monetization’ the article mentions?
It’s a business model in which companies pay app developers, including smart TV app developers, to integrate an SDK that turns the end user’s device into a node in a residential proxy network, later used to crawl websites without the device owner’s knowledge.
How many commits and forks does git.kernel.org have to handle?
linux.git has accumulated close to 1.48 million commits, and git.kernel.org hosts 922 forks of the repository, most sharing the same git objects on the backend but representing separate targets for a scraper.
Can I install Anubis on my own Git server or wiki?
Yes. Anubis runs as a standalone Docker container and sits in front of any HTTP backend through your usual reverse proxy, with no changes required to the protected application.
What other open source projects faced the same problem?
GNOME and VideoLAN publicly documented adopting Anubis to curb aggressive AI crawler scraping on their infrastructure, confirming that the pattern repeats across different open source ecosystems.
References
- Creepy crawlies, Konstantin Ryabitsev: the original post with git.kernel.org’s load figures.
- Anubis repository on GitHub: source code and documentation for the proof-of-work proxy.
- git.kernel.org: the cgit instance hosting linux.git and its forks.
- Web scraping, Wikipedia: general context on automated crawling techniques.
📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day. @programacion
0 Comments