⏱️ Reading time: 14 min
Meta updated its commercial terms on October 30, 2026: starting that date, any business account on Facebook, Instagram, or WhatsApp authorizes the company to crawl its website and use that content to train its AI models, with no option to opt out of the clause if you keep using those platforms. For any developer or outlet that doesn’t want in, learning how to block AI bots stops being a minor technical detail.
📑 En este artículo
- TL;DR
- What does it mean to block AI bots?
- Why blocking AI bots matters in 2026
- How robots.txt works against AI crawlers
- User-agent rules at the server level
- Getting started: blocking AI scrapers with Cloudflare
- Real use cases
- Common mistakes and best practices
- Comparison: robots.txt, server rules, and Cloudflare
- Going deeper: how bots get detected and how they evade blocking
- Frequently Asked Questions
- References
Meta’s crawler isn’t the only one scouring the web every day. OpenAI’s GPTBot, Anthropic’s ClaudeBot, Common Crawl’s CCBot, and Google-Extended also index content to train models. Each one identifies itself with a different user agent, and each respects (or ignores) blocking rules in its own way.
TL;DR
- A user-agent Disallow rule in robots.txt stops GPTBot, ClaudeBot, and CCBot without touching the rest of your traffic.
- Cloudflare blocks AI scrapers with a single switch in the dashboard, without touching the server or the code.
- User-agent rules in Nginx or Apache cut off the request before it reaches the application.
- robots.txt is voluntary: an unscrupulous scraper can simply ignore it, which is why it’s worth adding a server rule as well.
- Starting October 30, 2026, Meta authorizes its bots to crawl commercial sites to train AI.
What does it mean to block AI bots?
Blocking AI bots means applying robots.txt rules, user-agent filters on the server, and edge services like Cloudflare to prevent automated crawlers from AI companies from downloading a site’s content and using it to train or feed generative models.
The difference from blocking a traditional search engine is the ultimate purpose of the crawl. Googlebot indexes your page to show it in search results and sends traffic back to you. An AI scraper copies the text to train a model or to generate answers that compete directly with your site, with no link or return visit.
Not all AI crawlers are the same. Some, like GPTBot and ClaudeBot, publish their user agent and respect robots.txt because their companies publicly document how they crawl. Others, used by less transparent actors, rotate their user agent or simply ignore the file. That’s why none of the techniques in this article work alone at a hundred percent: they’re meant to be combined.
Why blocking AI bots matters in 2026
Meta’s case is the most recent and the broadest. Under its new commercial terms, in effect since October 30, 2026, the company can use automated software such as web scrapers, bots, robots, spiders, and crawlers to access, index, and analyze any domain linked to a business account, and use that content to train its AI products. There’s no way to opt out of just that clause: the alternative is to stop using Facebook, Instagram, or WhatsApp for commercial purposes.
Intellectual property attorney Matt Topic, of Loevy & Loevy, called the clause a remarkable provision to bury in terms of service, since it gives Meta access to material that would cost millions of dollars in a licensing market, as he told Nieman Lab. James Grimmelmann, a digital law professor at Cornell, went further: under the broadest reading, Meta could build a complete archive of all that content and serve it later, directly or via AI, even in competition with the outlet that originally published it.
This pattern isn’t new. X, formerly Twitter, restricted unauthorized scraping in its terms back in 2023, only to later sell that same access to Grok; LinkedIn has sued several companies over the past few years for scraping profile data. What’s different about Meta’s case is the scale: any business with a Facebook page falls under the clause, with no individual notice and no option for partial opt-out.
Meta isn’t the exception, it’s the norm becoming explicit. OpenAI, Anthropic, Google, Common Crawl, and ByteDance all operate their own crawlers for the same purpose: feeding language models with fresh text from the web. The difference is that most of those companies publish their user agent and respect, at least in theory, the robots.txt standard; Meta still hasn’t specified the exact user agent its crawler will use under the new terms.
How robots.txt works against AI crawlers
robots.txt is a plain text file that lives at the root of your domain (yoursite.com/robots.txt) and tells any bot which parts of the site it can or can’t visit. The standard, formalized as the Robots Exclusion Protocol, is a convention: the bot has to read it and choose to obey it. There’s no technical mechanism that forces compliance.
Most AI crawlers document their own user agent so you can block them selectively. The most common ones in 2026 are GPTBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), CCBot (Common Crawl, used to train many open models), Google-Extended (controls Gemini training without affecting Google’s normal indexing), Bytespider (ByteDance), and PerplexityBot (Perplexity AI).
A robots.txt file that blocks those specific crawlers while letting everyone else through looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: *
Allow: /Each User-agent block targets one specific crawler; Disallow: / forbids that user agent from accessing the entire site. The final block, User-agent: * with Allow: /, lets everyone else through, including Googlebot, so your site keeps showing up in search results. If you upload that file to yoursite.com/robots.txt and visit it in your browser, you should see that exact text; there’s no need to restart anything, bots re-read it on every visit.
The standard is case-insensitive for the user-agent name, so User-agent: gptbot and User-agent: GPTBot are equivalent for any bot that implements the protocol correctly. What does matter is the exact path after Disallow: / blocks the entire site, while /blog/ only blocks that folder.
flowchart TD
A["AI bot visits yoursite.com"] --> B["Reads /robots.txt"]
B --> C{"Is its user agent in Disallow?"}
C -->|"Yes, and it respects it"| D["Does not crawl the site"]
C -->|"Not listed"| E["Crawls the content"]
C -->|"Yes, but ignores it"| E⚠️ Heads up: robots.txt is a signal, not a lock. A badly behaved bot can read the file and crawl anyway; for those cases you need to block at the server or CDN level, not just declare the rule.
User-agent rules at the server level
When a crawler doesn’t respect robots.txt, the next layer is rejecting the connection before it reaches your application, by comparing the User-Agent header of every HTTP request against a blocklist. This runs in Nginx or Apache, so it applies no matter what language your backend is written in.
In Nginx, inside the server block of your configuration (usually at /etc/nginx/sites-available/yoursite.conf on Linux), add:
if ($http_user_agent ~* "(GPTBot|ClaudeBot|anthropic-ai|CCBot|Bytespider|PerplexityBot)") {
return 403;
}The regular expression ~* is case-insensitive. Any request whose User-Agent contains one of those names gets a 403 Forbidden before Nginx even passes the request to PHP, Node, or Python. After saving, test the syntax with nginx -t and reload with systemctl reload nginx.
In Apache, the equivalent goes in a <VirtualHost> block or in a .htaccess file at the root of the site:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|anthropic-ai|CCBot|Bytespider|PerplexityBot) [NC]
RewriteRule ^ - [F,L]The [NC] flag makes the comparison case-insensitive, and [F,L] forces a 403 and stops any further rewrite rules. You need the mod_rewrite module enabled (a2enmod rewrite on Debian/Ubuntu) before this will work.
To confirm the rule is active, simulate a bot visit with curl from any Linux or macOS terminal (on Windows, with PowerShell, swap the single quotes for double quotes):
curl -A "GPTBot" -I https://yoursite.com/If the rule works, the first line of the response should say HTTP/2 403 instead of the HTTP/2 200 you’d get with a regular browser.
Getting started: blocking AI scrapers with Cloudflare
If your site already runs through Cloudflare, you don’t need to touch robots.txt or the server configuration at all: the block is activated from the dashboard. Cloudflare maintains a managed list of known AI user agents and updates it on their end, so you don’t have to keep your own bot list up to date every month.
The steps, documented in Cloudflare’s bot panel, are:
- Go to
dashboard.cloudflare.comand select the domain. - Go to Security → Bots (on some plans it appears as Security → Settings).
- Turn on the AI Scrapers and Crawlers control (also known as “Block AI Bots” in older versions of the panel).
- Save the changes: the rule goes live immediately, no DNS propagation or restart needed.
Under the hood, that toggle adds a managed rule that compares the user agent and, on paid plans, also the client’s network fingerprint, something a simple Disallow in robots.txt can’t do. Cloudflare documented this feature on its blog as a way for any site, without touching code, to stop AI training crawlers.
sequenceDiagram
participant B as AI Bot
participant CF as Cloudflare
participant O as Origin server
B->>CF: requests page with User-Agent GPTBot
CF->>CF: compares against AI Scrapers list
alt user agent blocked
CF-->>B: 403 Forbidden
else user agent allowed
CF->>O: forwards the request
O-->>CF: responds with content
CF-->>B: delivers the page
endReal use cases
A personal blog that doesn’t want its content ending up quoted without attribution in chatbot answers can solve everything with a ten-line robots.txt: it doesn’t have enough traffic to justify Cloudflare or server access to touch Nginx.
A mid-sized news outlet, facing the scenario Nieman Lab describes about Meta, needs the server or CDN layer: a bot that trains models to compete with the outlet itself has a strong incentive to ignore a voluntary convention, so the rule has to be technical, not just declarative.
An online store that depends on showing up in Google Shopping and regular search results needs extra care: overblocking, for example with a generic Disallow: / that doesn’t specify a user agent, also kicks out Googlebot and kills organic traffic. The rule should always target one specific user agent, never all of them.
Common mistakes and best practices
- Blocking with User-agent: * by mistake: a generic Disallow without specifying the bot also kicks out the search engines that send you traffic.
- Relying only on robots.txt: it works with crawlers that document and respect it, but it won’t stop a scraper that deliberately rotates its user agent.
- Forgetting API subpaths: a bot can skip the HTML page and hit your REST or GraphQL API directly if it doesn’t have the same blocking rule.
- Not checking the server logs: without looking at which user agents actually visit you, it’s easy to block bots that no longer exist and miss a new one.
- Caching the 403 response: if your CDN caches that response for the wrong path, you can accidentally block real users who share the same generic user agent as an old browser.
Comparison: robots.txt, server rules, and Cloudflare
| Option | When to use it | Advantage | Limitation |
|---|---|---|---|
| robots.txt | Bots that document their user agent and respect the standard | Zero infrastructure, edited in seconds | It’s a convention: nothing forces anyone to obey it |
| Server rules (Nginx/Apache) | You have root access or access to the server configuration | Real network-level blocking, impossible to ignore | Requires manually maintaining the user-agent list |
| Cloudflare AI Scrapers and Crawlers | Your site already runs through Cloudflare | Managed list, one switch, no code to touch | Depends on a third party and which plan you have active |
Going deeper: how bots get detected and how they evade blocking
User-agent blocking assumes the bot identifies itself honestly. Nothing stops a scraper from sending a User-Agent header identical to Chrome on Windows; in that case, neither robots.txt nor a user-agent rule will catch it, because from the server it’s indistinguishable from a real human visit.
That’s where network fingerprinting, offered by services like Cloudflare, comes in: instead of just looking at the declared header, they analyze patterns in the TLS connection, the order in which HTTP headers arrive, and browsing behavior (whether it executes JavaScript, loads images, respects cookies) to estimate whether the client is a bot even if it lies about its identity. It’s a heuristic, not a certainty, which is why these systems also occasionally generate false positives against uncommon browsers or aggressive VPNs.
An additional layer, beyond the practical scope of this article but relevant for large sites, is the proof-of-work challenge: instead of blocking directly, the server asks the client to solve a brief calculation before serving the page. A real browser solves it in milliseconds without the user noticing anything; a scraper making thousands of requests per second sees its compute cost multiplied, which discourages mass crawling without fully shutting off access.
Grimmelmann warns, in his reading of Meta’s new terms, that none of these three techniques solves the underlying problem: a technical block stops the bot, but it doesn’t erase the contractual permission you already signed when you accepted the terms of a business account. The only way to revoke that permission, under the current terms, is to stop using the account.
flowchart TD
A["Request from a bot"] --> B["Layer 1: robots.txt"]
B --> C["Layer 2: user-agent rule on the server"]
C --> D["Layer 3: Cloudflare or CDN with fingerprinting"]
D --> E["Blocked or allowed"]💡 Tip: before blocking anything, check your server’s access logs for a week and filter by the User-Agent field. You’ll see, by name, exactly which of these bots actually visit you, instead of blocking a generic list you downloaded from an article.
Your next step: review last week’s server logs, filter by the User-Agent header, and build your own robots.txt with the bots that actually visit you before copying a generic list.
Frequently Asked Questions
Does robots.txt actually block GPTBot and ClaudeBot?
Yes, as long as OpenAI and Anthropic keep publicly documenting those user agents and configure their crawlers to respect the standard, as they do today. The risk is that a company’s policy changes or that a new bot doesn’t identify itself the same way.
Does Cloudflare block AI scrapers on the free plan?
Yes, the AI bot control is available for all domains using Cloudflare, including the Free plan; paid plans additionally add more granular network fingerprinting.
What happens if an AI crawler ignores robots.txt?
There’s no automatic consequence: robots.txt isn’t a technical barrier. If you genuinely care about keeping that bot out, you need to add a server or CDN rule that rejects the connection based on user agent or behavior.
Do user-agent rules affect Google SEO?
No, as long as the rule targets a specific user agent like GPTBot or CCBot. The risk of hurting SEO only appears if you use a generic Disallow: / that also shuts the door on Googlebot.
Does Meta-ExternalAgent respect robots.txt rules?
Meta documents its AI crawler under that name and says it respects robots.txt for that specific user agent, but the new commercial terms don’t specify whether that changes for the crawling they enable from business accounts starting October 30, 2026.
Should you block all AI scrapers or just some?
It depends on the site. An outlet that lives off attention and advertising makes sense blocking everyone that doesn’t send traffic back. An open source project that wants its documentation to feed code assistants might prefer to selectively let some through.
References
- Nieman Lab: coverage of Meta’s new commercial terms and their legal impact on news outlets.
- Google Search Central: official documentation of the robots.txt standard (Robots Exclusion Protocol).
- Cloudflare Developers: documentation of the bot control and the AI Scrapers and Crawlers block.
- Cloudflare Blog: original announcement of the one-click AI bot blocking feature.
- Wikipedia: history and general specification of the robots.txt file.
📱 Do you like this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day. @programacion
Featured image: Foto de Mohamed Nohassi en Unsplash
Did it work for you? Got a different error? Say so below: questions get answered and help the next reader.
Leave a comment
0 Comments