Which AI Crawlers Should You Allow So AI Can Cite You?
Contents
If your goal is to get cited by ChatGPT, Claude, Perplexity, or Google's AI features, the crawlers that matter most are not the ones everyone argues about. Training crawlers like GPTBot and ClaudeBot feed model pretraining and rarely produce a citation on their own. The crawlers that actually fetch a page to answer a specific question, like OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Perplexity-User, are the ones you need to keep open.
This distinction gets missed constantly because most robots.txt advice still treats "AI crawlers" as one undifferentiated group to allow or block. ChatGPT alone reaches roughly 900 million weekly active users, according to OpenAI's own February 2026 announcement (source: TechCrunch, 2026), and a blanket rule that blocks every AI-sounding user agent at once can silently cut you out of that entire audience's search answers, while doing nothing extra to protect your content from the training use you could have opted out of individually.
This article breaks down the nine crawlers that actually matter across OpenAI, Anthropic, Perplexity, and Google: which ones train a model, which ones decide whether you get cited, how to write a robots.txt file that separates the two, and what happens when a site blocks the wrong one by accident. It also covers how to confirm these crawlers are actually reaching your site, since a robots.txt rule is a request, not a lock.
What's the Difference Between an AI Crawler for Training and One for Citations?
A training crawler downloads your content to help improve a future AI model. A citation-relevant crawler fetches your page in real time to answer one specific user question, and that's the one that actually decides whether your brand shows up in an answer.
OpenAI's own documentation makes this split explicit across three separate user agents. GPTBot exists to train OpenAI's foundation models on your content; disallowing it means that content shouldn't be used for future training runs. OAI-SearchBot is different: it determines whether a page is eligible to be surfaced and cited inside ChatGPT's search answers, and OpenAI's documentation states that a robots.txt change takes about 24 hours to be reflected in that eligibility. ChatGPT-User is a third, separate agent again: it fires only when a live user takes an action inside ChatGPT, and it doesn't honor robots.txt at all, since it's triggered by a human, not by automated crawling (source: developers.openai.com/api/docs/bots, verified 2026-07-06).
Anthropic and Perplexity draw the same line, just with their own naming. Anthropic documents ClaudeBot as its training crawler, alongside Claude-SearchBot for search result quality and Claude-User for live, user-triggered fetches inside Claude, with all three explicitly stated to honor robots.txt directives (source: support.claude.com/en/articles/8896518, updated April 7, 2026, verified 2026-07-06). Perplexity's split works the same way: PerplexityBot indexes pages for visibility inside Perplexity's answers and is explicitly not used to train a foundation model, while Perplexity-User fires on a live user action and generally ignores robots.txt the same way a person's own browser request would (source: docs.perplexity.ai/guides/bots, verified 2026-07-06).
The reason a citation crawler needs to be treated differently from a training crawler comes down to how these systems generate an answer in the first place. Most AI search products run on retrieval-augmented generation (RAG): instead of answering purely from what a model learned during training, the system retrieves a set of live, relevant passages at query time and generates a response grounded in those passages. A page blocked from the retrieval crawler simply isn't in the candidate pool at answer time, no matter how much of its content a training crawler absorbed months earlier.
This training-versus-retrieval split sits at the center of what's often called generative engine optimization: getting recognized as a citable source inside an AI-generated answer, not just ranking a URL in a list of links. The retrieval side of that split is closer to how a traditional search index works than to how a model gets trained, which is why understanding the mechanics behind LLM SEO matters just as much as picking the right user agent to allow. Blocking the wrong one doesn't just opt you out of training, it can opt you out of ever being cited at all.
Which Crawlers Actually Need to Be Allowed to Get Cited by ChatGPT, Claude, Perplexity, and Google?
Six crawlers matter if citations are your goal: OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User (Perplexity). Google doesn't need a separate allow rule for its AI features, since AI Overviews use the same Googlebot index you're likely already allowing.
The table below covers all nine user agents across the four companies most likely to cite your content, including the two training-only crawlers (GPTBot, ClaudeBot) and Google-Extended, which isn't a crawler at all but a control token. One detail worth flagging: Anthropic's documentation states that all three of its bots, ClaudeBot, Claude-User, and Claude-SearchBot, honor robots.txt directives (source: support.claude.com, updated April 7, 2026), while OpenAI's ChatGPT-User and Perplexity's Perplexity-User are both live, user-triggered requests that generally ignore robots.txt the same way a person's own browser would. That distinction changes what an Allow or Disallow rule can actually accomplish for each one.
| Company | Crawler | Job | Respects robots.txt? | Allow for citations? |
|---|---|---|---|---|
| OpenAI | GPTBot | Trains OpenAI's foundation models | Yes | Optional (training only) |
| OpenAI | OAI-SearchBot | Decides eligibility for ChatGPT's search answers | Yes (about 24h to take effect) | Yes |
| OpenAI | ChatGPT-User | Fetches a page after a live user action inside ChatGPT | No (triggered by a human, not automated crawling) | Yes (needs open access regardless) |
| Anthropic | ClaudeBot | Trains Anthropic's generative models | Yes | Optional (training only) |
| Anthropic | Claude-SearchBot | Improves the quality of Claude's search results | Yes | Yes |
| Anthropic | Claude-User | Fetches a page after a user-triggered action inside Claude | Yes | Yes |
| Perplexity | PerplexityBot | Indexes pages for visibility in Perplexity's answers, not used for model training | Yes | Yes, Perplexity explicitly recommends allowing it |
| Perplexity | Perplexity-User | Fetches a page live in response to a user action | Generally no (treated like a live browser request) | Yes (needs open access regardless) |
| Google-Extended | Control token for Gemini/Vertex AI training use, not a crawler | Not applicable, it's a token | Doesn't affect Search or AI Overview inclusion either way |
Google is the one company in this table that doesn't need its own citation-specific allow rule. Google's documentation confirms that Google-Extended only controls whether your content can be used to train Gemini and Vertex AI models; it has no effect on whether your pages are included or ranked in Google Search or its AI features, which draw on the standard Googlebot index most sites already allow (source: developers.google.com/search/docs/appearance/ai-features, verified 2026-07-06). Getting cited inside that index still depends on the broader structural and trust signals covered in how to get cited by AI, not on a separate robots.txt rule.
How Do You Actually Allow or Block These Crawlers in robots.txt?
Allowing or blocking any of these crawlers works the same way as any other robots.txt rule: a User-agent line naming the exact bot, followed by an Allow or Disallow directive. The catch is precision. Because the training crawler and the citation crawler at each company use different, separately documented user agent strings, a single blanket rule aimed at "AI bots" as a category will almost always hit both at once, which defeats the entire point of separating training access from citation access.
Each rule pairs one User-agent line, the exact string a crawler self-identifies with in its HTTP request, with an Allow or Disallow directive underneath it. Multiple User-agent lines can share one directive block if the intended treatment is identical, but the moment two crawlers need different treatment, as GPTBot and OAI-SearchBot do here, they need their own separate blocks. Getting this wrong is rarely intentional. It usually happens because someone copied a "block all AI bots" snippet from a forum post or a security plugin without checking which of the nine user agents it silently grouped together.
A Sample Allowlist for Sites That Want to Be Cited
This block allows every citation-relevant crawler identified above, across all three companies that publish one, while leaving your existing Googlebot rule untouched since Google isn't listed separately here.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
How to Block Training Only While Keeping Citation Access Open
If the goal is to opt out of AI training specifically while keeping every citation path above open, add separate Disallow rules for the three training-only agents. These rules don't need to touch OAI-SearchBot, Claude-SearchBot, or PerplexityBot at all, since each is documented as a distinct user agent at its respective company.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
A robots.txt file only controls crawl access, it says nothing about whether the page itself is structured in a way an AI system can actually parse and quote once it arrives. Pairing a clean allowlist with proper markup, covered in schema markup for AI, addresses the other half of the problem: getting a crawler onto the page is only useful if what it finds there is easy to extract cleanly. For blog content specifically, that means pairing an open allowlist with correctly implemented Article and BlogPosting schema, since a citation crawler that can reach a post still needs clean markup to identify the headline, author, and publish date it's quoting.
This article, Blue could have written it for you: content optimized for Google + AI, without you writing a single word.
Try mentionLABDoes robots.txt Actually Stop These Crawlers From Accessing Your Site?
robots.txt is a voluntary courtesy protocol, not a lock. OpenAI, Anthropic, and Perplexity all publicly commit to honoring it, which is why the allowlist above works for them, but a large-scale academic study found that scraper compliance with robots.txt directives varies significantly across the wider web, with AI search crawlers among the categories least likely to check it at all (source: arxiv.org/abs/2505.21733, verified 2026-07-06), so pair it with server-log monitoring for anything you actually depend on.
The reason robots.txt works at all for well-behaved crawlers is that it's an actual internet standard, not an informal convention. The Robots Exclusion Protocol was published as an official IETF standard, RFC 9309, on the Standards Track in September 2022, co-authored by engineers at Google (source: rfc-editor.org/info/rfc9309/, verified 2026-07-06). That standardization is precisely why OpenAI, Anthropic, and Perplexity can document a predictable set of user agents and a compliance commitment for each one. A crawler that ignores the standard entirely isn't breaking a law, it's simply choosing not to follow a protocol nothing forces it to obey.
robots.txt also doesn't operate alone on most production sites. A Web Application Firewall (WAF) or CDN can block a crawler by IP range or by user agent string independently of whatever your robots.txt file says, sometimes without anyone realizing it. A site can have a perfectly correct allowlist in robots.txt and still return a 403 to every one of these crawlers if a security rule further up the stack is blocking bot traffic generically, which is exactly why the verification step later in this article checks actual server responses, not just the robots.txt file itself.
Should You Block GPTBot to Keep Your Content Out of AI Training?
Yes, if you want to opt out of AI training specifically, disallow GPTBot (OpenAI), ClaudeBot (Anthropic), and any equivalent training-only crawler. That decision is independent from the citation question above. You can block training entirely while still allowing the search/citation bots to fetch and cite your pages.
There are legitimate reasons to make this choice. Some site owners simply don't want their work used as AI training data without compensation or attribution, some are contractually restricted from allowing it, and some are managing crawl budget on a large site and would rather their server resources go to Googlebot and the citation-relevant bots than to a training crawler that doesn't send any traffic back. None of these reasons require blocking the citation-relevant crawlers in the same breath, since training and citation eligibility are controlled by entirely separate user agents at every company covered here.
Opting out of AI training has become common enough that infrastructure providers now build dedicated tooling around it. Cloudflare launched a Content Signals Policy on September 24, 2025, adding three separate signals, search, ai-input, and ai-train, that a robots.txt file can set independently, and more than 3.8 million domains already had Cloudflare-managed robots.txt active by that date (source: blog.cloudflare.com/content-signals-policy/, verified 2026-07-06). Starting September 15, 2026, new domains joining Cloudflare will default to blocking the Training and Agent categories on pages that carry advertising, while the Search category stays allowed by default (source: blog.cloudflare.com/content-independence-day-ai-options/, verified 2026-07-06), a strong signal of where the industry consensus is heading on this exact split.
What Happens If You Block the Wrong Crawler?
The most common mistake is a blanket robots.txt rule, or a WAF setting labeled "block AI bots", that disallows every AI-sounding user agent at once. That silently blocks OAI-SearchBot, Claude-SearchBot, and PerplexityBot right alongside the training crawlers, which kills your citation eligibility while doing nothing extra to protect your content from training use you could have opted out of individually.
Here's what that mistake actually looks like in practice. A site trying to "block AI" often ends up grouping every AI-sounding user agent under one Disallow rule, catching the citation crawler in the same net as the training crawler it actually meant to block:
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /
The fix is to split that single rule back into the three separate agents it lumped together, disallowing only the one meant for training:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
The same pattern repeats at Anthropic and Perplexity: a rule meant to stop ClaudeBot from training on your content can just as easily catch Claude-SearchBot or Claude-User if it's written as one broad match instead of three distinct ones. Since ChatGPT-User and Perplexity-User don't honor robots.txt anyway, disallowing them by name accomplishes nothing except confirming, in writing, an intention the crawler itself will ignore.
How Do You Verify These Crawlers Are Actually Reaching Your Site?
Check your server logs for the exact user-agent strings, confirm the requests return 200 (not 403, 429, or a redirect loop), and cross-reference the requesting IPs against each company's published IP ranges, since a user-agent string alone can be spoofed.
This verification step matters more than it might seem, because these crawlers can generate meaningful load on their own. An independent analysis covered by Search Engine Journal, based on more than 24 million proxy requests across 69 websites over 55 days between January and March 2026, found that OpenAI's live user-query crawler made 3.6 times more requests than Googlebot on that sample (source: Search Engine Journal, 2026). A site that's never checked its logs for these user agents has no way of knowing whether that crawl volume is even reaching it, or whether a WAF or CDN rule is silently returning a 403 before the crawler ever sees the page.
A user-agent string is only step one, since it can be spoofed by anyone; matching the requesting IP against each company's published range is what actually confirms a request came from the real crawler and not an impersonator. Once you've confirmed the right bots are reaching the right pages with a clean 200 response, the next question is whether any of that access is translating into an actual citation, which is a separate signal covered in AI visibility score.
The practical takeaway holds even with nine user agents in play: separate every AI crawler into training or citation before writing a single Allow or Disallow line, never group them under one "AI bots" rule, and confirm in your server logs that the citation-relevant ones are actually reaching your pages with a 200 response. Do that once, revisit it whenever a new AI search product launches a new user agent, and the robots.txt question stops being a source of accidental de-indexing from the one channel, AI-generated answers, that's growing the fastest right now.
FAQ
Does allowing GPTBot help my Google rankings?
No. GPTBot is OpenAI's training crawler and has no connection to Google's index (source: developers.openai.com/api/docs/bots, verified 2026-07-06). Google's own documentation confirms that Google-Extended, the equivalent training-only token for Gemini and Vertex AI, doesn't affect whether your pages are included or ranked in Google Search or its AI features (source: developers.google.com/search/docs/appearance/ai-features, verified 2026-07-06). Allowing or blocking any AI training crawler is a decision about model training, not about Google Search visibility.
Can I let AI cite my content without letting it train on that content?
Yes. Disallow the training-only crawlers, GPTBot at OpenAI, ClaudeBot at Anthropic, and Google-Extended for Google's AI models, while allowing the citation-relevant crawlers: OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, and PerplexityBot and Perplexity-User. Each company documents training and citation eligibility as separate, independently controllable user agents (sources: developers.openai.com, support.claude.com, docs.perplexity.ai, all verified 2026-07-06), so opting out of one doesn't require opting out of the other.
Do I need to allow every single AI crawler to show up in ChatGPT or Claude?
No. Only the citation-relevant crawler at each company controls whether you can be cited: OAI-SearchBot for ChatGPT's search answers and Claude-SearchBot for Claude's. Allowing those two, along with the live user-triggered agents ChatGPT-User and Claude-User, covers citation eligibility at both companies without also needing to allow GPTBot or ClaudeBot, which affect training only.
How long does a robots.txt change take to affect these crawlers?
For OpenAI specifically, about 24 hours: OpenAI's documentation states that a robots.txt update takes roughly a day to be reflected in OAI-SearchBot's citation eligibility (source: developers.openai.com/api/docs/bots, verified 2026-07-06). Anthropic and Perplexity's own documentation doesn't publish an equivalent number, so treat 24 hours as a reasonable minimum wait across the board before re-checking whether a change has taken effect.
Does robots.txt actually stop a crawler from accessing my site?
Not by itself. robots.txt is a voluntary protocol under the Robots Exclusion Protocol standard, RFC 9309 (source: rfc-editor.org/info/rfc9309/, verified 2026-07-06), and OpenAI, Anthropic, and Perplexity all publicly commit to honoring it for the crawlers covered here. A crawler that doesn't respect the standard can ignore your rules entirely, which is why server-log monitoring, and a Web Application Firewall for anything sensitive, matters alongside the robots.txt file itself.
Do I still need an llms.txt file if I already allow these crawlers in robots.txt?
They solve different problems. robots.txt controls whether a crawler can access your site at all; an llms.txt file, where a given AI system reads it, offers a curated summary of your most important pages for a system that's already crawling you. Allowing the right crawlers in robots.txt doesn't replace that summary layer. For a full breakdown of what llms.txt actually does, see what is llms.txt.
Blue handles your SEO and your GEO. On autopilot.
You approve, she produces content optimized for Google + AI.
Join the Lab · 5-day trial