Get AI Cited
ChatGPT
All free tools

Free tool · no sign-up · runs in your browser, nothing is stored

AI Crawler Robots.txt Builder

Choose which AI crawlers may read your site, and copy the rules. Crawlers that fetch pages for answers decide whether engines can cite you; training crawlers decide what models learn about you.

Fetch pages for answers

Allow these to be citable in ChatGPT, Perplexity, Claude, Google and Copilot answers.

Collect pages for training

Optional: what models learn about you without searching.

Result
# AI crawlers: generated with getaicited.co/en/tools/ai-robots-txt-builder

# OpenAI: ChatGPT search results and citations
User-agent: OAI-SearchBot
Disallow: /admin
Disallow: /checkout
Allow: /

# OpenAI: Pages ChatGPT opens while answering
User-agent: ChatGPT-User
Disallow: /admin
Disallow: /checkout
Allow: /

# Perplexity: Perplexity's index and citations
User-agent: PerplexityBot
Disallow: /admin
Disallow: /checkout
Allow: /

# Anthropic: Claude's web search results
User-agent: Claude-SearchBot
Disallow: /admin
Disallow: /checkout
Allow: /

# Anthropic: Pages Claude opens while answering
User-agent: Claude-User
Disallow: /admin
Disallow: /checkout
Allow: /

# Google: Google Search, AI Overviews and Gemini grounding
User-agent: Googlebot
Disallow: /admin
Disallow: /checkout
Allow: /

# Microsoft: Bing and Copilot (also feeds other engines)
User-agent: Bingbot
Disallow: /admin
Disallow: /checkout
Allow: /

# OpenAI: What ChatGPT knows without searching
User-agent: GPTBot
Disallow: /admin
Disallow: /checkout
Allow: /

# Anthropic: What Claude knows without searching
User-agent: ClaudeBot
Disallow: /admin
Disallow: /checkout
Allow: /

# Google: Gemini training (not Search)
User-agent: Google-Extended
Disallow: /admin
Disallow: /checkout
Allow: /

# Apple: Apple Intelligence training
User-agent: Applebot-Extended
Disallow: /admin
Disallow: /checkout
Allow: /

# Meta: Meta AI training
User-agent: Meta-ExternalAgent
Disallow: /admin
Disallow: /checkout
Allow: /

# Common Crawl: Open dataset many models learn from
User-agent: CCBot
Disallow: /admin
Disallow: /checkout
Allow: /

# Everyone else
User-agent: *
Disallow: /admin
Disallow: /checkout
Allow: /

Letting AI crawlers in is step one. See whether the engines then name and cite you.

Get a free AI visibility audit

Questions

Which AI crawlers should I allow?

To appear in AI answers, allow the crawlers that fetch pages for answers: OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot, Claude-SearchBot and Claude-User (Anthropic), Googlebot and Bingbot. Blocking those removes you from the sources those engines can cite.

What about training crawlers like GPTBot and ClaudeBot?

They collect pages that models learn from, which shapes what an engine says about you when it answers without searching. Blocking them is a legitimate business choice; the trade-off is that models learn less about you from your own site.

Does robots.txt actually stop them?

The crawlers listed here say they follow robots.txt. It is a request, not a lock: content behind a login or firewall is the only way to keep it from every bot.

Where does the file go?

At the root of your domain, so it opens at https://yourdomain.com/robots.txt. If you already have one, merge these groups into it rather than replacing your existing rules.

More free tools