Robots.txt Playbook for AI Crawlers

What this is

robots.txt is a plain-text file at your site root that tells crawlers which paths they may request. It does not block determined scrapers, hide secrets, or replace authentication — but it does signal intent to well-behaved search engines and AI training/indexing bots.

An interactive generator UI is shipping on this page later. Today, this playbook is the tool: copy a template below, paste it into your editor, deploy, and verify.

When to use it

• You are launching a site and need a sane default crawl policy.
• You want AI search visibility (ChatGPT, Claude, Perplexity) but need to block admin, staging, or API routes.
• Legal or compliance requires opting out of AI training crawlers while keeping Google Search access.
• You inherited a site with no robots.txt or a file that blocks / by mistake.

Step-by-step deployment

1. Pick a template from the section below that matches your policy (open, restricted, or training-opt-out).
2. Replace placeholders: https://yourdomain.com, /admin/, /staging/, and any custom paths.
3. Create the file at the web root:
   • Static hosting: public/robots.txt or site root folder.
   • Next.js App Router: use src/app/robots.ts (recommended) or public/robots.txt.
   • WordPress: root via FTP or a SEO plugin.
4. Deploy and confirm the live URL returns 200 with text/plain: https://yourdomain.com/robots.txt
5. Validate in Google Search Console → Settings → robots.txt, or fetch with curl -I https://yourdomain.com/robots.txt.
6. Re-check after migrations — a common post-launch bug is a stale disallow on /.

Production templates

Open crawl (recommended for most marketing sites)

Allows public content, blocks sensitive paths, declares sitemap. Explicitly permits major AI crawlers.

# robots.txt — open crawl with protected areas
# https://yourdomain.com/robots.txt

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /private/
Disallow: /_next/
Disallow: /studio/

# AI & search crawlers — explicit allow for GEO visibility
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: FacebookBot
Allow: /

User-agent: Bytespider
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: CCBot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml
Host: https://yourdomain.com

Staging / pre-launch (block everything)

Use on staging subdomains only. Never leave this on production.

User-agent: *
Disallow: /

Training opt-out (search yes, AI training no)

Blocks common AI training crawlers while keeping standard Googlebot access for search results. Review your legal team's guidance before deploying.

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: FacebookBot
Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

SaaS app (marketing open, app gated)

Marketing pages crawlable; authenticated app areas blocked.

User-agent: *
Allow: /
Allow: /blog/
Allow: /pricing
Disallow: /app/
Disallow: /dashboard/
Disallow: /api/
Disallow: /login
Disallow: /signup

Sitemap: https://yourdomain.com/sitemap.xml

AI crawler reference

New crawlers appear regularly. Subscribe to your CDN or SEO tool's crawler alerts and revisit this file quarterly.

Pre-flight checklist

• File lives at /robots.txt (not /public/robots.txt in the URL).
• Disallow paths match real routes — typos silently fail open.
• Sitemap URL is absolute (https://) and returns 200.
• Staging subdomain has Disallow: / if it mirrors production content.
• No secrets rely on robots.txt alone — use auth and noindex meta where needed.
• AI crawler rules align with your privacy policy and terms.
• Checked live file after deploy, not just the local copy.

Common mistakes

Blocking / on production. A single Disallow: / removes you from search and AI discovery. Always verify the live file after deploy.

Using robots.txt as security. Paths remain reachable if someone knows the URL. Block sensitive areas with login walls and X-Robots-Tag: noindex headers.

Conflicting rules. Crawlers apply the most specific matching User-agent block. Duplicate or contradictory lines create unpredictable behavior — keep one policy section per bot group.

Forgetting sitemap. Crawlers can still find pages via links, but an explicit Sitemap: line speeds discovery for new sites.

Ignoring CDN caches. After updating robots.txt, purge CDN cache or wait for TTL expiry before assuming the new rules are live.

Next steps

Robots.txt is one layer of AI visibility. Stack it with the rest of your GEO kit:

• llms.txt playbook — give AI systems a curated map of your best pages.
• Schema markup playbook — structured data for rich results and clearer entity signals.
• Meta description playbook — click-worthy snippets for search SERPs.
• Free operator guides — automation and AI stack playbooks from the MyGearHut library.
• Run a GEO readiness scan — see how AI search engines currently interpret your site.

More utilities live on the tools hub.