Robots.txt
A text file used by websites to communicate with web crawlers and other web robots.
Detailed Explanation
Robots.txt tells crawlers which pages they *can* and *cannot* request from your site. It is primarily used to manage 'crawl budget' and prevent bots from indexing internal admin pages or duplicate content. It is a 'voluntary' standard; while reputable bots (Google, Bing) follow it, malicious bots will ignore it entirely.
Quick Summary
robots.txt is a plain-text file at your site root that tells crawlers what they can and can't fetch. It's the original web standard from 1994, voluntary, polite, and ignored by malicious bots. Use it for crawl budget management, not security (since it actually advertises sensitive paths).
Key Takeaways
- Located at exactly `/robots.txt`, not enforced anywhere else.
- Syntax: `User-agent: *` followed by `Disallow:` or `Allow:` rules.
- Use `Sitemap:` directive to point crawlers to your sitemap.xml.
- Disallow ≠ noindex, Google may still index disallowed URLs if linked externally.
- Don't list secret paths in robots.txt, you're broadcasting them to everyone.
When to use it
- Blocking crawlers from `/admin`, `/api`, or `/internal` paths.
- Disallowing AI training crawlers (GPTBot, CCBot, etc.) for AI opt-out.
- Managing crawl budget on huge sites by blocking faceted-search URLs.
- Pointing crawlers to sitemaps without manual registration.
Common Mistakes
- Using robots.txt to hide sensitive pages, it's public and bots can ignore it.
- Disallowing `/` accidentally (blocks the entire site from indexing).
- Confusing `Disallow` with `noindex`, they do different things.
- Case-sensitivity errors, paths are case-sensitive on most servers.
- Blocking CSS/JS, Google needs these to render and rank pages.
Robots.txt, Frequently Asked
How do I block GPT/AI crawlers?
Add `User-agent: GPTBot` then `Disallow: /`. Also block CCBot, ClaudeBot, Google-Extended depending on your policy.
Will robots.txt remove a page from Google?
No, use a `noindex` meta tag instead. Robots.txt only prevents crawling, not indexing of already-known URLs.
What if I have no robots.txt?
Crawlers default to crawling everything. That's fine for most small sites.