Dispatch
Robots.txt: The Complete Guide to Crawler Access Control
Executive Summary Robots.txt controls which parts of a site search engine crawlers are allowed to request. Proper configuration prevents crawlers from wasting time on low-value paths while keeping important pages...
On this page
- Executive Summary
- Robots.txt Fundamentals
- Syntax and Directive Types
- Pattern Matching and Wildcards
- Common Configuration Patterns
- Google-Specific Considerations
- Testing and Validation
- Interaction with Other Directives
- Security and Robots.txt Limitations
- Maintenance and Change Management
- Quick Reference: Robots.txt vs. Other Crawl/Index Controls
- Frequently Asked Questions
- Related posts:
Executive Summary
Robots.txt controls which parts of a site search engine crawlers are allowed to request. Proper configuration prevents crawlers from wasting time on low-value paths while keeping important pages fully accessible. Misconfiguration is one of the easier ways to accidentally remove a site from search results, so the file deserves more care than its small size suggests.
Critical rules to keep in mind:
- Never use robots.txt to block pages you want indexed. Blocking crawling does not remove a page from the index if other sites already link to it; it just removes Google’s ability to see the page content.
- Test changes before deploying them, using the robots.txt report in Google Search Console.
- Keep the file at the domain root with a fast, reliable response time.
- Use specific, narrow patterns rather than broad rules that catch URLs you didn’t intend to block.
- Treat robots.txt as advisory. It works because well-behaved crawlers choose to honor it, not because it technically prevents access.
Robots.txt Fundamentals
Robots.txt is a plain text file at a domain’s root (example.com/robots.txt) that tells crawlers which URLs they may or may not request. It follows the Robots Exclusion Protocol, which Google, alongside other major search engines, formally documents and supports.
Location is fixed. The file for example.com must live at example.com/robots.txt. A file at blog.example.com/robots.txt governs only that subdomain; crawlers do not look anywhere else for instructions.
Each protocol and host is evaluated separately. http://example.com/robots.txt and https://example.com/robots.txt are technically distinct resources, though a working HTTP-to-HTTPS redirect typically means crawlers only ever see the HTTPS version.
The file is advisory, not a security mechanism. Reputable crawlers, including Googlebot, request and honor robots.txt. Many scrapers and malicious bots ignore it entirely. Treat it as a crawl-management tool, never as access control for sensitive content.
Caching delays the effect of changes. Google generally caches a site’s robots.txt for up to 24 hours, and may extend that window if it cannot fetch a fresh copy due to server errors, according to <a href="https://developers.google.com/search/docs/crawling-indexing/robots/robotstxt”>Google’s robots.txt documentation. A rule you publish today may not change crawler behavior immediately.
A missing file defaults to full access. If example.com/robots.txt does not exist (and returns a 404), crawlers assume there are no restrictions. That may be fine for a simple site, or it may mean you’re missing an opportunity to keep crawlers away from low-value paths.
Syntax and Directive Types
Robots.txt syntax is intentionally simple. A handful of directives cover the vast majority of real-world configurations.
User-agent declares which crawler the following rules apply to. Googlebot targets Google’s main crawler specifically; * is a wildcard matching any crawler that doesn’t have a more specific block. Rules apply to the user-agent above them until the next User-agent line.
User-agent: Googlebot
# Rules here apply only to Google's main crawler
User-agent: *
# Rules here apply to all other crawlersDisallow blocks the matching path prefix.
Disallow: /admin/
Disallow: /private/
Disallow: /temp.htmlAllow carves out exceptions within a disallowed path. It’s primarily useful for permitting a specific file or subfolder inside a directory that’s otherwise blocked.
Disallow: /images/
Allow: /images/public/Sitemap points crawlers to a sitemap file and can be placed anywhere in the file, independent of user-agent blocks.
Sitemap: https://example.com/sitemap.xmlCrawl-delay is recognized by some crawlers (notably Bing) but Google does not honor it; Google manages crawl rate through Search Console’s crawl-rate settings and its own load-based systems instead, per <a href="https://developers.google.com/search/docs/crawling-indexing/robots/robotstxt”>Google’s documentation.
Crawl-delay: 10Comments start with # and are ignored by crawlers, useful for documenting intent for the next person who edits the file.
# Block admin section from all crawlers
User-agent: *
Disallow: /admin/Pattern Matching and Wildcards
Beyond simple prefix matching, robots.txt supports two special characters for more flexible rules.
| Symbol | Behavior | Example |
|---|---|---|
| <!–INLINECODE11–> | Matches any sequence of characters | <!–INLINECODE12–> blocks all PDFs |
| <!–INLINECODE13–> | Anchors the match to the end of the URL | <!–INLINECODE14–> blocks <!–INLINECODE15–> but not <!–INLINECODE16–> |
| (none, prefix) | Default behavior for any path | <!–INLINECODE17–> blocks everything starting with that path |
A few practical notes:
- Paths are case-sensitive:
/Admin/and/admin/are different paths to a crawler. - Query strings can be matched with wildcards, e.g.
Disallow: /*?sort=. - When
AllowandDisallowrules conflict for the same crawler, Google resolves it by path length: the longer, more specific path wins, and if both rules are the same length, the less restrictive rule (Allow) wins. Other crawlers may resolve ties differently, so test combined Allow/Disallow rules with the Search Console robots.txt report rather than assuming the same behavior everywhere.
Common Configuration Patterns
A few patterns cover most real-world needs:
# Block admin and CMS interfaces
User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
# Block internal search result pages
User-agent: *
Disallow: /search
Disallow: /*?s=
# Block low-value parameter combinations
User-agent: *
Disallow: /*?sort=
Disallow: /*?sessionid=
# Block staging or development environments entirely
User-agent: *
Disallow: /That last pattern, a blanket Disallow: /, is worth flagging on its own: it’s the single most common accidental-deindexing cause when a staging robots.txt file gets pushed to production unchanged. Many teams discover this only after organic traffic drops sharply, then trace it back to a CMS migration that carried the staging robots.txt file along with it.
E-commerce sites tend to accumulate the largest, messiest robots.txt files, because faceted navigation (color, size, price-range filters) can generate enormous numbers of low-value parameter combinations. A typical pattern blocks the parameters themselves rather than trying to enumerate every URL:
User-agent: *
Disallow: /*?color=
Disallow: /*?size=
Disallow: /*?price_range=
Disallow: /*&color=Content sites with internal search commonly block the search results path outright, since search-result pages rarely offer unique value to someone arriving from an external search engine and can multiply indefinitely:
User-agent: *
Disallow: /search/
Disallow: /*?q=
Disallow: /*?keyword=Google-Specific Considerations
A few behaviors are specific to how Google interprets and applies robots.txt:
- Blocked URLs can still appear in search results. If a disallowed URL has external links pointing to it, Google may show it with just a URL and a title pulled from anchor text, without a description, because it was never crawled to generate one. Blocking prevents crawling; it does not guarantee a page stays out of the index.
noindexin robots.txt is not supported. Google never officially documented this rule, found it was used incorrectly in the vast majority of cases where it appeared, and formally stopped honoring it on September 1, 2019, per Google’s announcement. Use anoindexmeta tag orX-Robots-TagHTTP header instead, both of which require the page to be crawlable.- File size is capped at 500 KiB. Google’s parser stops processing content beyond that limit, per the <a href="https://developers.google.com/search/docs/crawling-indexing/robots/robotstxt”>official robots.txt specification. Extremely large files risk having their later rules silently ignored.
- The old robots.txt Tester tool is retired. Google sunset the standalone Robots.txt Tester and replaced it with the robots.txt report inside Search Console, which shows the robots.txt files Google found for your top hosts, when they were last crawled, and any parsing errors or warnings, per Search Console Help.
Testing and Validation
- Search Console’s robots.txt report is the authoritative place to confirm what Google currently sees and whether it’s parsing your file without errors. It also supports requesting a recrawl of the file.
- Stage changes before deploying to production. Push robots.txt edits to a staging environment first if your workflow allows it, and verify behavior there.
- Crawl your own site with a third-party crawler (e.g., Screaming Frog) configured to respect robots.txt, and review which URLs it skips, to catch unintended blocking before Google does.
- Check server logs to see which URLs crawlers are actually requesting and which are returning blocked responses, comparing that against your intended rules.
- Version-control the file. Treat robots.txt like code: track changes, document why a rule exists, and keep the ability to roll back quickly if a deployment causes problems.
Interaction with Other Directives
Robots.txt is one of several crawl- and index-control mechanisms, and conflating them is a common source of mistakes.
- Disallow vs. noindex. Robots.txt blocks crawling; it does not, by itself, prevent indexing if links exist elsewhere. A
noindexmeta tag or header prevents indexing but requires the page to be crawled first so Google can see the tag. For a page you genuinely want out of search entirely,noindex(left crawlable) is usually the correct tool, notDisallow. - Canonical tags vs. Disallow. Canonical tags consolidate signals between near-duplicate pages while leaving both crawlable. Blocking one version with robots.txt prevents Google from even seeing the canonical relationship. For duplicate content, canonicalization is usually the better approach.
- Sitemaps vs. Disallow. Don’t list blocked URLs in your sitemap; the contradictory signal wastes crawl attention and can confuse Google’s understanding of your site’s priorities.
- Meta robots / X-Robots-Tag vs. robots.txt. Robots.txt acts before a page is fetched; meta robots and
X-Robots-Tagact after a page is fetched and parsed. They serve different purposes and frequently need to be used together rather than as substitutes for one another.
Security and Robots.txt Limitations
Robots.txt is not an access-control or security tool, and using it as one creates real exposure:
- It does not prevent access. It tells cooperative crawlers what not to request. Malicious scrapers, security scanners, and other automated traffic frequently ignore it outright.
- It’s public and readable by anyone. Listing
/internal-admin-panel/in aDisallowrule announces that path exists to anyone who views the file, which is the opposite of hiding it. - Real protection requires authentication. Sensitive content needs login requirements, IP restrictions, or similar genuine access controls. Robots.txt should never be the only thing standing between the public internet and content that actually needs to stay private.
Maintenance and Change Management
Robots.txt tends to be written once and forgotten, which is exactly how stale or accidentally broad rules accumulate.
- Review periodically, especially after major site changes (CMS migration, new site sections, URL structure changes), since old rules can silently block paths that have since become important.
- Document the reasoning behind non-obvious rules so a future editor understands why a path is blocked before they “clean it up” and break something.
- Test every change before it goes live; a single misplaced
/in aDisallow: /line has taken entire production sites out of search results. - Monitor after deployment using the Search Console robots.txt report and crawl stats to confirm the change behaved as intended.
Quick Reference: Robots.txt vs. Other Crawl/Index Controls
| Mechanism | Blocks crawling? | Blocks indexing? | Visible to crawler before fetch? |
|---|---|---|---|
| <!–INLINECODE36–> (robots.txt) | Yes | Indirectly (no guarantee) | Yes |
| <!–INLINECODE37–> meta tag | No | Yes | No, requires a fetch first |
| <!–INLINECODE38–> header | No | Yes | No, requires a fetch first |
| Canonical tag | No | Consolidates signals, doesn't block | No, requires a fetch first |
| Authentication / IP restriction | Yes (effectively) | Yes (effectively) | Yes |
This table is a useful gut-check during an audit: if the goal is “keep this out of the index entirely,” robots.txt alone is usually the wrong tool, because it can’t see or act on content it never fetched. If the goal is “stop crawlers from wasting time here,” robots.txt is exactly the right tool.
Frequently Asked Questions
Does robots.txt affect rankings directly?
Not directly. It affects what gets crawled, which affects what can be indexed and ranked. A page that’s blocked from crawling can’t be properly evaluated for ranking, but robots.txt itself isn’t a ranking signal.
Should I block CSS and JavaScript from crawlers?
No. Google needs to render pages, which requires fetching CSS and JavaScript, to evaluate them accurately. Older advice to block these files predates Google’s rendering capabilities and is now counterproductive.
How quickly do robots.txt changes take effect?
Google generally caches robots.txt for up to 24 hours, so allow at least a day for changes to propagate, longer if Google previously encountered fetch errors on the file. The Search Console robots.txt report lets you request a recrawl for urgent cases.
Can I use robots.txt to remove a page that’s already indexed?
No. Blocking a crawled, indexed page with robots.txt prevents future crawling but does not remove the existing listing, and can actually make removal slower because Google can no longer see a noindex tag if you add one later. For removal, use the Search Console removal tool for a temporary block, or add a noindex tag (keeping the page crawlable) for permanent removal.
What happens if robots.txt is temporarily unreachable?
Google’s response escalates over time, per its <a href="https://developers.google.com/search/docs/crawling-indexing/robots/robotstxt”>robots.txt documentation. For roughly the first 12 hours of server errors on robots.txt, Google pauses crawling but keeps retrying. If the errors continue, Google falls back to the last known-good cached copy of the file for up to 30 days while still retrying (a 503 status triggers more frequent retries). After 30 days with no resolution, Google treats the site as if it has no robots.txt at all, meaning full access is assumed, provided the rest of the site is still reachable; if the whole site appears down, Google stops crawling altogether. A 404 on robots.txt is handled differently from the start: it’s treated as “no restrictions” immediately, not as an error condition.
Should I block AI training crawlers?
You can add Disallow rules for specific AI training crawlers, such as GPTBot (OpenAI), Google-Extended (controls Gemini and Vertex AI training and grounding; it’s a separate token from Googlebot and has no effect on Google Search itself, per Google’s AI features documentation), CCBot (Common Crawl, whose archives feed many labs’ training pipelines, per Common Crawl’s own documentation), and ClaudeBot (Anthropic), if you don’t want content used for model training. Training crawlers and search/retrieval crawlers from the same company are usually separate user-agents controlled independently: disallowing GPTBot doesn’t block OAI-SearchBot or ChatGPT-User, and disallowing ClaudeBot doesn’t block Claude-SearchBot, per OpenAI’s and Anthropic’s own crawler documentation. Whether to block training crawlers depends on your content strategy; reputable AI crawlers generally do honor robots.txt rules. Separately, llms.txt (a different file some sites publish alongside robots.txt) isn’t part of the Robots Exclusion Protocol and has no formal standards backing; Google has stated plainly that Search ignores it, so it neither helps nor hurts rankings or AI Overviews, per Google’s AI optimization guide.
How do I handle robots.txt across multiple subdomains?
Each subdomain needs its own file at its own root. example.com/robots.txt and blog.example.com/robots.txt are independent; you can make them identical or tailor each to that subdomain’s actual content needs.
Can robots.txt help with crawl budget?
Yes. Blocking low-value paths (internal search results, faceted-navigation parameter combinations, admin interfaces) prevents crawlers from spending time there, which is one piece of a broader crawl-budget strategy, particularly relevant for very large sites.
Do I need a robots.txt file if I have nothing to block?
Not strictly. A missing file simply means full access is assumed. Many smaller sites run without one. That said, including an empty or minimal file with at least a Sitemap directive is low-effort and avoids the 404 logged every time a crawler checks for a file that doesn’t exist, which is a minor but real source of crawl-log noise on some hosting setups.
Can different crawlers be given different rules in the same file?
Yes, that’s one of the format’s core purposes. Separate User-agent blocks let you, for example, allow Googlebot full access while restricting a more aggressive third-party crawler:
User-agent: Googlebot
Allow: /
User-agent: AhrefsBot
Disallow: /Crawlers match against the most specific applicable User-agent block; a crawler with no matching specific block falls through to the * rules if present.
What’s the most common robots.txt mistake you’ll find in an audit?
Beyond the blanket Disallow: / left over from staging, the next most frequent issue is a Disallow rule that’s broader than intended, for example Disallow: /blog (no trailing slash) unintentionally blocking /blog-archive/ and any other path that happens to start with the same string. Precise path matching, confirmed with the Search Console robots.txt report rather than assumed, avoids this category of error entirely.