Robots.txt: The Complete Guide to Crawler Access Control

Robots.txt is a plain text file at the root of a site that tells crawlers which URLs they may request. It controls crawling, not indexing, and it works only because well-behaved crawlers choose to follow it; it is not security. Used carefully, it keeps crawlers away from low-value paths such as internal search results and filter combinations. Used carelessly, a single line can block an entire site.

Where the File Lives and What It Covers

Google’s robots.txt specification sets the rules:

The file must sit in the top-level directory, for example https://example.com/robots.txt.
“The rules listed in the robots.txt file apply only to the host, protocol, and port number where the robots.txt file is hosted.” A file on example.com does not cover blog.example.com, http://example.com, or another port.
Paths in rules are case-sensitive: /Admin/ and /admin/ are different.

Changes take time to apply: “Google generally caches the contents of robots.txt file for up to 24 hours, but may cache it longer in situations where refreshing the cached version isn’t possible.”

When the File Is Missing or Unreachable

Response for robots.txt	What Google does
4xx, except 429	Treats the site as having no robots.txt; crawling is unrestricted
5xx, first 12 hours	Stops crawling the site and keeps trying to fetch the file
5xx, up to 30 days	Uses the last good version while still trying; with no cached version, assumes there are no crawl restrictions
5xx, after 30 days	Behaves as if there is no robots.txt if the site is otherwise available; stops crawling if the site has general availability problems
Redirects	Follows at least five hops, then treats the file as a 404

A 503 response, the specification notes, “results in fairly frequent retrying.” A server that returns errors for robots.txt can pause the crawling of a whole site.

Syntax

# Comments start with a hash
User-agent: *
Disallow: /admin/
Allow: /admin/public/

User-agent: Googlebot
Disallow: /admin/
Allow: /admin/public/
Disallow: /internal-search/

Sitemap: https://example.com/sitemap.xml

The specification states: “Only one group is valid for a particular crawler.” Googlebot follows its own group here and ignores the * group entirely, which is why the admin rules are repeated in it. Without them, Googlebot could crawl /admin/. The specification also covers the opposite case: when a file has several groups for the same specific user agent, their rules are merged into one group, while a specific group and the * group are never merged.

User-agent names the crawler the following rules apply to; * matches any crawler without a more specific group. Google’s crawlers follow “the group with the most specific user agent that matches the crawler’s user agent,” and only one group applies to a given crawler.
Disallow blocks paths that start with the given value.
Allow makes an exception inside a blocked path.
Sitemap gives the full URL of a sitemap, anywhere in the file, and the sitemap does not have to be on the same host.

Wildcards and Precedence

Symbol	Meaning	Example
<!–INLINECODE14–>	Zero or more of any character	<!–INLINECODE15–> blocks paths containing <!–INLINECODE16–>
<!–INLINECODE17–>	End of the URL	<!–INLINECODE18–> blocks paths ending in <!–INLINECODE19–>
None	Prefix match	<!–INLINECODE20–> blocks <!–INLINECODE21–>, <!–INLINECODE22–>, and <!–INLINECODE23–>

When rules conflict, the specification states that “crawlers use the most specific rule based on the length of the rule path,” and adds: “In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule.” With Allow: /folder and Disallow: /folder, the URL is allowed. Other crawlers can resolve conflicts differently.

The prefix rule is easy to miss: Disallow: /blog without a trailing slash also blocks /blog-archive/ and any path beginning with the same letters.

Common Patterns

# Admin areas
User-agent: *
Disallow: /admin/
Disallow: /wp-admin/

# Internal search results
User-agent: *
Disallow: /search/
Disallow: /*?q=
Disallow: /*&q=

# Sorting and session parameters
User-agent: *
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?sessionid=
Disallow: /*&sessionid=

Each parameter needs two rules, because /*?sort= matches only when sort is the first parameter. With both rules, /products?sort=price and /products?color=red&sort=price are blocked, while /products?color=red stays crawlable. Test patterns like these against real URLs from your site before relying on them, and remember that blocking a URL stops crawling, not indexing.

For a staging site, put a login in front of it; Disallow: / only asks crawlers to stay out and does not keep people or links away from it. Sites that rely on that line run a second risk: the same file being copied to production during a launch or migration. Check robots.txt as part of every deployment.

What Google Does Not Support

noindex in robots.txt. Google’s 2019 note on unsupported rules announced it was “retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019.” Use a robots meta tag or an X-Robots-Tag header, on a page Google can crawl.
crawl-delay. Google does not support it. Search Console’s crawl rate limiter was retired as well: Google announced in November 2023 that the tool would be deprecated on January 8, 2024, and pointed site owners to server responses for slowing Googlebot.
Files over 500 KiB. “Google enforces a robots.txt file size limit of 500 kibibytes (KiB). Content which is after the maximum file size is ignored.”

Robots.txt and Indexing

Blocking a URL stops Google from fetching it, not from listing it. A blocked URL that other pages link to can still be indexed without its content. To keep a page out of results, leave it crawlable and use noindex. Do not block a page in robots.txt and add noindex at the same time; Google will never see the noindex.

Mechanism	Stops crawling	Stops indexing	Read before fetching
<!–INLINECODE40–> in robots.txt	Yes	No guarantee	Yes
<!–INLINECODE41–> meta tag	No	Yes	No
<!–INLINECODE42–> header	No	Yes	No
Canonical tag	No	Consolidates duplicates	No
Login or IP restriction	Yes	Keeps content out; a known URL can still exist	Yes

Robots.txt Is Not Security

The file is public. Listing /internal-admin/ in a Disallow rule tells anyone who reads it that the path exists. Scrapers and malicious bots can ignore it. Protect private content with authentication.

AI Crawlers

Each AI crawler has its own token, and blocking one does not block another from the same company. OpenAI’s crawler documentation explains that a site “can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot” for training. Anthropic separates ClaudeBot, Claude-SearchBot, and Claude-User. Google’s list of common crawlers states that “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

A file named llms.txt is not part of robots.txt, and Google’s AI optimization guide says Google Search does not use such files.

Testing and Maintenance

The robots.txt report in Search Console shows “which robots.txt files Google found for the top 20 hosts on your site,” when they were last crawled, and any errors or warnings, and it lets you request a recrawl of the file in an emergency.
Crawl your own site with a crawler set to obey robots.txt and review what it skips.
Keep robots.txt under version control, with a comment for every rule whose reason is not obvious.
Review it after migrations, new site sections, and URL changes, when old rules can block new paths.

Questions About Robots.txt

Should I block CSS and JavaScript files?
No. Google renders pages and needs those files to see them as visitors do.

Do I need a robots.txt file if I block nothing?
No; a missing file means unrestricted crawling. A short file with a Sitemap line still gives crawlers the sitemap’s location.