What robots.txt does — and the one thing it doesn't
A robots.txt file tells search engine crawlers which paths on your site they may request. It sits at the root of your domain (example.com/robots.txt), it's the first file Googlebot looks for before crawling, and it controls crawling — not indexing. That distinction is the source of the most expensive mistake in the whole file: using a Disallow rule to hide a page from search results. Disallow stops Google from fetching the page, but it doesn't stop Google from listing the URL. If enough links point to a blocked page, Google can still show it in results with no description, which its own documentation on robots.txt confirms.
So the mental model is simple: robots.txt is a traffic instruction for well-behaved bots, not a lock on your content. If you want a page kept out of the index, robots.txt is the wrong tool. This guide covers what it is the right tool for, the exact syntax, the precedence rules Google actually applies, and the traps that quietly break sites. It's part of our technical SEO pillar, alongside deeper pieces on crawling and indexing.
robots.txt syntax rules
The file is plain text, grouped into records. Each record starts with one or more User-agent lines naming the crawler, followed by Allow and Disallow rules. A minimal, real-world file looks like this:
User-agent: * Disallow: /admin/ Disallow: /cart Allow: /cart/shipping-info Sitemap: https://example.com/sitemap.xml
What each line means:
- User-agent — which crawler the rules apply to. The asterisk is a wildcard matching every bot that doesn't have its own named record.
- Disallow — a path prefix bots should not request. Disallow: / blocks the entire site; an empty Disallow: allows everything.
- Allow — carves an exception out of a broader Disallow. Google supports it, though it wasn't in the original 1994 exclusion standard.
- Sitemap — an absolute URL to your XML sitemap. It's independent of user-agent records and helps discovery. See our guide on XML sitemaps for how the two work together.
Two wildcards are supported inside paths: the asterisk matches any sequence of characters, and the dollar sign anchors the end of a URL. So Disallow: /*.pdf$ blocks every URL ending in .pdf. Paths are case-sensitive, and rules apply only to the exact protocol, host, and port the file is served from. Your HTTP and HTTPS robots.txt files are treated as separate files.
The precedence rule that trips people up
When two rules could apply to the same URL, Google does not read top to bottom. Per Google's specification on how it interprets robots.txt, it picks the most specific rule, measured by the number of characters in the rule path. If specificity ties, the least restrictive rule wins.
Consider this pair:
User-agent: * Disallow: /downloads/ Allow: /downloads/free/
A request for /downloads/free/guide.pdf matches both rules. The Allow path is longer (more specific), so Google allows the crawl. This is why order in the file is irrelevant for matching — length is what counts. Getting this backwards is a common reason a page you meant to open stays blocked, or a page you meant to block stays open. When something looks wrong, count the characters.
robots.txt vs noindex: the difference that matters most
This is the single most important concept, and it's where the "URL indexed with no description" problem comes from.
| robots.txt Disallow | noindex meta tag / header | |
|---|---|---|
| Controls | Crawling (fetching the page) | Indexing (listing the page) |
| Where it lives | /robots.txt at the root | In the page's HTML head or HTTP response header |
| Keeps a URL out of results? | No | Yes |
| Requires the page to be crawlable? | N/A | Yes — Google must fetch the page to see it |
To reliably keep a page out of Google, use a noindex directive, either as a meta robots tag or an X-Robots-Tag HTTP header, per Google's block-indexing documentation. And here's the trap that permanently damages sites:
Never Disallow a page you're trying to noindex. If a URL is blocked in robots.txt, Google can't fetch it, which means it can never read the noindex tag. The page stays crawl-blocked and, because inbound links exist, can stay indexed indefinitely with no description. The correct sequence to remove a page: allow crawling, add noindex, wait for Google to recrawl and drop it, and only then consider disallowing it (usually you never need to). This ordering conflict is one of the checks worth building into a technical SEO audit.
What robots.txt is genuinely good for
Used correctly, the file solves real problems:
- Crawl budget on large sites. Block faceted-search URLs, internal search results, and infinite calendar pages so Googlebot spends its budget on pages that matter. On a 100,000-URL store, keeping bots out of ?sort= and ?filter= parameter permutations is the difference between your new products getting crawled this week or next month. Google notes crawl budget only becomes a real concern on sites with large numbers of URLs, in its guide to managing crawl budget.
- Server load. Blocking heavy, low-value endpoints reduces request volume.
- Staging and internal tools. Disallow paths that shouldn't be crawled at all — though for truly private content, use authentication, not robots.txt. The file is public and anyone can read it.
What it should not do: block CSS and JavaScript that Google needs to render your pages. If you disallow your asset directories, Google renders a broken version of your site, which hurts both indexing and how it evaluates layout. This overlaps directly with JavaScript SEO, where render-blocking is a frequent culprit behind missing content.
Crawl-delay, file size, and other limits Google enforces
A few hard technical facts, straight from Google's robots.txt specification:
- Google ignores crawl-delay. The crawl-delay directive is not supported by Googlebot. Bing and some others honor it, but for Google you control crawl rate through Search Console (or by fixing the server responses that make Google back off). Leaving crawl-delay in your file does no harm; it just does nothing for Google.
- 500 KiB parse limit. Google reads only the first 500 kibibytes of the file and ignores anything past that, a limit stated in its robots.txt technical reference. Keep long lists of rules near the top and don't let auto-generated files balloon.
- 24-hour caching. Google generally caches robots.txt for up to 24 hours. A rule you add won't take effect instantly, and if the file returns a 5xx error, Google may treat the whole site as disallowed temporarily — so a broken robots.txt can accidentally deindex you.
The protocol itself was formalized in 2022 as RFC 9309, published by the IETF, which is worth reading if you want the exact grammar rather than a vendor's interpretation.
Managing AI and LLM crawlers
The newer question is what to do about AI crawlers — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and others. These respect robots.txt the same way search crawlers do, using named user-agent records:
User-agent: GPTBot Disallow: /
User-agent: Google-Extended Disallow: /
Google-Extended controls whether your content helps train Gemini and grounds AI features, separately from normal Googlebot crawling, as Google explains in its Google-Extended documentation. Blocking it does not remove you from Google Search. This is where robots.txt has quietly shifted from a pure SEO tool to a content-policy tool: you're deciding not just what gets crawled, but what gets used to train models and answer AI queries. If AI-driven traffic matters to you, blocking every AI agent may cost you visibility in AI answers — a trade worth tracking rather than defaulting one way. Our AI visibility monitoring exists partly to measure that trade-off.
Test before you deploy
Because a single misplaced slash can block your whole site, never ship robots.txt changes blind. Fetch the live file at yourdomain.com/robots.txt to confirm it's actually served (not returning a soft-404 HTML page), and use the robots.txt report in Google Search Console to see how Google parses it. When you're auditing a site, a full site audit should flag important pages that are blocked from crawling and surface the robots.txt-versus-noindex conflicts described above before they cost you rankings. If you're weighing tools for this, our Screaming Frog and Ahrefs alternatives both run these checks from ₹499/month, billed in INR with UPI and GST.
Robots.txt is a small file with an outsized ability to help or hurt. Get the precedence rules right, keep Disallow and noindex in their separate lanes, and test every change against the live parser.