Most website owners assume that a URL blocked by robots txt never appears in Google's search results. That assumption is wrong, and Google has now addressed it directly. According to Google's official documentation and statements from the Search Relations team, blocking a URL in robots.txt prevents Googlebot from crawling it - but it does not prevent that URL from being indexed and showing up in search results. Understanding why this happens, and what to do about it, is one of the more important distinctions in technical SEO.
What Robots.txt Actually Does - and What It Does Not Do
The robots.txt file is a crawl instruction, not an access control system. When Googlebot visits a site, it reads the robots.txt file first. If a URL is disallowed, Googlebot respects that rule and does not open the page. It never reads the content. It never processes the HTML. But this is where many SEO practitioners make a critical mistake: they assume that because Googlebot cannot read the page, the page cannot appear in Google's index.
Google can learn that a URL exists without ever crawling it. The most common way this happens is through backlinks. If another website links to a URL that is blocked by robots.txt, Google sees that link, notes the URL, and can include it in the index based on the external signal alone. The URL may appear in search results with no title, no description, and no snippet - just the bare address - because Google has no content to draw from. This is the specific scenario Google has explained repeatedly: a URL blocked by robots txt can appear in search results when it receives links from other crawlable pages.
The Key Difference: Crawling vs Indexing
Crawling and indexing are two separate processes that operate independently. Crawling is the act of visiting a page and reading its content. Indexing is the act of storing information about a URL in Google's database and making it eligible for search results. Robots.txt controls crawling. It has no direct authority over indexing.
This is not a bug or a loophole. It is how the system was designed. Google's core function is to map the web's link structure, and links are signals that exist independently of whether Googlebot can read a specific page. When Google sees many sites linking to a blocked URL, it treats that URL as a meaningful web address, even without content.
The correct tool for preventing indexing is the noindex directive. Unlike robots.txt, which operates at the crawl layer, noindex operates at the indexing layer. A noindex tag inside the page's HTML head, or delivered as an HTTP response header, tells Google explicitly not to add the page to its index. However, there is an important catch: Google must crawl the page to read the noindex instruction. If robots.txt is blocking crawling and noindex is on the same page, Google never reads the noindex tag, and the indexing instruction is never processed.
Noindex vs Robots Txt: The Conflict That Traps Many Sites
The noindex vs robots txt conflict is one of the most misunderstood issues in technical SEO, and it is the direct cause of many indexing problems. Here is the scenario that catches site owners out: a page is considered low-value, so the developer adds a noindex meta tag to the page head to remove it from search results. But then, they also add a Disallow rule in robots.txt for that same URL to save crawl budget. Because Googlebot is blocked from crawling, it can never read the noindex tag, and the page remains in the index indefinitely.
How to Correctly De-index a Page
To successfully remove a page from search results, you must follow the correct order of operations:
- Remove the Disallow rule from robots.txt to allow Googlebot to crawl the URL.
- Keep the noindex tag in the page's HTML head or as an HTTP response header.
- Wait for Google to crawl the page, read the noindex tag, and drop it from search results.
- Block the URL in robots.txt after it has been successfully de-indexed (if you need to conserve crawl budget).
Frequently Asked Questions
Q: Why is my robots.txt blocked page showing in search results?
A: It likely has external backlinks or internal links pointing to it, allowing Google to index the URL based on those signals even without crawling the content.
Q: How do I hide a page from Google completely?
A: Use a `noindex` tag in the HTML head (or HTTP header) and make sure the page is *not* blocked in robots.txt so Googlebot can crawl and read it.
Q: Does Googlebot respect Disallow in robots.txt?
A: Yes. Googlebot respects Disallow for crawling, meaning it will not fetch the page content. It just doesn't prevent indexing if other links point to it.