Robots.txt disallow stops Googlebot from crawling a URL — it doesn't remove an already-indexed page. Noindex removes a page from the index, but only if Google can still crawl it to see that instruction. Disallow a URL before it carries noindex, and it can stay indexed indefinitely, unreachable for the removal Google never sees.
Key takeaways
- Disallow blocks crawling. Noindex removes from the index. They are not interchangeable and one cannot substitute for the other.
- The order matters more than which directive you pick. Noindex first, disallow later — never the reverse if the goal is removal.
- Blocking before noindexing is the single most common version of this mistake — usually during a migration or a bulk cleanup done in a hurry.
- A page stuck this way shows as a URL with no snippet in search results — indexed, but with nothing Google can read to describe it, which is a visible tell something's wrong.
- The fix is to remove the disallow rule, let Google back in to see the noindex tag, confirm removal, and only then reapply the block if you still want one.
What each directive actually does
| Directive | Where it lives | What it does | What it doesn't do |
|---|---|---|---|
| Disallow | robots.txt | Stops Googlebot from requesting the URL at all | Does not remove an already-indexed page |
| Noindex | Meta tag or X-Robots-Tag header on the page itself | Tells Google to drop the page from the index on its next crawl | Only works if Google is still allowed to crawl the page and see it |
The trap, exactly
Add a robots.txt disallow rule for a URL that's already indexed, and Googlebot obeys it immediately — it stops visiting that page. If you add a noindex tag to the page afterward, it doesn't matter: Google can't see it, because it isn't crawling the page anymore. The result is a URL that stays in the index, often reduced to a bare link with no title or description, because Google has nothing left to read.
The correct order
- Add noindex first. Meta tag or X-Robots-Tag header, applied while the URL is still fully crawlable.
- Leave it crawlable and wait. Google needs to actually revisit the page to see the tag — this isn't instant, and forcing it via Search Console's "Request Indexing" can speed it up but doesn't guarantee a same-day result.
- Confirm removal in Search Console. The URL Inspection tool will show whether Google has processed the noindex and dropped the page.
- Only then, if you still want it, add the robots.txt disallow. At that point blocking is safe — the removal already happened, and disallow is now just preventing future crawl attempts on a URL that's already out.
When disallow is actually the right tool
For URLs that should never be indexed and, ideally, never even fetched — admin routes, internal search result pages, infinite filter-combination URLs that would otherwise burn crawl budget — disallow from the start is correct, because there's no already-indexed page to remove. The trap is specific to URLs that are already in the index when you reach for robots.txt as the fix.