Is my robots.txt blocking Google?

A robots.txt containing "User-agent: *" followed by "Disallow: /" tells every search engine to stay out of the entire site. Nothing will be crawled and nothing new will be listed, no matter how good the content. It is the single most damaging line a site can contain, and it is nearly always a leftover from the site being built in private.

Why it matters

Search engines read robots.txt before anything else and obey it. With a full block, Google cannot see your pages, so it cannot rank them for anything. Pages that were already indexed may hang on for a while with no description, then fade out. New pages never appear at all. Because the rest of the site works perfectly for visitors, the problem can go unnoticed for months.

How it happens

Developers block crawlers on a staging site so that the unfinished version does not show in search. When the site goes live, the file is copied across with the block still in it. Some hosting platforms and "coming soon" plugins write the same line while the site is hidden and do not always remove it.

How to check

Our check reads your robots.txt and warns if the wildcard rule disallows everything. Open yourdomain.co.uk/robots.txt in a browser and look for a line that says "Disallow: /" with nothing after the slash. Google Search Console's robots.txt report shows what Google sees.

How to fix it

Remove the "Disallow: /" line, or change it to disallow only the directories you mean, such as /wp-admin/. On WordPress, check that "Discourage search engines" under Settings, Reading is unticked, because that setting writes the block into the generated file. Then go to Google Search Console, request a re-crawl of the home page, and submit the sitemap. Recovery takes days to weeks depending on how long the block was in place.

Not sure how your own site does on this? Run the free check: it takes about twenty seconds and reports on this and thirty other things.

Check my website

More guides