How to fix "Blocked by robots.txt" in WordPress
"Blocked by robots.txt" in Search Console means Googlebot asked for permission before crawling a URL and your robots.txt said no. On WordPress the cause is usually not a hand-written file at all: it is a checkbox in Settings > Reading that makes WordPress serve a virtual robots.txt containing Disallow: /, which closes the entire site. The check on this page fetches your live robots.txt and shows what it actually contains, so you can tell that case apart from a real file with narrow rules before you change anything.
Check your site
-
Run the check and read the actual robots.txt
Enter your domain above. The checker fetches https://your-domain/robots.txt from outside your network and shows the file as a crawler receives it, whether a Disallow: / is present, and whether a sitemap is declared. Read that result first. If it shows Disallow: / under User-agent: *, the whole site is closed and the cause is almost certainly step 2. If it shows only narrow rules with no Disallow: /, then only specific URLs are blocked, so go to step 4. If robots.txt is missing or empty, crawling is allowed and robots.txt is not what is stopping you: look for a noindex tag, HTTP password protection, or a firewall rule instead.
-
Turn off the WordPress indexing block first
In wp-admin open Settings > Reading and uncheck "Discourage search engines from indexing this site", then save. This is the number one cause: it gets switched on for the staging copy and stays on after launch. While it is on, WordPress serves a virtual robots.txt with Disallow: / and also adds a noindex tag to every page. Do this before editing any file, because a physical robots.txt overrides the virtual one and will mask what the setting is doing.
-
Check whether a real robots.txt file exists in the site root
Re-run the check. If Disallow: / is still there after you unchecked the setting, a physical file exists and takes precedence, so the WordPress toggle changes nothing. In your hosting control panel's file manager open the folder where WordPress is installed (usually public_html) and look for robots.txt, with hidden files shown. Edit or delete it. SEO plugins such as Yoast, Rank Math and All in One SEO have their own robots.txt editor that writes this same real file, so check there too. Only the file at the domain root counts; one in a subfolder is never read.
-
Make sure CSS and JS directories are not blocked
Google renders the page before deciding what it is about, so it needs your stylesheets and scripts. Copy-pasted legacy rules like Disallow: /wp-includes/ or Disallow: /wp-content/plugins/ break that rendering and should be removed. A safe minimal WordPress robots.txt is:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Anything beyond those lines should have a reason you can explain out loud. -
Declare the sitemap
The check also reports whether a Sitemap line is present. Add it on its own line at the end of the file, as a full absolute address: Sitemap: https://your-domain/wp-sitemap.xml for the sitemap WordPress generates itself (5.5 and newer), or the plugin address instead, which for Yoast and Rank Math is usually /sitemap_index.xml. The Sitemap line applies to all crawlers and is independent of the User-agent blocks above it.
-
Re-run the check, then ask Google to revalidate
When the check shows the clean file, purge your page cache and your Cloudflare cache if you use one, because a stale copy of robots.txt can keep being served for a while. Then in Search Console open Settings > robots.txt to see the version Google last fetched and request a recrawl, and in the Page indexing report open the "Blocked by robots.txt" issue and press Validate Fix. Google refetches robots.txt roughly once a day, and validation runs over days, not minutes.
How to verify the result
Open https://your-domain/robots.txt in a private window: you should see the rules you wrote and no Disallow: / line. Re-run the check on this page to confirm that is also what a crawler gets from outside your network, and that the Sitemap line is there. The final confirmation is in Search Console: run URL Inspection on your homepage and click Test live URL, which should report the URL as available to Google.
Tip: robots.txt controls crawling, not indexing. A blocked URL can still show up in search results, with no description, if other sites link to it, and that is exactly what "Indexed, though blocked by robots.txt" means. So never try to remove a page from search by adding a Disallow rule: a crawler that cannot fetch the page can never see its noindex. To keep a page out of the index, allow crawling and serve noindex through a meta robots tag or an X-Robots-Tag header, and add the Disallow only later, if at all, once the URL has dropped out.