Robots.txt Checker
Check whether robots.txt is reachable, valid, and isn't accidentally blocking pages you want indexed, as part of one unified CheckSitePulse scan.
Scan your website
Free account, real checks, a prioritized fix list.
What the robots.txt check covers
File reachability
Whether robots.txt exists at the root and returns a normal response.
Directive parsing
Whether the Allow/Disallow rules parse the way crawlers expect.
Accidental blocking
Rules that unintentionally disallow important pages or your whole site.
Sitemap declaration
Whether robots.txt points crawlers to your sitemap.
Part of one unified scan
Robots.txt health feeds into the same crawl as the sitemap check and the SEO check — one crawl, one prioritized fix list.
Need help fixing what the scan finds?
Run the check yourself, free. If you would rather have someone fix the issues for you, these services cover them.