🤖Robots.txt Checker

Fetch, parse and test a site’s robots.txt rules

About the Robots.txt Checker

Robots.txt Checker fetches a site’s robots.txt, parses the rules per user-agent, lists declared sitemaps, and lets you test whether a specific path is allowed or blocked.

It is the fastest way to confirm crawlers can (or can’t) reach your important pages.

Common use cases

Confirming important pages aren’t accidentally disallowed
Checking which user-agents have special rules
Finding the sitemap URL declared in robots.txt
Testing whether a path is crawlable

How to use the Robots.txt Checker

Enter a domain.
Click Check to fetch /robots.txt.
Review the parsed rules and test a path against the User-agent: * rules.

Frequently asked questions

What happens if there is no robots.txt?

A missing robots.txt (404) means crawlers may access everything by default.

Does Disallow remove a page from Google?

No — Disallow blocks crawling, not indexing. To remove a page from results use a noindex meta tag (and allow crawling so it’s seen).

How is “allowed” decided?

For conflicting rules, the most specific (longest matching) Allow or Disallow path wins.