Free Robots.txt Tester - Check if a URL is Blocked

Fetching happens through U2L's stateless fetch-text endpoint, because a browser cannot read another site's robots.txt directly under CORS. The tool takes whatever URL you enter, reduces it to its origin, and requests /robots.txt at that origin; the endpoint follows up to five redirects, times out after 8 seconds, and returns the body (first 1 MB), the final URL, the HTTP status, and the content type. The status is interpreted the way crawlers interpret it: a 2xx body is parsed; any 4xx other than 429 is reported as no robots.txt, which Google treats as unrestricted crawling; a 5xx is reported as unreachable, which RFC 9309 says a crawler must treat as complete disallow and which Google handles by halting crawling for 12 hours, then using its cached copy for up to 30 days.

Parsing follows RFC 9309 section 2. Each line is stripped of comments after #, split at the first colon into a case-insensitive field and a value, and classified. Consecutive User-agent lines open one group; the first Allow or Disallow after them closes the agent list; the next User-agent line starts a new group. Rules that appear before any User-agent line are flagged and ignored, matching crawler behaviour. A UTF-8 byte order mark is detected and reported, since Google ignores it but some parsers treat the first line as malformed. Sitemap lines are collected from anywhere in the file, and Crawl-delay is recorded with a note that Google does not support it. Anything else (Host, Clean-param, Noindex, typos such as Disalow) is listed as an unknown directive.

Group selection mirrors Google's parser: the user-agent value is reduced to its product token (letters, hyphens, and underscores, so Googlebot/2.1 becomes googlebot) and compared case-insensitively with the crawler you chose. If several groups name the same token, their rules are merged, as RFC 9309 requires. Sub-crawlers fall back to their parent: Googlebot-Image uses a googlebot-image group if there is one, otherwise googlebot, otherwise *. If no group matches and there is no * group, the crawler is unrestricted. Matching a path uses the RFC's most-specific-match rule. Each rule's path is converted to an anchored pattern where * means any run of characters and a trailing $ means end of URL; every rule that matches the test path is a candidate, the candidate with the longest path string wins, and an Allow beats a Disallow of identical length. An empty Disallow: value matches nothing, which is why Disallow: on its own line allows everything.

The tested URL is normalised with the URL API so you can paste either /path?query or a full URL; the path and query are matched, the fragment is dropped, and non-ASCII characters are percent-encoded before comparison, as RFC 9309 section 2.2.2 specifies. The result shows the verdict, the winning rule with its line number, the group that was selected and whether it was the specific token or the * fallback, and every other candidate rule so you can see why a longer Allow overrode a shorter Disallow. The raw file is printed with line numbers and the winning line highlighted. Nothing is stored: the fetch is a pass-through with a 60-second edge cache header, and the parser and matcher run entirely in your browser, so pasted drafts never leave your machine.