Robots.txt Tester

Checks whether a specific user-agent (e.g. Googlebot) can access a page by reading the rules in robots.txt. Handy to diagnose indexing blocks before going live.

How to use it: Provide the URL to test and the user-agent. The tool will display the matching Allow/Disallow rule and the reason behind the verdict.

We might appear in the target site's logs; avoid internal URLs or ones with credentials.