Robots.txt tester
Paste a robots.txt file, choose a crawler, and enter either a complete URL or only its path. The result shows the user-agent group and the exact rule that decides the test.
example.com/robots.txt; this page only evaluates the text you paste.Includes common search and AI crawler presetsNo rule tested yet.
- Tested path
- —
- Matched group
- —
- Matched rule
- —
How it works
Start with Blocked example and Allowed example. They use the same pasted file but select different crawler and path combinations.
- 01
Open the website's /robots.txt file and paste its contents into the left field.
- 02
Choose a crawler. Enter a full URL such as https://example.com/preview/article or only /preview/article.
- 03
Read the verdict together with the tested path, matched group, and winning rule. Use the two example buttons if you want to see both outcomes first.
Rules this tester evaluates
- User-agent groups and multiple groups with the same specificity
- Allow and Disallow rules
- Wildcard asterisks and end-of-path dollar markers
- Longest-match selection with Allow winning an equal-length tie
Important boundaries
- It does not download the live file or verify its HTTP status and content type.
- It does not normalize percent-encoded octets or reproduce crawler-specific extensions.
- A robots.txt rule controls crawler access. It is not a password, and an allowed URL is not guaranteed to be indexed.
Technical reference
The matching order and group selection follow the core Robots Exclusion Protocol rules in RFC 9309.