robots.txt generator & tester

← All tools

Build a robots.txt or paste one to test whether a path is allowed for a given user-agent. Runs entirely in your browser — no live fetches.

~~~
Honest note: robots.txt is advisory. Well-behaved crawlers respect it; malicious bots ignore it. It is not access control — use auth and firewall rules for that.
~~~

About this tool

robots.txt tells crawlers which paths they may fetch. It lives at/robots.txt on your site root. Well-behaved bots read it before crawling; malicious scrapers ignore it entirely.

The generator builds rule groups with presets for common cases — allow all, block staging paths, or opt out of AI training crawlers. The tester simulates a path and user-agent without hitting your live server. It uses longest-match precedence with *wildcards and $ end anchors per RFC 9309; Allow wins on equal-length ties. robots.txt is not access control — pair it with auth for anything sensitive.

~~~

Read more