Check whether any crawler — search or AI — can fetch a given path, using Google's actual matching rules.
Matched "Allow: /admin/public/" in the "*" group. The most specific matching rule wins (14 characters, excluding wildcards).
Rules never merge across groups — a named group replaces the wildcard entirely.
robots.txt looks simple and behaves subtly. Most people assume rules are read top to bottom and that the first match wins. They are not, and it does not. This tester implements the specification Google actually follows, so the answer matches what the crawler will really do.
No, and this is the most consequential misunderstanding in SEO. robots.txt stops crawling, not indexing. Google can still list a blocked URL it found through links, showing it without a description. To remove a page, allow crawling and use a noindex meta tag.
Add a group per bot: GPTBot for OpenAI training, ClaudeBot for Anthropic, PerplexityBot for Perplexity, and Google-Extended for Gemini training. Note that Google-Extended never appears as a user-agent string — it exists only as a robots.txt token, and blocking it does not affect Google Search.
The major ones do. It is a voluntary protocol with no enforcement, so scrapers and some AI crawlers ignore it. If you need to guarantee a bot cannot fetch a page, block it at the server or CDN.
Usually because a longer Disallow matched. Specificity is measured in characters excluding wildcards, so "/admin/" (7) beats "/a*" (2). Paste both rules above and the tester will show which one won and why.
Google ignores it entirely — set the crawl rate in Search Console instead. Bing and Yandex do honor it.
robots.txt tells crawlers what they may do. RankVault's crawler analytics shows what they actually did — including which AI bots are reading your content right now.