Enter a website or page address to fetch its robots.txt and test whether a URL may be crawled. The tester applies Google's rules: it picks the group for the most specific user agent, then the longest matching Allow or Disallow path wins, with * wildcards and the $ end anchor. It tells you the exact line that decided, and shows the result for 22 crawlers at once, including Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
The file is editable: change a rule and the results update instantly, so you can try a fix before you upload it. You can also paste a robots.txt without fetching anything. The tester also points out syntax errors, unknown directives, rules outside a group, and directives that Google ignores such as Crawl-delay and Noindex.
How to use the Robots.txt Tester tool
- Enter a site or page address and click Fetch robots.txt (or paste a file into the editor).
- Type the URL path to test, such as
/blog/post?ref=1. - Pick a crawler, or choose Other and type any user agent token.
- Read the verdict and the matching line, then check the table for all crawlers and the problem list.
Why use ToolOrca Robots.txt Tester
- Google's matching rules: most specific user agent group, longest path wins, Allow wins ties, * and $ wildcards
- Results for Googlebot, Googlebot-Image, Bingbot, GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended and more
- Shows the exact rule and line number behind every verdict
- Live editing: test changes before uploading them
- Finds syntax errors, misspelled directives, unsupported Noindex and rules before any User-agent
- Handles missing files (404 = allow all) and server errors (5xx = Google pauses crawling)
How Google reads robots.txt
A crawler obeys only one group: the one whose User-agent line matches its name most specifically. If a file has a group for Googlebot and one for *, Googlebot ignores the * group entirely, which surprises many site owners. Inside the group, the rule with the longest matching path wins; if an Allow and a Disallow match with the same length, Allow wins.
robots.txt controls crawling, not indexing. A blocked page can still appear in Google with no description if other sites link to it. To keep a page out of search results, allow crawling and add <meta name="robots" content="noindex"> instead.
Blocking AI crawlers
AI companies use separate user agents: GPTBot (OpenAI training), OAI-SearchBot and ChatGPT-User (ChatGPT search and browsing), ClaudeBot (Anthropic), PerplexityBot, CCBot (Common Crawl), Google-Extended (Gemini training; it does not affect Google Search) and Applebot-Extended. Add a group with Disallow: / for the ones you want to keep out; the robots.txt generator has checkboxes for them.
robots.txt is a request, not a lock. Reputable crawlers follow it, but it does not stop scrapers that ignore it.
Frequently asked questions
What happens if a site has no robots.txt?
If /robots.txt returns 404 (or another 4xx), crawlers treat everything as allowed. If it returns a server error (5xx) or times out, Google stops crawling the site until it can read the file again.
Does Google support Crawl-delay?
No. Google ignores Crawl-delay and adjusts its crawl rate itself. Bing and Yandex do use it.
Is the URL path case-sensitive?
Yes. Disallow: /Admin/ does not block /admin/. User-agent names, however, are not case-sensitive.
Can I test a robots.txt before uploading it?
Yes. Paste or edit the file in the editor; nothing is fetched or saved, and every change is tested instantly in your browser.
Is anything stored?
No. Our server only downloads the robots.txt you ask for; testing and editing happen in your browser.