Skip to content

Robots.txt Tester

Find out whether Googlebot, Bingbot or an AI crawler may fetch a URL, and why.

This tool loads the page through our server, because a browser cannot read other websites directly. Only the address you enter is sent; results are kept for a few minutes and not stored. Privacy

Or paste a robots.txt below and test it without fetching anything.

Edits are tested instantly. Nothing is saved or sent anywhere.

Report a problem

Don't include personal or sensitive information.

What happened, and what did you expect? (10–2000 characters) 0 / 2000

Your browser and system, screen size and recent error messages on this page. Never what you typed or your files. Privacy

What is sent
 

Enter a website or page address to fetch its robots.txt and test whether a URL may be crawled. The tester applies Google's rules: it picks the group for the most specific user agent, then the longest matching Allow or Disallow path wins, with * wildcards and the $ end anchor. It tells you the exact line that decided, and shows the result for 22 crawlers at once, including Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and Google-Extended.

The file is editable: change a rule and the results update instantly, so you can try a fix before you upload it. You can also paste a robots.txt without fetching anything. The tester also points out syntax errors, unknown directives, rules outside a group, and directives that Google ignores such as Crawl-delay and Noindex.

How to use the Robots.txt Tester tool

  1. Enter a site or page address and click Fetch robots.txt (or paste a file into the editor).
  2. Type the URL path to test, such as /blog/post?ref=1.
  3. Pick a crawler, or choose Other and type any user agent token.
  4. Read the verdict and the matching line, then check the table for all crawlers and the problem list.

Why use ToolOrca Robots.txt Tester

  • Google's matching rules: most specific user agent group, longest path wins, Allow wins ties, * and $ wildcards
  • Results for Googlebot, Googlebot-Image, Bingbot, GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended and more
  • Shows the exact rule and line number behind every verdict
  • Live editing: test changes before uploading them
  • Finds syntax errors, misspelled directives, unsupported Noindex and rules before any User-agent
  • Handles missing files (404 = allow all) and server errors (5xx = Google pauses crawling)

How Google reads robots.txt

A crawler obeys only one group: the one whose User-agent line matches its name most specifically. If a file has a group for Googlebot and one for *, Googlebot ignores the * group entirely, which surprises many site owners. Inside the group, the rule with the longest matching path wins; if an Allow and a Disallow match with the same length, Allow wins.

robots.txt controls crawling, not indexing. A blocked page can still appear in Google with no description if other sites link to it. To keep a page out of search results, allow crawling and add <meta name="robots" content="noindex"> instead.

Blocking AI crawlers

AI companies use separate user agents: GPTBot (OpenAI training), OAI-SearchBot and ChatGPT-User (ChatGPT search and browsing), ClaudeBot (Anthropic), PerplexityBot, CCBot (Common Crawl), Google-Extended (Gemini training; it does not affect Google Search) and Applebot-Extended. Add a group with Disallow: / for the ones you want to keep out; the robots.txt generator has checkboxes for them.

robots.txt is a request, not a lock. Reputable crawlers follow it, but it does not stop scrapers that ignore it.

Frequently asked questions

What happens if a site has no robots.txt?

If /robots.txt returns 404 (or another 4xx), crawlers treat everything as allowed. If it returns a server error (5xx) or times out, Google stops crawling the site until it can read the file again.

Does Google support Crawl-delay?

No. Google ignores Crawl-delay and adjusts its crawl rate itself. Bing and Yandex do use it.

Is the URL path case-sensitive?

Yes. Disallow: /Admin/ does not block /admin/. User-agent names, however, are not case-sensitive.

Can I test a robots.txt before uploading it?

Yes. Paste or edit the file in the editor; nothing is fetched or saved, and every change is tested instantly in your browser.

Is anything stored?

No. Our server only downloads the robots.txt you ask for; testing and editing happen in your browser.