Choose whether search engines may crawl your whole site, list folders to keep out, tick common rules for WordPress, online shops, internal search and filter parameters, and add your sitemap. The generator writes a valid robots.txt as you click and checks it with the same parser as our robots.txt tester.
There are checkboxes for the main AI crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, ClaudeBot, PerplexityBot, CCBot, Applebot-Extended, Bytespider, Meta and Amazon), and you can add rules for a single crawler. Download the file and upload it to the root of your site.
How to use the Robots.txt Generator tool
- Keep Allow everything unless the site must stay out of search (for example a staging copy).
- Add folders to block, tick the common rules you need and choose AI crawlers to block.
- Enter your sitemap URL.
- Click Download robots.txt and upload it so it opens at https://your-domain/robots.txt, then click Test it.
Why use ToolOrca Robots.txt Generator
- Allow-all or block-all starting point
- Presets for WordPress admin, internal search, cart and checkout, and sort/filter parameters
- One-click blocking of 12 AI crawlers
- Per-crawler Allow and Disallow rules that keep your general rules
- Sitemap and optional Crawl-delay lines
- Live syntax check and a link to test the result
A good default robots.txt
Most sites need very little: allow everything, block only areas with no search value (admin, cart, internal search results) and name the sitemap. Do not block CSS, JavaScript or image folders; Google needs them to render your pages and judge them as mobile-friendly.
Never use robots.txt to hide private content. The file is public, so a Disallow line tells everyone where the private folder is. Protect such areas with a password instead.
Why named crawler groups repeat your rules
A crawler follows only the most specific group that names it. If you add a group for Googlebot, Googlebot stops reading the * group. The generator therefore copies your general Disallow lines into every named group, so adding one special rule does not accidentally open the rest of the site.
Frequently asked questions
Where do I put robots.txt?
In the root folder of your site, so it opens at https://your-domain/robots.txt. Each subdomain needs its own file.
Will blocking Google-Extended remove me from Google Search?
No. Google-Extended only controls whether your content is used to train and ground Gemini. Googlebot, which crawls for Search, is separate.
How do I block my whole site?
Choose Block everything. It writes User-agent: * and Disallow: /. To remove pages that are already indexed, use noindex or Search Console's removal tool instead.
Does WordPress need a robots.txt?
WordPress serves a virtual one by default. Uploading a real file replaces it; keep /wp-admin/admin-ajax.php allowed, as the preset does.
Is anything sent to a server?
No. Everything is generated in your browser, so it works offline once the page has loaded and nothing you type leaves your device.