Menu
PDF Tools
Image Tools
Converters & Utilities
Seller & Print Tools
Accounting & Business
Calculators
Text & Writing Tools
Developer & Web Tools
Creator & Social Tools
Browser Utilities & Games
Document & Design Makers
Text & Developer Tools
Exam & Form Tools

Robots.txt Generator

This tool writes a robots.txt file. You list the paths to block, add exceptions and a sitemap address, and can block common AI crawlers.

How to use Robots.txt Generator online

  1. Name the crawler. An asterisk means all of them.
  2. List the blocked paths. One per line, each starting with a slash.
  3. Add a sitemap. And allowed exceptions if needed.
  4. Copy the file. Save it as robots.txt.

Why use this free Robots.txt Generator

  • Disallow and Allow rules
  • Sitemap line
  • Crawl delay
  • AI crawler block
  • Checks for paths and addresses
  • Share by link

What robots.txt is

Robots.txt is a small text file at the top of a website that tells crawlers, such as search engine bots, which parts of the site they may visit. Well-behaved crawlers read it before they crawl. It is a request, not a lock: it does not hide pages or stop anyone who ignores it.

The rules

A group begins with User-agent, which names the crawler, or an asterisk for all of them. Disallow lines list the paths it should avoid, and Allow lines make exceptions inside a blocked area. A Disallow with nothing after it allows everything, and Disallow: / blocks the whole site. The Sitemap line tells crawlers where your sitemap is.

A worked example

Blocking /admin/, /cart/ and /search for all crawlers, with a sitemap, gives:\nUser-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /search

Sitemap: https://example.com/sitemap.xml\nThe file has 6 lines. Save it as robots.txt in the top folder, so it can be read at your-site/robots.txt.

AI crawlers and delays

You can add groups that block common AI training crawlers, such as GPTBot, CCBot, ClaudeBot, Google-Extended and Applebot-Extended. Whether a company respects the file is its choice. A crawl delay in seconds asks some crawlers to slow down; major search engines ignore it or have their own settings.

Mistakes to avoid

Do not list secret addresses, because the file is public and shows the paths to anyone. Do not block files that pages need, such as CSS and scripts, since that can hurt how search engines see your pages. To keep a page out of results, use a noindex tag instead. Paths are case-sensitive and must start with a slash.

Using the tool

Change the text or any option of the Robots.txt Generator and the answer moves straight away. Open a text file into the box, or drop one on it.

Robots.txt Generator: frequently asked questions

Where does robots.txt go?
In the top folder of the site, at /robots.txt.
Does it hide a page?
No. It asks crawlers not to visit, but the page stays reachable.
How do I block the whole site?
Set everything else to blocked, which writes Disallow: /.
Can I block AI crawlers?
Yes, with the AI option. Respecting it is up to each company.
What does an empty Disallow mean?
Everything is allowed.
Can anyone see the text I put into the Robots.txt Generator?
Your text stays on this device when you use the Robots.txt Generator. The editor may keep the last text locally and offers to bring it back; you can dismiss it.
Can I copy the Robots.txt Generator result?
Press Use result as input and the Robots.txt Generator answer becomes the new input.

Related searches for Robots.txt Generator

  • Robots.txt Generator online
  • free Robots.txt Generator
  • Robots.txt Generator online free
  • Robots.txt Generator for mobile
  • Robots.txt Generator without sign up
  • Robots.txt Generator – Text & Writing Tools

Robots.txt Generator is a free online tool on MAITBOX, in the Text & Writing Tools group. It runs in your browser, so what you type is not sent to a server. It works on Android and iPhone, and on Windows, Mac and Linux, with no download and no account.

More tools in Text & Writing Tools →

Search tools