Robots.txt Generator
This tool writes a robots.txt file. You list the paths to block, add exceptions and a sitemap address, and can block common AI crawlers.
- Free, no sign-up
- Nothing is sent to a server
- Worldwide
How to use Robots.txt Generator online
- Name the crawler. An asterisk means all of them.
- List the blocked paths. One per line, each starting with a slash.
- Add a sitemap. And allowed exceptions if needed.
- Copy the file. Save it as robots.txt.
Why use this free Robots.txt Generator
- Disallow and Allow rules
- Sitemap line
- Crawl delay
- AI crawler block
- Checks for paths and addresses
- Share by link
What robots.txt is
Robots.txt is a small text file at the top of a website that tells crawlers, such as search engine bots, which parts of the site they may visit. Well-behaved crawlers read it before they crawl. It is a request, not a lock: it does not hide pages or stop anyone who ignores it.
The rules
A group begins with User-agent, which names the crawler, or an asterisk for all of them. Disallow lines list the paths it should avoid, and Allow lines make exceptions inside a blocked area. A Disallow with nothing after it allows everything, and Disallow: / blocks the whole site. The Sitemap line tells crawlers where your sitemap is.
A worked example
Blocking /admin/, /cart/ and /search for all crawlers, with a sitemap, gives:\nUser-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /search
Sitemap: https://example.com/sitemap.xml\nThe file has 6 lines. Save it as robots.txt in the top folder, so it can be read at your-site/robots.txt.
AI crawlers and delays
You can add groups that block common AI training crawlers, such as GPTBot, CCBot, ClaudeBot, Google-Extended and Applebot-Extended. Whether a company respects the file is its choice. A crawl delay in seconds asks some crawlers to slow down; major search engines ignore it or have their own settings.
Mistakes to avoid
Do not list secret addresses, because the file is public and shows the paths to anyone. Do not block files that pages need, such as CSS and scripts, since that can hurt how search engines see your pages. To keep a page out of results, use a noindex tag instead. Paths are case-sensitive and must start with a slash.
Using the tool
Change the text or any option of the Robots.txt Generator and the answer moves straight away. Open a text file into the box, or drop one on it.
Robots.txt Generator: frequently asked questions
Where does robots.txt go?
Does it hide a page?
How do I block the whole site?
Can I block AI crawlers?
What does an empty Disallow mean?
Can anyone see the text I put into the Robots.txt Generator?
Can I copy the Robots.txt Generator result?
Related searches for Robots.txt Generator
- Robots.txt Generator online
- free Robots.txt Generator
- Robots.txt Generator online free
- Robots.txt Generator for mobile
- Robots.txt Generator without sign up
- Robots.txt Generator – Text & Writing Tools
Robots.txt Generator is a free online tool on MAITBOX, in the Text & Writing Tools group. It runs in your browser, so what you type is not sent to a server. It works on Android and iPhone, and on Windows, Mac and Linux, with no download and no account.