Generators

Robots.txt Generator

Crawler Rules

Build Your Robots.txt

Configure crawler access rules below and generate a ready-to-publish robots.txt file.
Quick Templates
User Agents & Rules

No robots.txt generated yet

Configure your settings above and click Generate robots.txt to create your file.

Tool features

Professional robots.txt generation with SEO best practices
4 features

SEO Control

Control web crawler access to your website

Multiple User Agents

Configure rules for different crawlers

Sitemap Integration

Include sitemap URLs and host preferences

Crawl Delay

Control crawling speed and server load

What is robots.txt & how it works

How you tell crawlers where they may go

robots.txt is a plain-text file at the root of your site (https://example.com/robots.txt) that tells search engine crawlers which paths they may and may not crawl. Each group starts with a User-agent line followed by Allow and Disallow rules, and the file can point crawlers to your XML sitemap.

It controls crawling, not indexing or access: a disallowed URL can still appear in search results if other sites link to it, and the file is public, so it is no place to hide private areas. Use it to keep crawlers out of admin pages, search results and duplicate content, and to save crawl budget for the pages that matter.

User-agent
Which crawler a group of rules applies to; * matches all of them.
Disallow / Allow
Path prefixes the crawler must skip, or may crawl despite a broader Disallow.
Sitemap
Absolute URL of your XML sitemap, so crawlers can find every page.

How to use this tool

Create a robots.txt file in four steps
1
Set the default rules
Choose whether all crawlers may crawl the site, and list the paths to disallow.
2
Add specific crawlers
Add groups for individual user agents, such as Googlebot or AI crawlers, if they need different rules.
3
Add your sitemap
Enter the full URL of your XML sitemap and an optional crawl delay.
4
Publish and test
Upload the file as /robots.txt at the site root, then check it with the Robots.txt Tester.

Related tools

Keep debugging with tools from the same suite
4 tools
Robots.txt Tester
Test any URL against a site’s robots.txt — which crawlers may fetch it, which line decides it, and every syntax error that quietly disables a rule
Crontab Generator
Generate cron expressions for scheduled tasks with visual interface and examples
OCSP Checker
Check SSL certificate revocation status using the Online Certificate Status Protocol (OCSP)
HTTP Header Checker
Analyze HTTP response headers for any website or URL with detailed header information

Frequently asked questions

Common questions about robots.txt
5 Q&A

At the root of each host, e.g. https://example.com/robots.txt. Crawlers do not look in subfolders, and each subdomain needs its own file.

No. Disallow stops crawling, but a blocked URL can still be indexed from links elsewhere. To keep a page out of search results, allow crawling and add a noindex meta tag or X-Robots-Tag header instead.

No. Googlebot ignores Crawl-delay; Bing and some other crawlers honour it.

Add a group per crawler with Disallow: /, for example User-agent: GPTBot, User-agent: Google-Extended or User-agent: CCBot. Compliance is voluntary.

Disallow: with no path allows everything for that user agent. Disallow: / blocks the whole site, which is a common and costly mistake on a live site.