Industry's Lowest Fees: 5% Buyer Fee | 10% Seller Fee — Save up to 75% vs competitors! Learn more

Robots.txt Generator

Generate robots.txt file for your website

Configure Robots.txt

Default Behavior

Sitemap URL

Crawl Delay (seconds)

0 = no delay (recommended for most sites)

Disallow Paths (one per line)

Allow Paths (one per line)

Common Blocks

/admin/ /wp-admin/ /cgi-bin/ /private/ /api/ /tmp/

Bot-Specific Rules

Block GPTBot (OpenAI) Block Claude-Web (Anthropic) Block Bingbot

Generated robots.txt

Important Notes

Place robots.txt in your root directory (e.g., example.com/robots.txt)
Robots.txt is a request, not a security measure
Always include your sitemap URL
Don't block important resources (CSS, JS, images)

Test Your Robots.txt

Google Robots.txt Tester Bing Robots.txt Tester

Live Activity

Join 2,500+ Marketers

Get quality backlinks & guest posts from verified publishers.

Start Free →

Verified Sites 4.9 Rating