🤖 Robots.txt Generator

Create a robots.txt file controlling how search engines crawl your website.

Configuration

All Robots
Googlebot (Google)
Bingbot (Bing)
Yahoo Slurp
DuckDuckBot
Baiduspider
YandexBot
Facebook Bot
Twitter Bot

📖 Common Disallow Paths

PathPurpose
/admin/Admin area
/wp-admin/WordPress admin
/cgi-bin/CGI scripts
/tmp/Temp files
/private/Private files
/*.pdf$All PDFs
/?s=Search results