About this tool
The Robots Exclusion Protocol (RFC 9309, stored as `robots.txt` in a website's root directory) standardizes crawl instructions between site owners and automated web search spiders (Googlebot, Bingbot). Configuring robots.txt instructs crawlers which directories to ignore (e.g., admin consoles, internal cart endpoints), specifies XML Sitemap locations, and optimizes SEO crawl budgets. This tool provides visual rules configuration and standard robots.txt generation.
How to Use This Tool (Step-by-Step)
Select Target User-Agents
Define rules targeting all automated crawlers (`User-agent: *`) or configure individual directives for Googlebot or Bingbot.
Add Allow & Disallow Directives
Specify directory paths to block (e.g., `/admin/`, `/cart/`) alongside explicit allow exceptions (`/api/v1/public/`).
Declare Sitemap and Export
Append your absolute XML Sitemap URL (`https://example.com/sitemap.xml`). Preview the standard output and download the `.txt` file.
RFC 9309 International Standard
Supports longest-prefix match resolution, wildcards (`*`), and end-of-line anchors (`$`) compliant with modern crawler standards.
Multi-Sitemap Declaration
Allows declaring multiple sitemap endpoints, guiding search bots to index language-specific and sectioned content maps efficiently.
100% Client-Side Privacy
File assembly executes entirely within browser memory. Internal administrative URLs and private paths are never uploaded.
Crawl Budget Optimization & robots.txt Gotchas
- •robots.txt is public: Anyone can read `/robots.txt`. Never list confidential backend paths like `/secret-admin/`, as this advertises targets to attackers.
- •Disallow is NOT Noindex: If external sites link to a disallowed URL, Google will still index its title and link. To completely prevent indexing, use `<meta name="noindex">`.
- •Never block CSS and JavaScript: Modern Googlebot renders pages fully with headless Chromium. Blocking assets causes Google to treat pages as mobile-unfriendly.
Code & Transformation Examples
Standard Modern Website robots.txt Benchmark
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /*?*sort=
# 明确声明站点地图地址
Sitemap: https://example.com/sitemap.xml• User-agent: * : 适用于所有搜索引擎蜘蛛
• Allow: / : 允许抓取整站所有公开页面
• Disallow: /admin/ : 禁止抓取管理后台
• /*?*sort= : 利用通配符禁止抓取带排序参数的重复页面
• Sitemap: : 告诉爬虫地图所在绝对路径* `robots.txt` must reside strictly in the domain root and use all-lowercase naming.
Blocking All Spiders for Staging Environments
User-agent: *
Disallow: /切勿将此配置错误部署到正式生产环境!一个斜杠 Disallow: / 会导致全站所有页面在一周内被 Google、百度全部清空移出搜索结果!* Always audit production robots.txt prior to launch to ensure `Disallow: /` was removed from staging.
Frequently Asked Questions
Q: Where must robots.txt be located?
A: It must reside strictly at the root of the domain: `https://example.com/robots.txt`. Spiders completely ignore files in subdirectories (`example.com/sub/robots.txt`).
Q: What happens if a website lacks a robots.txt file?
A: Returning HTTP 404 indicates zero crawl restrictions; bots assume full crawling permissions. However, declaring sitemaps via robots.txt improves crawl efficiency.
Q: Do all crawlers obey robots.txt directives?
A: Legitimate search engines adhere strictly to RFC 9309. Malicious scrapers ignore it. Defending against rogue scraping requires Web Application Firewalls (WAF).