obfus.link

Robots.txt Generator emits canonical robots.txt from structured rule definitions and simulates URLs against the rules before deployment. The SEO Impact Simulator uses Google's path-matching algorithm with longest-prefix-wins and Allow precedence on ties. Warnings flag critical SEO mistakes including blocked CSS or JavaScript files, total-site Disallow, and missing sitemap directive recommendations.

Tier 2generators✓ TDD Verified

Robots.txt Generator

Generate robots.txt with SEO Impact Simulator and pre-deploy URL testing

Disallow (one per line)
Allow (one per line)
Disallow (one per line)
Allow (one per line)
Each URL is tested against every user-agent rule group · paths or absolute URLs both accepted

How to use

  1. Add rule groups — Click Add Group for each User-agent. Use * for the wildcard rule that covers all crawlers, or specific names like Googlebot or Bingbot.
  2. Fill allow / disallow — One pattern per line. Supports * wildcards (e.g., /api/*/internal) and $ end-anchor (e.g., /*.pdf$). Empty Disallow means allow everything.
  3. Optional: sitemap & crawl-delay — Add an absolute sitemap URL and a crawl-delay in seconds. Note: Googlebot ignores Crawl-delay.
  4. Optional: simulate URLs — Paste one URL per line to test against your rules. The simulator tests each URL against every agent group using Google's longest-prefix-wins algorithm.
  5. Run and review warnings — Click Generate. Read the SEO Warnings panel for critical mistakes (blocked CSS/JS, total-site Disallow) before deploying.
Read technical article

MCP / API

Call robots_txt_gen directly from any MCP-compatible agent:

// MCP TypeScript SDK
const result = await client.callTool({
  name: "robots_txt_gen",
  arguments: {
    "rules": []
  }
});

// curl
curl -X POST https://obfus.link/mcp \
  -H "Authorization: Bearer <SPT>" \
  -H "Content-Type: application/json" \
  -d '{"method":"tools/call","params":{"name":"robots_txt_gen","arguments":{"rules":[]}}}'

Related tools

Sitemap Validator →
Validate XML sitemaps with a crawl budget analyzer for SEO health
URL Parser →
Parse any URL into components with deep query decoding and security analysis
Header Inspector →
OWASP-graded HTTP security headers scorecard with CORS issue detection

FAQ

What is the difference between Disallow and Allow when both match?

Google's algorithm chooses the rule with the longest matching pattern. If Allow and Disallow have equal pattern length, Allow wins (this lets you carve out exceptions like Allow: /admin/public/ inside a broader Disallow: /admin/).

Why does the simulator flag blocked CSS files as CRITICAL?

Modern search engines render pages with CSS to detect layout, mobile-friendliness, and content visibility. Blocking CSS can demote your site in rankings, trigger mobile-usability penalties, and cause Googlebot to misjudge above-the-fold content. The same applies to blocked JavaScript — crawlers execute JS to render client-rendered content.

Are robots.txt rules a security boundary?

No. robots.txt is advisory — well-behaved crawlers respect it, but malicious bots and scrapers ignore it entirely. Sensitive paths must be protected with authentication, IP allowlisting, or server-side authorization. Treat robots.txt as an SEO control, not a security control.

Does the simulator handle * and $ correctly?

Yes. * matches any sequence of characters (zero or more). $ anchors the pattern to the end of the URL path (so /*.pdf$ matches /file.pdf but not /file.pdf?download=1). Multiple patterns are evaluated by longest-prefix-wins, with Allow taking precedence on equal-length ties.

Can I use this tool via the MCP API?

Yes. The tool is registered on the obfus.link MCP server at https://obfus.link/mcp. Call it from any MCP-compatible agent with a Shared Payment Token. The MCP tool name matches the snake_case slug shown in the integration snippet.