robots.txt Generator
Build robots.txt visually: User-agent groups, Allow/Disallow rules, crawl-delay and sitemap URLs, with crawler presets and templates.
User-agent: * Allow: / Sitemap: https://fuyeai.com/sitemap.xml
What is Robots Txt Generator
Build a valid robots.txt without memorizing syntax: User-agent groups with Allow/Disallow rules, presets for Googlebot, Bingbot, GPTBot, ClaudeBot and Google-Extended, plus WordPress and Next.js templates.
How to use Robots Txt Generator
- Start from a template or an empty config and add User-agent groups
- Add Allow/Disallow rules — every path must start with /
- Enter your sitemap URLs, then copy or download the robots.txt for your site root
Frequently asked questions
Does crawl-delay work for every crawler?
No — Google ignores crawl-delay entirely (rate is tuned in Search Console). Bing and several other crawlers do honor it; the value is in seconds.
Does Disallow keep a page out of search results?
Not fully. Disallow blocks crawling, but a linked page can still appear with no description. To truly deindex, allow crawling and serve a noindex meta tag instead.
Will blocking GPTBot or ClaudeBot hurt my Google ranking?
No. GPTBot is OpenAI’s training crawler and ClaudeBot is Anthropic’s — both are independent of Googlebot. Google-Extended only controls Gemini/AI training use, not Search.
Is Robots Txt Generator free?
Robots Txt Generator is completely free — open the page and use every feature with no hidden charges.
Do I need to download or install Robots Txt Generator?
No download or installation is needed. It is an online tool that runs in the browser on your phone, tablet and computer.
Do I need an account to use Robots Txt Generator?
No. Use the tool right away — no account or sign-in required.
Does Robots Txt Generator work on mobile?
Yes. The page is fully responsive and works on phones, tablets and desktop browsers.