Robots.txt Generator

Generate robots.txt files with user-agent rules, allow/disallow paths, sitemaps, and presets for common platforms.

Robots.txt Generator

Generate robots.txt files to control search engine crawling

Rule Group 1
Add sitemap URLs to help search engines discover your pages

Configure rules above and click "Generate"

What is a Robots.txt Generator?

A robots.txt generator creates the robots.txt file that tells search engine crawlers which pages and directories on your website they are allowed to access. The robots.txt file sits at the root of your domain and uses a simple syntax with User-agent, Disallow, Allow, and Sitemap directives to control crawling behavior.

CodeHelper's Robots.txt Generator provides a visual interface for building rules, quick presets for common platforms (WordPress, Next.js), support for multiple user-agent groups, and sitemap URL inclusion.

Key Features

  • Visual Rule Builder: Add user-agents, disallow paths, and allow overrides through a clean interface.
  • Quick Presets: One-click presets for Allow All, Block All, Standard, WordPress, and Next.js/Nuxt configurations.
  • Multiple Rule Groups: Create separate rules for different crawlers (Googlebot, Bingbot, etc.).
  • Sitemap Integration: Add sitemap URLs so crawlers can discover your page structure.
  • Download File: Download the generated robots.txt file ready to upload to your server.

How to use the Robots.txt Generator

  1. Choose a quick preset or build rules manually.
  2. Add disallow and allow paths for each user-agent.
  3. Add your sitemap URL(s).
  4. Click Generate, then copy or download the file.

Common robots.txt rules for developers

  • Block all crawlers: User-agent: * followed by Disallow: /, useful for staging environments.
  • Block specific paths: Disallow /admin/, /api/, or /private/ to keep internal routes out of search indexes.
  • Block AI training crawlers: The Block AI crawlers preset adds groups for GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, CCBot, PerplexityBot and others to opt out of AI training data collection.
  • Sitemap directive: Add Sitemap: https://example.com/sitemap.xml so crawlers automatically discover your page structure.
  • Crawl-delay: Set a crawl delay (in seconds) to limit server load from aggressive crawlers.

Whether you are launching a new website, blocking admin pages from search engines, or optimizing crawl budget, this free robots.txt generator ensures your file follows the correct syntax.

Frequently Asked Questions

Where do I put the robots.txt file?

At the root of the host, so it is reachable at https://example.com/robots.txt. Each subdomain and each protocol needs its own file, a robots.txt on www.example.com does not cover blog.example.com. Crawlers fetch it before any other page and cache it for up to a day.

Does Disallow remove pages from Google?

No. Disallow only stops crawling, a URL that is linked from elsewhere can still appear in results without a snippet. To keep a page out of the index, let it be crawled and add a noindex meta tag or X-Robots-Tag header, or protect it with a password. Never rely on robots.txt to hide private data.

How do I block AI training bots but keep search engines?

Use the Block AI crawlers preset. It keeps a permissive group for every other user agent and adds a Disallow: / group for GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, CCBot, PerplexityBot and similar agents. Google-Extended controls Gemini training without affecting Google Search, and well-behaved AI crawlers honour these rules.

What is the difference between Allow and Disallow, and does order matter?

Both are prefix rules. When several rules match a URL, Google and Bing apply the most specific one, the longest path, so Allow: /admin/login next to Disallow: /admin/ keeps the login page crawlable. Paths must start with a slash, and a wildcard * and an end anchor $ are supported by the major engines.

Related

toolCron Expression GeneratorVisual cron expression builder with presets, human-readable descriptions, and next execution time preview.articleFrom Git Push to Social Post: Building a Registry-Driven SEO and Automation PipelineHow a TypeScript registry pattern powers SEO, routing, and fully automated social media publishing for a Nuxt static site, using GitHub Actions, GitHub Secrets, and a self-hosted n8n instance on a VPStoolJSON ↔ YAML ConverterConvert JSON to YAML and YAML to JSON with auto-detection, syntax highlighting, and format indicators.toolREADME GeneratorProfessional README.md generator with templates, badge builder, and live preview. Create GitHub documentation with customizable sections, Shields.io badges, and markdown rendering.

Free forever, no ads, no tracking. Support the project