Build policy
Free · No sign-up

AI Robots.txt Generator & Access Checker

Set separate rules for AI search, user requests and training-related crawlers. Generate a robots.txt draft or check a bot’s access to a path.

Enter one path per line to block even for otherwise allowed bots.
Optional. Leave blank if you do not have a sitemap.

How to use this tool

Search crawlers, user-requested fetchers and training crawlers have different roles. Allowing a crawler does not guarantee inclusion in AI answers; blocking it depends on whether that crawler follows the rules.

  1. Choose Build policy and select which crawler groups to allow.
  2. Add protected paths and an optional sitemap URL.
  3. Generate, copy or download the draft. Merge it with existing rules rather than blindly replacing them.
  4. Publish robots.txt at the root of your site, then confirm the actual file and server behavior.

Examples and scope

The default permits OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User outside protected paths, while disallowing GPTBot, ClaudeBot and Google-Extended.

To test path rules, switch to Check access and paste an existing robots.txt. The checker handles exact user-agent groups, wildcard fallback and longest matching paths; it does not reproduce every crawler’s behavior or HTTP rules.

Frequently asked questions

Does Disallow make a page private?

No. robots.txt is a request to cooperative crawlers, not authentication or access control.

Does blocking GPTBot also block OpenAI search?

GPTBot and OAI-SearchBot have separate roles and controls. User-initiated requests can be treated differently from automatic crawling.

Does Google-Extended block Google Search?

Google-Extended controls certain Gemini-related uses and is not the Googlebot search crawler. Consult the current official documentation before applying a policy.

Official references

English edition reviewed on October 3, 2026. The tool is a starting point, not a substitute for checking your live website and current provider documentation.

About this tool

UsefulSet builds small browser-based tools. AI assists with development and explanations; automated tests check generation rules and edge cases. This is an independent tool and is not an official service of Google, OpenAI or Anthropic.

Contact and corrections

Report an issue to [email protected]. Include reproduction steps and the expected result, but remove personal or confidential information.

Privacy

Updated October 3, 2026. Your tool inputs and generated results are processed in this browser and are not sent to our server or analytics events. Copies and downloads remain in the clipboard or files you choose.

Cloudflare provides hosting and may process IP addresses, requested URLs and browser information for service delivery and security. Cloudflare privacy policy.

Google Analytics loads only if you allow usage analytics. It can collect page visits, device information and general tool-use, copy or download events using cookies and identifiers. Tool inputs and generated results are excluded from these events. You can change your choice in Analytics preferences; withdrawing consent stops future collection and does not automatically erase past records. Google privacy policy.

Your analytics choice and color theme are stored in this site’s browser storage. Email inquiries are used to respond and investigate issues; unnecessary information is deleted when no longer needed, subject to required retention. Contact us for access, correction or deletion requests.

AdSense ads are currently disabled. Before enabling ads, applicable consent requirements and Google-certified CMP settings must be reviewed. The analytics choice is not an advertising CMP. Advertising providers may use cookies and prior visits for ad delivery. Google ad settings · Google partner-site data use.

Terms of use

This free tool provides a draft and explanatory guidance. You are responsible for verifying the output, accuracy of your inputs, applicable requirements and the effect on your website. Availability, search rankings, crawler compliance and revenue are not guaranteed. Do not enter confidential customer data. Use generated output only after reviewing it against the relevant official specifications.