Structured crawl control for WordPress

Better Robots.txt Pro and Better Robots.txt Premium replace the default WordPress robots.txt workflow with a guided configuration process. Instead of working from a blank textarea, you can select a starting preset, review plain-language options, adjust crawler policies, and inspect the generated file before saving it.

The paid editions are designed for websites that need broader governance, stronger protection options, and more detailed control over automated access. They create on the guided workflow and essential crawl controls available in the product family while adding advanced AI, crawler, cleanup, and protection modules.

Manage different crawler categories

Better Robots.txt helps separate crawler decisions instead of treating every automated agent the same way. You can work with search engine crawlers, AI and LLM crawlers, SEO utilities, bad bots, archive services, feed crawlers, social preview bots, and other automated agents.

• Set advanced policies for AI training crawlers and AI search or discovery crawlers.
• Use per-bot AI crawler rules and custom user agents in the advanced editions.
• Manage AI usage signals such as search, ai-input, and ai-train preferences.
• Generate llms.txt and use additional machine-readable governance documents.
• Control archive and Wayback access, feeds, crawl traps, and social preview crawlers.
• Review WooCommerce crawl cleanup options and rules for CSS, JavaScript, and image crawling.

AI-related agents are organized into practical groups, including training crawlers, search and discovery crawlers, and user-action fetchers. The product documentation notes that user-action fetches may not follow robots.txt in the same way as automated crawling, so stronger enforcement may require server logs, WAF rules, provider verification, or IP-based controls where available.

Presets and advanced rules

Setup begins with preset choices that help match the configuration to your website. Essential provides a clean starting point for general websites. AI-First focuses on AI-ready governance without shutting down discovery. Fortress is intended for websites seeking stronger protection against scraping, archive capture, and unnecessary crawl activity. Custom provides a starting point for more tailored decisions.

After choosing a preset, you can revisit the settings, modify individual modules, and regenerate the robots.txt file. Advanced directives such as crawl-delay and custom rules are also supported, giving experienced site owners more control over the final output.

Review before publishing

The final Review & Save step lets you inspect the generated robots.txt before it goes live. This makes the workflow practical for both beginners who prefer guided explanations and advanced users who want to verify the resulting directives before publishing.

Pro and Premium editions

Pro adds advanced governance and protection modules, including additional AI, crawler, and cleanup controls. Premium unlocks the most restrictive and advanced protection options, including the Fortress preset and additional high-control modules. Options in the interface are marked by edition so you can identify which capabilities require a paid plan.

Installation and setup

Install the Better Robots.txt add-on in WordPress, activate it, and open its settings workflow. Select a preset, configure the available crawler and governance controls, review the generated robots.txt, and use Review & Save when the output matches your intended policy. Settings can be revisited and regenerated later.