Should Your Small Business Block AI Bots? A Plain-English robots.txt Guide for 2026

Open your website’s robots.txt file and there’s a good chance it was written years ago, by a plugin, a theme or a developer you no longer talk to. Meanwhile, the visitors reading your site have changed. Alongside Googlebot, your pages are now fetched by ChatGPT, Claude, Perplexity and Gemini, sometimes to train a model and sometimes because a real customer just asked, “Who’s the best plumber near me?”

Illustration of a website gate with a shield letting some AI bots through and stopping others at a barrier

That makes one small text file a business decision. Block the wrong bot and you can quietly disappear from AI answers. Allow everything without thinking and you hand your content to model training without ever choosing to. This guide explains, in plain English, which AI bots exist, what each one actually does, and what we recommend for most small businesses in the US and UK.

Short version: for most local and service businesses, the smart default is to allow the bots that power AI search and answers, make a deliberate choice about the bots that collect training data, and check that your security tools aren’t blocking AI visitors without you knowing.

Why this matters right now

Three things changed recently.

First, AI search is now a real way people find businesses. Google is putting ads inside AI Mode answers, has said those queries are on average three times longer than traditional searches, and plans to bring AI Mode ads to the UK in Q4 2026 (Google Ads & Commerce Blog). When the biggest search company starts selling space in AI answers, it’s a sign those answers are where customers are looking.

Second, the rules for saying “don’t train on my content” are getting simpler. On October 30, Google said it will support a shared robots.txt signal for AI training preferences, based on work at the IETF (the body that sets internet standards), alongside similar moves from Bing, OpenAI and Anthropic (Search Engine Land). Google’s own crawler documentation now notes it intends to support this signal, written as user-agent: __usage_ai-training, in early 2027 (Google Search Central). The working group behind it is called AI Preferences (aipref).

Third, a lot of sites are blocking AI crawlers without the owner realising. In July 2025 Cloudflare changed its default so that AI crawlers are blocked for new domains unless the site owner allows them (Cloudflare blog). That’s a reasonable choice for big publishers. For a dental clinic or a roofing company that wants to be recommended, it may be the opposite of what you want.

Not all AI bots do the same job

The most common mistake we see is treating “AI bots” as one thing. The major AI companies actually run separate bots for separate purposes, and they’ve documented them publicly. Broadly there are three kinds.

1. Search and answer bots (these bring you customers)

These bots index your pages so an AI assistant can find and cite you when someone asks a question.

OAI-SearchBot (OpenAI). OpenAI says sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers,” though they can still appear as navigational links (OpenAI crawler docs).
Claude-SearchBot (Anthropic). Anthropic says disabling it “may reduce your site’s visibility and accuracy in user search results” (Anthropic help centre).
PerplexityBot (Perplexity). Perplexity says it is used to surface and link websites in its search results and “is not used to crawl content for AI foundation models” (Perplexity crawler docs).

Our view: if you want new customers, allow these. Blocking them is like asking to be left out of the phone book.

2. User-triggered fetchers (a real person asked about you)

When someone pastes your link into ChatGPT or asks Claude to check your opening hours, the assistant visits your page on that person’s behalf. These are ChatGPT-User, Claude-User and Perplexity-User. OpenAI and Perplexity both note that because a user started the request, robots.txt rules may not apply to these fetchers. So robots.txt isn’t the main lever here anyway, but there’s rarely a good reason for a small business to try to block a prospective customer’s assistant.

3. Training crawlers (your content helps build the model)

These collect content that may be used to train future AI models.

GPTBot (OpenAI). OpenAI says disallowing it “indicates a site’s content should not be used in training” its foundation models, and that this setting is independent of OAI-SearchBot.
ClaudeBot (Anthropic). Blocking it signals that your future content should be excluded from Anthropic’s training data.
Google-Extended (Google). This isn’t a separate crawler; it’s a control token. Google says it governs whether its crawled content can be used to train Gemini models and for grounding in Gemini Apps, and that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”

Our view: this is the genuinely optional part, and it’s your call. More on how to decide below.

What we recommend for most small businesses

Here’s the honest trade-off. Training crawlers don’t send you clicks directly. But models learn what businesses exist, what they’re known for and how they’re described partly from the open web. If you sell a local service, being well understood by AI is usually worth more than keeping your service pages out of a training set. Your pricing page and “areas we cover” page aren’t trade secrets; they’re marketing.

So for a typical plumber, salon, clinic, contractor, consultant or small online shop, we suggest:

Allow search and answer bots: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
Allow Googlebot and Bingbot as normal (never block these by accident).
Usually allow training crawlers (GPTBot, ClaudeBot, Google-Extended), unless you fall into one of the exceptions below.
Block private areas for every bot: admin pages, cart and checkout, account pages, internal search results and staging copies.

A sensible starting robots.txt for a WordPress site might look like this:

# Search engines and AI search: welcome
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Everyone: keep private areas out
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /?s=

Sitemap: https://www.example.com/sitemap_index.xml

Change the paths to match your site (Shopify, for example, generates its own robots.txt with its own rules), and replace the sitemap address with yours. If you’d rather not hand-write it, our free llms.txt and AI robots.txt generator builds both files for you in a couple of minutes.

One thing to watch: robots.txt isn’t the only place AI bots get turned away. Security plugins, hosting firewalls and CDN bot settings can block AI crawlers even when your robots.txt welcomes them, so check those settings too before assuming the file tells the whole story.

When blocking training makes sense

We’d lean towards disallowing GPTBot, ClaudeBot and Google-Extended if:

Your content is the product, such as paid courses, original research, recipes, premium guides or detailed industry reports.
You’ve invested heavily in original photography or long-form writing that competitors could benefit from.
Your industry has legal or confidentiality expectations around published material.

Even then, keep the search bots allowed. OpenAI and Anthropic both treat search and training as separate settings, so you can say “find me, don’t train on me.” That’s the best of both for most content-heavy businesses.

The hidden blockers: check these before anything else

Many sites that “aren’t showing up in ChatGPT” don’t have a robots.txt problem at all. The block is somewhere else.

Your CDN or firewall. Cloudflare and other security services can block AI crawlers at the network level, before robots.txt is even read. Check your bot settings, and remember Cloudflare’s 2025 default change for new domains. Perplexity’s docs even include notes on allowing its bots through common firewalls.
Security plugins. Some WordPress security and “bad bot” plugins block unfamiliar user agents. That’s good for scrapers, but it can catch legitimate AI search bots too.
A leftover “discourage search engines” setting. In WordPress, the reading setting meant for development sites sometimes survives launch. It’s worth a 10-second check.
Content that only loads with JavaScript. If your opening hours, prices or service areas only appear after scripts run, some bots may never see them. Put important facts in plain HTML.

One security note: user-agent names can be faked. Google explicitly warns that its user agent strings “can be spoofed”, and OpenAI, Anthropic and Perplexity publish IP ranges or verification guidance for this reason. If you let a bot through your firewall, allow it by verified IP range rather than by name alone, so attackers can’t simply pretend to be ChatGPT.

Where llms.txt fits in

robots.txt tells bots where they may go. llms.txt is a newer, separate idea: a short Markdown file at yoursite.com/llms.txt that tells AI agents what your business is and which pages matter most. The proposal’s author, Jeremy Howard, says thousands of sites now publish one, and OpenAI, Anthropic and Google’s Gemini publish llms.txt files for their own developer docs (llmstxt.org).

Let’s be clear about what it isn’t: it’s not a ranking factor anyone has confirmed, and it won’t fix a site that bots can’t reach. But it’s cheap, it’s harmless, and it gives AI tools a clean summary of your services, locations and contact details instead of making them dig through menus and pop-ups. We think every small business site should have one, and you can generate yours free.

A 15-minute AI bot check for your site

Visit yoursite.com/robots.txt in a browser. Look for any Disallow: / under OAI-SearchBot, Claude-SearchBot, PerplexityBot, or under User-agent: *.
Log in to your CDN or firewall (if you use one) and look for an “AI bots” or “AI crawlers” setting. Decide on purpose, don’t inherit a default.
Check your security plugin’s bot blocking list.
In WordPress, go to Settings, then Reading, and make sure the option to discourage search engines is unticked.
Decide on training crawlers (GPTBot, ClaudeBot, Google-Extended) using the guidance above, and write it down so the next developer knows it was a choice.
Publish an llms.txt file.
Ask ChatGPT and Perplexity about your business and service area a few days later, and note what they say and which sources they cite.

One last tip: robots.txt is unforgiving. A single Disallow: / under User-agent: * blocks your entire site from every crawler, including Google. After any change, open yourdomain.com/robots.txt in a browser and read it line by line.

If that list feels like a lot, we’ll do it for you. Our free site audit checks your security, speed, Google rankings and AI visibility, including whether AI bots can actually reach your site, and we reply within 24 hours.

Frequently asked questions

Will blocking GPTBot remove my business from ChatGPT?

Not on its own. OpenAI treats GPTBot (training) and OAI-SearchBot (search) as independent settings. Blocking GPTBot signals you don’t want your content used for training; blocking OAI-SearchBot is what keeps you out of ChatGPT search answers.

Does blocking Google-Extended hurt my Google rankings?

No. Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls use of your content for Gemini training and grounding.

Do AI bots actually obey robots.txt?

The major companies say their automatic crawlers do. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not, because a person asked for that page. Bad actors ignore robots.txt entirely, which is why it’s a preference signal, not a security control. Use a firewall for actual protection.

What is the new AI training opt-out Google announced?

It’s a shared robots.txt signal from the IETF’s AI Preferences work that lets you state an AI training preference once for all participating providers instead of listing each bot. Google says it intends to support it in early 2027, and Google-Extended keeps working meanwhile.

Is llms.txt the same as robots.txt?

No. robots.txt controls access. llms.txt is a helpful summary that points AI agents to your most important pages. You’ll want both.

I’m on Shopify, not WordPress. Does this still apply?

Yes. The bots are the same. Shopify generates robots.txt for you, and changes are made through a theme template rather than by uploading a file, so the steps differ but the decisions don’t.

The bottom line

AI assistants are becoming a front door to your business. Make sure the bots that power AI search can get in, make a conscious choice about training, and check that nothing upstream is quietly blocking the whole thing. It’s one of the cheapest visibility wins available to a small business right now. Start with our free llms.txt and AI robots.txt generator, or get a free site audit and we’ll check it for you.