Tell AI assistants what your business is about, and choose which AI crawlers may read your site. Fill in the form and get two ready-to-upload files. Everything runs in your browser and nothing is sent to us.
When you’re done, yourdomain.com/llms.txt and yourdomain.com/robots.txt should open in a browser as plain text.
wp-config.php (often public_html) with your host’s File Manager or SFTP. Yoast SEO and AIOSEO can also create one for you.robots.txt.liquid template. Keep Shopify’s default rules and add the AI sections underneath./llms.txt to that file’s link.Sitemap: line as /sitemap.xml. Shopify creates it for you.public/ folder (Astro, Next.js, Vite) or the root of your static site.public/ are served as they are at /llms.txt and /robots.txt.@astrojs/sitemap, your sitemap is /sitemap-index.xml.No hype. llms.txt is new, and here’s what it can and can’t do.
llms.txt is a plain-text Markdown file at yourdomain.com/llms.txt that gives AI assistants a short summary of your site and links to its most useful pages. It was proposed by Jeremy Howard in 2024 and is documented at llmstxt.org. Think of it as a curated table of contents for AI, not a list of every page like a sitemap.
Nobody can promise that. llms.txt is a community proposal, not an official standard, and none of the big AI companies has said it uses llms.txt to decide which businesses to mention. Adoption is growing: according to llmstxt.org, thousands of sites publish one, and AI labs publish them for their own developer docs. It takes ten minutes and has no downside, but your content, reviews and reputation still matter far more.
robots.txt sets rules: which crawlers may visit which parts of your site. llms.txt gives guidance: what your site is about and where the important information is. They work together. A crawler blocked in robots.txt shouldn’t fetch your llms.txt either.
Most businesses want to be found, so we recommend allowing AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, which let ChatGPT, Claude and Perplexity show and link your pages. Training crawlers such as GPTBot, ClaudeBot and CCBot collect content to train AI models. Blocking them doesn’t remove you from AI search results, according to each vendor’s documentation.
No. Google says Google-Extended doesn’t affect your inclusion or ranking in Google Search. It controls whether your content is used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. Google’s AI Overviews are part of Search and follow your Googlebot rules.
No. robots.txt is a request, not a lock. Reputable crawlers follow it, but fetchers that act on a user’s request (such as ChatGPT-User and Perplexity-User) may not apply robots.txt rules, according to their vendors, and bad bots ignore it completely. To actually enforce limits you need firewall or bot-management rules on your server or CDN.
No. The files are built in your browser with JavaScript. Nothing you type is uploaded. If “Remember in this browser” is on, your answers are saved only in this browser’s local storage, and “Clear form” removes them.
These two files are a good start. Our free site audit checks your AI crawler rules, structured data, SEO, security and speed in one go, and you get the results in plain English.
Crawler names and descriptions were last checked against each vendor’s documentation on 2 November 2026. Vendors add and rename crawlers, so check these pages if something looks off.
Bytespider (ByteDance) has no official public documentation, so it’s listed based on its widely reported user-agent name.
We use analytics cookies to see how visitors use our site. You can accept or decline. See our Privacy Policy.