Can AI assistants read your website?
We read your robots.txt file the way each AI and search crawler does, and tell you who is allowed in.
Why it matters
ChatGPT, Claude, Gemini, Perplexity and Copilot can only recommend what they can read. Many websites block their crawlers by accident: a security plug-in, a hosting setting or a copied robots.txt file.
Blocking the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) keeps you out of AI answers. Blocking the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) is a choice some businesses make; it can mean the assistants know less about you.
How to unblock a crawler
Open yourbrand.com/robots.txt. Remove the “Disallow: /” line under the crawler's name, or the whole group. If a plug-in or your host writes the file, look for a setting about bots or AI crawlers.