How to Actually Know If AI Bots Can Access Your Website

Google used to be, in most cases, the main way people found their way to your website. Now you also have to take into consideration the different AIs out there: ChatGPT, Gemini, Perplexity, Claude.

Each of these platforms runs several bots that crawl websites to build their own indexes, because a user asked them to, to train their models… The question here is: can they do that without any problem, or will they hit some kind of block along the way?

The Windguru Examples and the “False Block” from robots.txt

Before diving into the official documentation, I want to walk through two cases where AI couldn’t read a page’s content with its user-request bots, for different reasons:

Windguru and Its Complicated Relationship with AI

I already touched on AI’s trouble accessing Windguru in my article on the future of SEO. It’s a clear example that, even when you name the source yourself, the AI will go ahead and answer by citing different sources if it can’t get to the one you asked for.

ChatGPT says Windguru’s content is stuck on “Loading”. That’s because it didn’t execute the JavaScript and couldn’t see the content. If we disable JavaScript via Chrome’s console, we get a decent idea of what ChatGPT actually saw.

Robots.txt It’s Blocking Me… But That’s Not the Real Story

Before auditing a website, I asked Claude to tell me what the business was about. Its answer: it couldn’t access the page because robots.txt was blocking it.

However, after checking the robots.txt file and reading its actual directives, there was no block at all.

User-agent: *
Disallow: /wp-content/uploads/wpo-plugins-tables-list.json
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: */?s=
Allow: /wp-admin/admin-ajax.php

Sitemap: https://silveiratech.pt/sitemap_index.xml

If robots.txt wasn’t the problem, what was causing the block? In this guide we’ll look at some of the most common bots, the ways to control their access according to each company’s own official sources, and how to check whether they can actually access and read a page’s content.

The Bots We’ll Cover

Each AI platform runs several bots with distinct jobs. Knowing what each one is for helps you make better decisions about whether you actually need to block it or not.

There are three types:

• Index bots: they save the URL along with its content in a database. The AI uses this index to answer users and cite websites that might be useful.
• Training bots: the content gets used to build the model. It ends up baked in, with no links or sources, and doesn’t update until the next training run.
• User-request bots: they only show up when someone gives the AI a URL and asks it to read it. They don’t crawl anything on their own.

Not every company has all three. OpenAI and Anthropic do. Perplexity only has index and user-request bots: it says explicitly that it doesn’t use what it crawls to train its models (opens in a new tab). Google and Apple use the same bot for several jobs, with a separate mechanism just for training.

Google-Extended (opens in a new tab) and Applebot-Extended (opens in a new tab) are how you decide if that training happens. Google-Extended also decides whether Gemini uses your content when it answers.

Ways to Control Access for These Bots

Every site is different. Maybe to protect your servers, to have more control over how your content gets used, or [insert whatever reason applies to you], you might want to block one of these AI bots.

Whatever the reason, let’s break down the different ways you can block these bots, from the lightest to the most forceful:

Robots.txt

A file SEOs have known about for ages. It’s so widely used that the AI companies themselves have adopted this standard, so site owners can control how accessible these bots are to their sites.

OpenAI and Perplexity both warn that a robots.txt change can take about 24 hours to take effect.

Claude-User is the only user-request bot that, according to its own company, respects robots.txt. That’s why the excuse from the start of this guide was plausible, even though it turned out not to be the case.

Groups Don’t Add Up, They Replace Each Other

One mistake that con occur in robots.txt: if you have general rules under User-agent: * and then give another bot its own group, that bot stops obeying anything under * and only follows its own specific directives.

In this robots.txt, for example, you end up opening the search and cart folders to GPTBot, even though you’d closed them off for everyone else:

User-agent: *
Disallow: /buscar/
Disallow: /carrito/

User-agent: GPTBot
Allow: /

If you give a bot its own group, repeat whatever rules you want it to keep following (opens in a new tab). And if you’re not planning to treat it any differently on your site, the general rules already have you covered.

Why Use robots.txt?

Robots.txt doesn’t block anything by itself: it sets out guidelines, and each bot decides whether to follow them. A bot that doesn’t want to play along can just ignore it. Even so, there are good reasons to use it:

• It distinguishes by purpose. You can shut the door on GPTBot, which trains models, while letting OAI-SearchBot in, since it can actually bring you traffic.
• It’s the only way to opt out of training for Google and Apple without losing their search engines, via Google-Extended and Applebot-Extended.
• It’s easy to maintain. A text file you write once and edit whenever you need to.
• The majority of the bots respect it. Every company in the table states that its index and training bots follow it.

Given how simple it is, I think it should be the first option for controlling how AI bots crawl your site.

Search Console: Controlling Generative AI

Since August 31, 2026, Search Console lets you decide whether your content shows up in AI Overviews, AI Mode and Discover’s generative AI features.

There are three options: include (the default), exclude, or inherit the setting from the parent property.

According to Google, this setting:

• Isn’t a ranking signal: it doesn’t affect your position in Search.
• Doesn’t stop Gemini’s training. That’s what Google-Extended is for.
• Doesn’t remove you from Google. That’s what noindex is for.

Exclusion isn’t instant: Google says content gets excluded within one to two days of the change taking effect, and that caching can push it a bit further out. The AI’s answer will just get built from other sites’ content instead. What you lose is the citation, the impressions and the traffic.

nosnippet: Doesn’t Block Entry, Limits The Use

nosnippet doesn’t control access. The bot can still get into your page and read it. What you’re doing here is asking it not to show certain content in its results. You can apply it in three ways. If two rules clash, the more restrictive one wins:

In the HTTP header, useful for files where you can’t add tags, like a PDF:

X-Robots-Tag: nosnippet

In the page’s <head>, to affect the whole thing:

<meta name="robots" content="nosnippet">

As an HTML attribute, to mark just one specific part of the content:

<span data-nosnippet>Este texto no se mostrará</span>

For Google, nosnippet affects all of its results, AI Overviews and AI Mode included. To be cited with a link in those features, Google requires the page to be eligible to show with a snippet (opens in a new tab). So a page-level nosnippet, whether in the or in the HTTP header, leaves you out of those citations.

For Apple, it’s how you stay out of Siri’s AI answers and Apple’s search, something robots.txt doesn’t control. Apple says to target it specifically at its bot:

<meta name="applebot" content="nosnippet">

If you need to apply it to a PDF or an image, Apple supports the same directive via header: X-Robots-Tag: applebot: nosnippet

No other company in this guide documents that it respects nosnippet.

Blocks at the CDN or Server Level

These blocks are a lot more drastic. You’re no longer suggesting, like with robots.txt — you’re enforcing a rule. The bot gets blocked directly, with something like a 403, and it can’t get in. It’s the real barrier against bots that don’t respect robots.txt, but it’s also the one that can backfire the hardest.

Before going further, it’s worth understanding where a request actually travels. First DNS resolves, deciding where it goes. If you’re using a CDN like Cloudflare, the request lands on its network: the firewall there decides whether it passes, and then it’s served from cache if there’s a copy saved. If there isn’t, the request continues on to your server, which might have its own protections before it hands over the file.

There are three ways to block:

Blocking by user-agent name is easier to get around. That’s why these companies publish ways to verify that whoever claims to be their bot actually is: all of them publish their crawlers’ IP ranges, and Google, Microsoft, Apple and Common Crawl also support verification through confirmed reverse DNS.

You have to be careful with IP-based blocks. The classic mistake is blocking US addresses and taking Googlebot down with them, since it mostly crawls from there. Google says that if it detects a site is blocking US requests, it may try crawling from other countries instead (opens in a new tab), but that’s not something you should count on.

The CDN

Sticking with Cloudflare as an example, it lets you block AI bots from its dashboard by behavior, using the same categories we’ve already seen: Search, Agent and Training. For each one you can choose to block it site-wide, block it only on pages with ads, or not block it at all. You can also go bot by bot with AI Crawl Control. Three nuances for anyone using this platform, like me:

• On the free plan, AI Crawl Control only recognizes bots by name. A bot that spoofs one of these user agents could slip past this check.
• It only works if your traffic actually goes through Cloudflare. If your domain is in “DNS only” mode — the grey cloud in the dashboard — Cloudflare just returns your server’s IP, visits go there directly, and none of these rules ever get applied.
• The old Block AI bots toggle has been in the process of being retired since September 15, 2026. And watch out for the new defaults on new domains: Training and Agent bots come blocked by default on pages that show ads.

The Server or Hosting

The most popular servers, Apache and Nginx, support all three types of blocking. The option providers recommend is combining user-agent and IP address: rejecting anything that claims to be an official bot without actually coming from its official IPs.

This creates a maintenance headache: IP lists change. Microsoft recommends not hardcoding addresses in your config and, if you use their list, updating it daily (opens in a new tab). Fail to keep the lists updated and you risk blocking something you never meant to.

If you’re on shared or managed hosting, your provider might be applying bot protections automatically. This is the hardest cause to spot: on the surface there’s no block, and the bot still gets an error anyway. In that case, the move is to ask your host what protections it has turned on.

Methods That Don’t Control Access

There are directives that can get mistaken for access blocks:

meta noindex or X-Robots

These directives control indexing, not access. On my own site, for example, the cookies page carries noindex:

<meta name="robots" content="noindex, follow">

It’s not in Google. And yet ChatGPT reads it without any trouble:

The same goes for X-Robots-Tag, which is just the header version of the same instruction.

Just because they don’t block access doesn’t mean they don’t have an effect. Google requires a page to be indexed to show up as a supporting link in AI Overviews and AI Mode, so a noindex locks you out of that too (opens in a new tab).

llms.txt

It’s a file some sites use to hand AIs a neatly organized version of their content. The idea is reasonable enough, but none of the companies in this guide mention it anywhere in their crawler documentation. Having one doesn’t hurt, but it doesn’t control their bots’ access to your site either.

Access The Content Isn’t the Same as Reading IT: Content Behind JavaScript

A bot being able to get into your page doesn’t mean it can read it. Content that depends on JavaScript complicates accessibility, and some bots may not execute it at all. With client-side rendering (CSR), the bot can end up with an empty skeleton, exactly like we saw with Windguru at the start.

Carlos Sánchez Donate tested this with ten different AIs, replicating an earlier experiment by Andre Alpar: he put a discount code on a page that only shows up if JavaScript runs, then asked each AI about it.

According to his latest update, from June 25, 2026, with five tests per AI, neither ChatGPT, Claude, Gemini, Perplexity nor Copilot found the code. Grok did execute the JavaScript, but ignored the result. Only Qwen got it consistently, with Kimi, DeepSeek and Mistral giving irregular results. Here’s the link to his article on SEO and LLMs (opens in a new tab).

Official sources don’t say much, and what little they do say isn’t exactly encouraging: Apple confirms Applebot does render (opens in a new tab), Common Crawl says CCBot doesn’t execute JavaScript (opens in a new tab), and everyone else doesn’t mention it at all.

Why? None of them explain it, but cost is the most likely reason. Rendering takes a lot more machinery and time than just downloading HTML. Google, which has been doing this for years, describes it like this: it queues pages up and renders them whenever its resources allow. And for user-request bots, the cost is even more obvious: there’s someone waiting on the other end, and rendering adds seconds to every query the AI runs against every source.

The upshot is the same one we’ve been repeating with Google for years, just now with more bots in the mix: the more your content depends on JavaScript, the harder it’ll be for AIs to read it.

How to Know If AI Can Read Your Content

There are a few ways to check whether your page’s content loads via JavaScript:

Look at the HTML as It Arrives in Your Browser

The fastest way is to open the page’s source code: Cmd+Option+U on Mac, Ctrl+U on Windows. That’s the HTML exactly as it arrives, with no JavaScript executed. You can use Cmd+F to search for the important content: a headline, a price, a paragraph… If it’s not there, it’ll be invisible to any bot that doesn’t execute JavaScript. And as we’ve seen, that’s most of them.

You can also do this from the terminal, identifying yourself as one of the bots along the way, with a command like this:

curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://tuweb.com/pagina

For Googlebot there’s an official route: Search Console’s URL Inspection makes the request from Google itself and shows you the rendered HTML exactly as it sees it.

View the Page Without JavaScript

Another way to check is to browse your site the way a bot that doesn’t execute JavaScript would:

1. Open the page in Chrome.
2. Open DevTools (F12, or Cmd+Option+I on Mac).
3. Press Cmd+Shift+P (Ctrl+Shift+P on Windows), type “Disable JavaScript” and hit Enter.
4. Reload the page without closing DevTools.

Unlike the source code view, here you see your site already assembled and can keep browsing it. You’ll be able to spot things like menus that won’t open, or product pages that come up empty…

Another tip about robots.txt: be careful not to block JavaScript or CSS folders your content needs to render. Apple says this explicitly about Applebot.

How to Audit Whether AI Can Access Your Content

So far we’ve covered which bots exist, how to control them, and the whole JavaScript situation on the client side. When it comes to accessibility, what you’ve configured and what’s actually happening can be two different things. To know what’s really going on with your site, you need to audit who’s actually accessing it:

The Problem with Online Tools

There are dozens of tools out there that check whether AI bots can get into your site. The usual method goes like this: they request your page identifying themselves as GPTBot, ClaudeBot and the rest, then check how your server responds. If they get a 403, they tell you you’re blocking that bot.

Otterly.AI, one of the best-known tools, explains it this way in its AI Crawler Simulation: “It sends actual HTTP requests using the specific user-agent strings of each AI crawler, but does not check robots.txt files — it only tests how the server responds to the crawler’s user-agent.” In other words, it sends real requests using each crawler’s user-agent and just checks how the server responds to that name. (Retrieved 30/09/2026)

Here’s the problem: that request isn’t coming from OpenAI’s IPs or Anthropic’s, it’s coming from the tool provider’s own data center. As we’ve seen, official sources recommend checking both the user-agent and the IP. A site that follows that advice will see a GPTBot that isn’t coming from OpenAI, and correctly block it, if that’s how it’s configured.

Result: the tool reports a block that doesn’t actually exist. If a tool tells you you’re blocked, verify it before touching anything.

Logs: The Best Evidence of Who’s Really Getting In

Perplexity, Google and Microsoft all recommend not relying on the user-agent name alone, but also checking the official IPs, to make sure the bot hitting your site is actually the AI it claims to be.

To know what’s really going on, you have to look at the logs. They record every request your site gets: the date, the IP, the URL requested, the response code and the user-agent. There can be two kinds of logs — the CDN’s (if you use one), which sees every request, and the server’s own, which only sees what the firewall let through.

The most practical approach is to search for the name within the user-agent, rather than the full string: version numbers change often, and almost every bot has multiple variants, desktop and mobile.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +claude-user@anthropic.com)

From there, it’s time to dig through it and check whether your setup is actually working the way you think. Among other things, you can spot:

• Bots you actually want getting an error, especially a 403, which means a deliberate block.
• Bots that read your robots.txt and then never request anything else.
• Bots accessing paths that were supposed to be disallowed in robots.txt.
• Bots that never show up at all, which means they’re not crawling you.

Watch out for situations that can muddy the results, like paywalls or CAPTCHA-style challenges, which return a 200 and look like legit visits.

Finally, the step that gives you certainty: checking each visit’s IP against that bot’s official list, or against reverse DNS. That’s how you separate genuine AI visits from fake ones.

Impersonation does happen: Common Crawl warns that some crawlers pose as CCBot.

If an IP isn’t on the list, the honest conclusion is “unverified.” It could be an impersonator, but it could also be an outdated list, or the wrong one: Google publishes several, depending on the bot.

Ask the AI Itself If It Can Access Your Content

For a quick check, there’s a method anyone can use: just ask the AI to open one of your pages. It has the advantage that the request comes from the provider’s real IPs, so it’s a genuine visit.

The trick is in how you phrase it. “Can you access my site?” doesn’t work, because a yes proves nothing. Ask for something it could only know if it had actually read the page: a specific line, a price, the title of your latest post.

What does this page say about [something very specific]? https://yoursite.com/page

If it gives you back that detail and it matches what’s actually on your page, it got in and read it. If it answers with vague generalities or information that isn’t there, ask it what it got:

What did you see when you opened the page?

That’s where the interesting answers show up: a “Loading”, a blank page, an error… And also the wrong ones, like Claude’s at the start of this guide. Whether it accessed the page or not, you can trust. The reason it gives you for it, not so much.

Two limits:

• It only tests the user-request bot, so it tells you nothing about the ones building the index they’d cite you from.
• It tests one page, not the whole site.

Tracking It with Analytics Tools

Analytics tools like Google Analytics show you the people landing on your site from an AI’s answer. Someone asked a question, the AI cited you, and that person clicked through. It’s the final payoff of everything we’ve covered in this guide, and the sign that the whole chain actually worked: the bot got in, read your content, the AI used it, and someone ended up on your site.

That makes it a good early-warning system. If a site averages 50 visits a month from ChatGPT and that suddenly drops to zero, something’s changed. Maybe you’ve stopped showing up in its answers, or maybe something’s blocking access.

What analytics won’t give you is the diagnosis: it gives you clues about what’s happening, not why.

One note: for that traffic to get attributed properly, it’s worth checking how your analytics groups AI sources. If it doesn’t recognize them, that traffic can end up lumped in elsewhere and you won’t spot it as easily. It’s one of the first things I check when I work on a project’s AI SEO.

All three methods complement each other. Analytics helps with day-to-day tracking, and asking the AI directly can settle the occasional one-off question. Log analysis is the tool to reach for when you notice a change or want to audit the site’s accessibility properly.

A Summary of AI Accessibility

The key points, in five

1. Each company runs several bots with different jobs. Blocking the training bot isn’t the same as blocking the one building the index it cites you from, even if they belong to the same company.
2. Weigh the pros and cons of each control method. robots.txt is the simplest and most flexible option; server-level blocks are the most restrictive.
3. A bot getting in doesn’t mean it reads. If your site depends on JavaScript, most AIs will struggle to see the content.
4. Online tools can give false positives, because they test from their own IPs. A “blocked” result can mean the exact opposite: a site correctly checking who’s knocking at its door.
5. Logs are the most reliable source, cross-checking the user-agent against each provider’s official IPs.

This guide on AI bot accessibility grew out of my own experience auditing and analyzing websites. Not just from the examples above, but from the odd 403 I’ve run into myself, spoofing an AI bot’s user-agent in Chrome’s console, that turned out not to be a real block at all — just a site checking that I wasn’t who I claimed to be.

Digging into this, I came to the conclusion that, while these tools can be useful in plenty of cases, they can also set off false alarms and push you toward the wrong decisions. The AI’s own self-reported diagnosis isn’t always reliable either. This guide is my attempt to shed some light on that, and to help you avoid the false scares I’ve run into myself in the past.

I’ve tried to be as rigorous as possible with every official source. If you’ve made it this far and found something that doesn’t add up or has gone out of date, I’d be glad to fix it.

If you don’t have time to run through all of this yourself, get in touch and I’ll review your site: I’ll tell you which AIs are getting in, which ones are genuine, and what they find when they do.

Sources