Is AI actually crawling your site? Read your logs for these bots

The bots, and what each one is for

The vendor pages are OpenAI's bots documentation, Anthropic's crawler help article and Perplexity's bots guide. Each publishes a list of IP ranges as JSON. One name you will see recommended in blocking guides is missing from the table on purpose: Google-Extended is documented as a standalone product token with no separate HTTP user agent, so it controls how Google may use content but never shows up as a visitor in a log.

Where your logs are

This depends entirely on your hosting. A server you run yourself writes access logs by default. A static host or CDN may keep none, or offer them only through a paid or separate feature, so look at what yours provides before you conclude anything from an empty result. Browser-based analytics cannot see these bots at all, because they do not run your tracking script. And since most retrieval crawlers do not execute JavaScript, as the AI citation guide notes, a page whose content only exists after scripts run can be fetched and still look empty to them.

The script

It reads a standard combined-format access log, counts requests from the eight user agents above, and checks each request's source IP against the vendor's published ranges. Run it with the command below:

python3 ai_bots.py access.log

import re, sys, json, ipaddress, urllib.request, collections

BOTS = {  # user-agent token: (operator, purpose, published IP ranges)
    'GPTBot':          ('OpenAI',     'training crawler',      'https://openai.com/gptbot.json'),
    'OAI-SearchBot':   ('OpenAI',     'search crawler',        'https://openai.com/searchbot.json'),
    'ChatGPT-User':    ('OpenAI',     'user-requested fetch',  'https://openai.com/chatgpt-user.json'),
    'ClaudeBot':       ('Anthropic',  'training crawler',      'https://claude.com/crawling/bots.json'),
    'Claude-SearchBot':('Anthropic',  'search crawler',        'https://claude.com/crawling/bots.json'),
    'Claude-User':     ('Anthropic',  'user-requested fetch',  'https://claude.com/crawling/bots.json'),
    'PerplexityBot':   ('Perplexity', 'search crawler',        'https://www.perplexity.ai/perplexitybot.json'),
    'Perplexity-User': ('Perplexity', 'user-requested fetch',  'https://www.perplexity.ai/perplexity-user.json'),
}

LINE = re.compile(r'^(\S+) .*?"(?:GET|HEAD|POST) (\S+) [^"]*" (\d{3}) .*?"[^"]*" "([^"]*)"\s*$')

ranges = {}
def published(url):
    if url not in ranges:
        req = urllib.request.Request(url, headers={'User-Agent': 'ai-bot-check/1.0'})
        data = json.load(urllib.request.urlopen(req, timeout=20))
        ranges[url] = [ipaddress.ip_network(p.get('ipv4Prefix') or p.get('ipv6Prefix')) for p in data['prefixes']]
    return ranges[url]

def verified(ip, url):
    addr = ipaddress.ip_address(ip)
    return any(addr in net for net in published(url))

hits = collections.defaultdict(lambda: {'n': 0, 'ok': 0, 'paths': collections.Counter(), 'codes': collections.Counter()})
for line in open(sys.argv[1], encoding='utf-8', errors='replace'):
    m = LINE.match(line)
    if not m:
        continue
    ip, path, code, ua = m.groups()
    for token, (_, _, url) in BOTS.items():
        if token.lower() in ua.lower():
            h = hits[token]
            h['n'] += 1; h['paths'][path] += 1; h['codes'][code] += 1
            h['ok'] += verified(ip, url)
            break

if not hits:
    print('No requests from the listed AI user agents.')
for token, h in sorted(hits.items(), key=lambda kv: -kv[1]['n']):
    operator, purpose, _ = BOTS[token]
    print('%-16s %-10s %-22s %4d requests, %d from published IP ranges, %d not' % (token, operator, purpose, h['n'], h['ok'], h['n'] - h['ok']))
    print('    top paths: ' + ', '.join('%s (%d)' % p for p in h['paths'].most_common(3)) + '   status: ' + ', '.join('%s x%d' % c for c in h['codes'].items()))

It was tested on a synthetic eight-line log (not real traffic) containing three GPTBot requests, one of them from an address outside OpenAI's published ranges, plus ChatGPT-User, ClaudeBot and PerplexityBot requests from addresses inside their ranges, a Googlebot line and an ordinary browser line. The output, trimmed to the summary lines:

GPTBot           OpenAI     training crawler          3 requests, 2 from published IP ranges, 1 not
    top paths: /hosting/ (2), /seo/seo-audit-checklist (1)   status: 200 x3
ChatGPT-User     OpenAI     user-requested fetch      1 requests, 1 from published IP ranges, 0 not
ClaudeBot        Anthropic  training crawler          1 requests, 1 from published IP ranges, 0 not
PerplexityBot    Perplexity search crawler            1 requests, 1 from published IP ranges, 0 not

Googlebot and the browser line were correctly ignored, and the spoofed GPTBot request was flagged. One practical detail from building it: one vendor's range file returned a 403 to Python's default user agent, so the script sends its own.

Reading the results

A page that these bots can fetch still has to be one your own search ranking supports. If your pages are not in Google's index, see Discovered, currently not indexed and Crawled, currently not indexed. The llms.txt study found no measured link between that file and AI citations, so do not read its absence in your logs as a signal either way.

robots.txt: independent settings

The vendors say each token is controlled separately. OpenAI says a site can allow OAI-SearchBot to appear in search while disallowing GPTBot, and Perplexity says each setting works independently. Both say changes can take up to about 24 hours to apply. Anthropic documents blocking ClaudeBot with a plain rule and says it supports the non-standard Crawl-delay extension; it also warns that blocking IP addresses may not reliably opt you out, because it can stop the crawler reading your robots.txt.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

That example blocks OpenAI's training crawler while allowing its search crawler. This site's own robots.txt is a single rule that allows every crawler, so none of these bots are blocked.

What this does not cover