Traffic dropped overnight: a 30-minute triage, in the order that rules things out fastest

1. Is the drop real? (2 minutes)

Open Search Console, Performance, and compare the last 7 days to the previous 7. If clicks are flat there but analytics fell, your tracking broke, not your rankings. Also check the shape: a drop that starts at an exact date and stays flat points to something that changed on that date, such as a deploy. A gradual slide points elsewhere, usually toward content or competition.

2. What did you ship? (10 minutes)

Look at deploys from the day before the drop. The failures that hide whole sites from Google are boring:

• A noindex meta tag or an X-Robots-Tag: noindex header left over from staging.
• A robots.txt that now disallows the site, or a robots.txt that errors (Google documents that a 5xx on robots.txt can pause crawling).
• Canonical tags pointing at another host or a staging domain.
• Key pages returning 404, 500 or a redirect loop.

Checking by eye across a few hundred URLs is slow, so I use a script. It reads a list of URLs and reports each problem it finds, using Google's own robots.txt rules for Googlebot.

The checker script

Put your top landing pages (from the Performance report) in a text file, one URL per line:

import re, sys, urllib.request, urllib.robotparser, urllib.parse, urllib.error

UA = 'launch-check/1.0'

def fetch(url):
    req = urllib.request.Request(url, headers={'User-Agent': UA})
    try:
        with urllib.request.urlopen(req, timeout=20) as r:
            return r.status, r.geturl(), r.headers, r.read().decode('utf-8', 'replace')
    except urllib.error.HTTPError as e:
        return e.code, url, e.headers, ''

robots = {}
def allowed(url):
    root = '{0.scheme}://{0.netloc}'.format(urllib.parse.urlparse(url))
    if root not in robots:
        rp = urllib.robotparser.RobotFileParser()
        code, _, _, body = fetch(root + '/robots.txt')
        # 4xx (no file) means allow all; 5xx means treat as unreachable, so this script treats both as allow-all
        rp.parse(body.splitlines() if code == 200 else [])
        robots[root] = rp
    return robots[root].can_fetch('Googlebot', url)

for url in (l.strip() for l in open(sys.argv[1]) if l.strip() and not l.startswith('#')):
    status, final, headers, html = fetch(url)
    issues = []
    if status != 200:
        issues.append('returns %s' % status)
    if 'noindex' in (headers.get('X-Robots-Tag') or '').lower():
        issues.append('X-Robots-Tag header says noindex')
    for m in re.finditer(r'<meta[^>]+name=["\'](?:robots|googlebot)["\'][^>]*content=["\']([^"\']*)', html, re.I):
        if 'noindex' in m.group(1).lower():
            issues.append('meta robots says noindex')
    c = re.search(r'<link[^>]+rel=["\']canonical["\'][^>]*href=["\']([^"\']+)', html, re.I)
    if not c:
        issues.append('no canonical')
    elif urllib.parse.urljoin(final, c.group(1)).rstrip('/') != final.rstrip('/'):
        issues.append('canonical points to %s' % urllib.parse.urljoin(final, c.group(1)))
    if not re.search(r'<title>\s*\S', html, re.I):
        issues.append('empty or missing title')
    if not allowed(final):
        issues.append('blocked for Googlebot by robots.txt')
    print(('FIX  ' if issues else 'ok   ') + url)
    for i in issues:
        print('       - ' + i)

Run on this site's live URLs it reported something real: pages whose canonical points at an .html address that redirects, which is the exact conflict covered in Google picked a different canonical. It also reported one false alarm of my own making, a mistyped URL that returned 404, which shows why it is worth reading the output before reacting. An early version also flagged every page as blocked, because Python's robots parser treats a 403 on robots.txt as "disallow everything"; the script now fetches robots.txt itself.

3. Search Console messages (5 minutes)

Check Security & Manual actions for both reports, then Pages (Page indexing) for a jump in "Excluded" or "Not indexed" reasons. If the count of indexed pages fell on the same date as the traffic, you have found the shape of the problem. The reasons it lists are explained in Discovered, currently not indexed and Crawled, currently not indexed.

4. Is Google reporting something? (3 minutes)

Google keeps a Search Status Dashboard listing ranking and indexing incidents and update rollouts with start and end dates. If your drop coincides with an open core update, Google's guidance is to wait until the rollout finishes before judging, then compare performance to before the update. Nothing in that guidance promises a quick recovery or a specific fix, so be wary of anyone who does.

5. Which queries and pages lost clicks? (10 minutes)

In Performance, compare the two periods and sort by click difference. Then ask: are the losses concentrated on a handful of pages, one folder, one query type, one device? Concentrated losses point at a page-level or template problem. A uniform drop across everything, with impressions falling too, points at indexing or site-level trust. If impressions held but clicks fell, the pages still rank and something changed in how they appear, which is the territory of low-CTR pages and title rewrites.

What this does not cover