Automated accessibility tools catch roughly 30-40% of WCAG issues on a typical page. In a 2018 UK Government Digital Service study, testers deliberately planted 142 documented barriers into a page and ran 13 automated tools against it: the best tool found 40%, WAVE found 30%, axe found 29%. Every tool missed reading order, focus indicators, and context-dependent color entirely.
Key takeaways
- The ceiling is structural, not a tooling gap that better software fixes. Only about 25-30 of WCAG 2.1's 78 success criteria can be mechanically evaluated at all — the rest need human judgment.
- Keyboard navigation failures are the biggest blind spot. Whether someone using only a keyboard can reach and operate every control came in close to zero across all 13 tools tested.
- A 100% automated score is not a passing accessibility audit. It means zero violations of the third of WCAG that's checkable by rule, and says nothing about the rest.
- Two tools together still land around 50% coverage — GDS's own guidance after the study, not full coverage from doubling up.
- Automated tools are still worth running — they catch real, common structural defects fast and consistently. The mistake is treating a clean report as the finish line.
The study behind the 40% number
In 2018, the UK Government Digital Service took a real page of content and deliberately introduced 142 documented accessibility barriers into it — missing labels, bad heading structure, insufficient contrast, unreachable controls, the kind of thing that actually ships. They then ran 13 automated accessibility testing tools against that page and counted, on a strict pass/fail basis, how many of the 142 known problems each tool flagged.
The best performer, SortSite, caught 40%. WAVE caught 30%. axe — one of the most widely used tools in CI pipelines today — caught 29%. Every single tool missed reading order problems, the absence of visible focus indicators, and color used as the only signal for meaning. None of the 13 reliably caught keyboard-navigation or focus-order failures, which is close to the whole ballgame for anyone who can't use a mouse.
| Claim | Status | Basis |
|---|---|---|
| Automated tools catch missing alt text, low contrast, missing form labels | Documented | Core function of rule-based checkers; consistent across tools |
| Best automated tool caught 40% of planted issues in the 2018 GDS study | Documented | Published GDS research, specific and dated |
| Keyboard navigation failures are the hardest for tools to catch | Documented | Same study; near-zero detection across all 13 tools tested |
| Roughly 25-30 of 78 WCAG 2.1 criteria are mechanically checkable at all | Inferred | Consistent across multiple accessibility-testing sources; not a single cited count |
| Modern AI-assisted checkers have substantially closed the gap since 2018 | Unsupported | No independently replicated study at this scale since; vendor claims aren't evidence |
Why the gap is structural, not a bug in the tools
WCAG's success criteria split roughly into two kinds: things a program can verify against the DOM (does this image have an alt attribute, is the contrast ratio above 4.5:1) and things that require understanding what the page means (is this alt text actually descriptive, does this reading order make sense, is this focus indicator visible enough to actually see). Automated tools are built for the first kind. The second kind is not a detection problem software gets better at — it requires judgment a rule can't encode.
What tools reliably catch
- Missing or empty alt attributes on images
- Form inputs with no associated label
- Color contrast ratios below WCAG thresholds
- Missing document language, missing page title
- ARIA attributes used with invalid values
What tools reliably miss
- Reading order that doesn't match visual order for screen reader users
- Focus indicators that technically exist but aren't visible enough to use
- Color as the only way information is conveyed, in context
- Whether alt text actually describes the image, versus just being present
- Keyboard traps and broken focus order through interactive components
What actually closes the gap
GDS's own recommendation after the study: run two or three automated tools in combination, which lands around 50% coverage — better than any single tool, still short of half the picture on its own. The rest is layered, not automated:
- Automated tools in CI, on every build. Fast, consistent, catches real structural defects before they ship — the right job for the ~30-40% they're good at.
- Actual keyboard-only testing. Unplug the mouse and try to complete the core task on the page. This alone catches the category every automated tool missed almost entirely.
- Screen reader spot checks on the pages that matter most — checkout flows, forms, primary navigation — not the whole site, but not zero either.
- Manual review by someone who knows WCAG, on a schedule, not a one-time audit before launch that never gets revisited as the page changes.