Manual accessibility testing vs automated: what each one actually catches

EAA Comply · Updated July 2026 · 9 min read

Every accessibility vendor will tell you their scan "checks your compliance." Almost none of them tell you what their scan cannot see. That gap matters because the European Accessibility Act doesn't care whether your dashboard shows a green score — it cares whether a disabled customer can actually complete a purchase on your site. Automated testing and manual testing catch genuinely different things, and understanding the split is the difference between a report that looks reassuring and one that's actually true.

What automated testing does

Automated accessibility tools work by scanning your rendered DOM, HTML, CSS and ARIA attributes against a library of machine-checkable rules. The most widely used engine is axe-core, Deque Systems' open-source rules engine — it's also the engine built into Chrome DevTools' own accessibility panel, and it powers most of the paid scanning tools on the market whether they admit it or not.

A rules engine like axe-core can reliably detect things like:

• Text and background colour combinations that fall below the 4.5:1 (or 3:1 for large text) contrast ratio.
• Images with a missing alt attribute entirely.
• Form inputs with no associated <label>, aria-label or aria-labelledby.
• Invalid or misused ARIA roles and attributes (a role="button" with no accessible name, an aria-hidden="true" on focusable content).
• Duplicate id values that break assistive-technology mapping.
• Malformed heading structure and missing page landmarks.

This is genuinely valuable work, and it's fast: a full scan runs in seconds to minutes per page, it's consistent (the same code always produces the same result), and it's cheap to run across hundreds of pages at once. That's exactly why it's the first pass in every audit we run.

What automated testing cannot catch

A machine can confirm that an alt attribute exists. It cannot confirm the attribute says something true. This is the core limitation, and it repeats across almost every category of manual-only testing:

• Meaningful vs. merely present alt text. alt="IMG_4021.jpg" and alt="blue cotton crew-neck t-shirt, front view" both pass an automated check. Only one is useful to a blind shopper.
• Logical keyboard focus order. A scanner can confirm elements are focusable. It cannot tell you that tabbing through your product gallery jumps from the price to the footer and back, skipping the "add to basket" button.
• Whether custom widgets are actually operable. Dropdowns, modals, carousels, tab panels, accordions — these are usually built from <div> soup with ARIA sprinkled on top. Automated tools check that the ARIA attributes are valid; they can't check that a modal actually traps focus correctly, that closing it returns focus to the trigger, or that a screen reader announces "expanded" and "collapsed" when it should.
• Whether error messages genuinely help. "Please check the form" passes every automated rule about labels being present. It's still useless to someone who can't see which field turned red.
• Reading order vs. visual order. CSS can make content look right while the underlying DOM order — what a screen reader actually reads aloud — is scrambled.
• Real CAPTCHA alternatives and drag-and-drop alternatives. A scanner can see that a CAPTCHA element exists; it can't tell you whether a non-visual, non-drag alternative actually works end to end.

Catching these requires a human tester doing a full keyboard-only pass through the page — unplug the mouse, use only Tab, Shift+Tab, Enter, Space and arrow keys — and, ideally, running the same journey with a real screen reader: NVDA or JAWS on Windows, VoiceOver on Mac and iOS. There is no rule engine that substitutes for actually listening to what gets announced.

Automated vs. manual, side by side

Why EAA Comply runs both

This is the actual shape of our Online shop audit, and it's why we don't sell a pure dashboard subscription. We run an axe-core automated scan across your site's pages to catch the ~57–80% of issues that are genuinely machine-detectable, cheaply and consistently. Then a human tester goes through your home page, a product page, the cart and the full checkout with a keyboard and a screen reader, because that's where the remaining defects concentrate and where they cost you the most — a customer who can't complete a purchase, or a plaintiff's lawyer who can.

Neither half replaces the other. A scan alone tells you your labels exist; it can't tell you your checkout is actually usable. Manual testing alone doesn't scale to auditing every page on a large catalogue. Combined, they cover what each one is actually good at.

Why not just buy a cheap automated-only scanner subscription

Because the number it shows you is true and misleading at the same time. A €20/month automated-only tool can genuinely, accurately report "92% compliant" while a keyboard user is completely stuck at your payment step — trapped in a modal, unable to reach the "place order" button, with no way out except closing the tab. The scanner isn't lying. It's just answering a narrower question than the one you're actually asking, which is "can a disabled customer buy from me?"

This is the same underlying failure as an accessibility overlay widget, even though the two look nothing like each other. An overlay never touches your code at all; a pure automated scanner reads your code but only understands the fraction of it that machines can judge. Both give you a reassuring number. Neither one verifies that a real person can actually use your site. If the goal is a dashboard, either will do. If the goal is EAA compliance and staying out of a courtroom, neither is sufficient on its own.

Not ready for a full audit? Start with our free automated scan to see the machine-detectable issues on your site today, then read our online shop accessibility checklist for what to prioritise, our plain-English guide to WCAG 2.1 AA for the standard behind all of this, and the European Accessibility Act guide for online shops for who the law applies to. Back to the EAA Comply homepage.

• Sources:
• Deque / axe-core project, discussion of real-world issue coverage percentages
• Deque, axe-core open-source accessibility testing engine
• W3C Web Accessibility Initiative, "Evaluating Web Accessibility"