Why a 95% pass rate isn't necessarily an accessible website
Most models pass 88% to 98% of our acceptance criteria across 22 common components. However, the failures that remain can create barriers for people with disability, and they add up quickly across a page.

We gave eight models the exact same prompts to build common web components, generating each of them three times with and without our skill loaded. Then we tested the generated components using our automated test suite, combined with a light human review. This article covers what we found.
What the results show
On the surface, the results seem impressive. Most models pass between 88% and 98% of our acceptance criteria:
- Claude Opus 5.5, Claude Fable 5.1, and GPT-6 Astra: 96% to 98%
- Grok 4.7, GPT-6.1 Sol, GPT-6 Luna, Claude Haiku 5.5, Claude Sonnet 5.5: 88% to 96%
However, lots of passing tests doesn’t mean the code is highly accessible. When we get audit results, we're used to just seeing what fails, not what passes. So we have little frame of reference for the percentage of pass to fail tests. This means what might look "good" here is not necessarily better or worse that what we see on real sites.
When we look at the failing tests for each component, we notice that models in the 88% to 96% range have one or two failing tests for each component. This adds up quickly when looking at a whole page and a whole site.
We often see this in audits: one bad component repeated across a page can have a serious impact on the accessibility of the page as a whole. If each component even has just one issue, the number of reported problems compounds.
Even the best models fail on average once every two components. On a page built from 20 components, that could mean around 10 issues before anyone looks at how the components work together.
Where the models fail
We group the acceptance criteria into five main categories. Without the skill:
- Visual design had the most failures. 12.5% of these criteria failed, and they're about 39% of all failures. Most are focus indicators and field borders without enough contrast. Models style focus rings but don't check them.
- Adaptive UI had the highest failure rate, 16.8%, but it has the fewest tests. Most failures are tabs, links, tooltips and tables that don't reflow at narrow widths.
- Semantic markup and keyboard each failed about 4% of criteria. Because they have the most tests, together they make up about half of all failures. Common problems are missing states, focus that isn't managed in menus, and widgets that don't follow expected keyboard patterns.
- Labels and messaging had no failures. Every model that we tested labelled form fields correctly.
What the skill changes
With the skill loaded, every model improved. Overall failures fell from 5.7% to 0.3% of criteria. Semantic markup and keyboard failures almost disappeared. Most of the failures that remained were visual design, mainly from GPT-6 Luna and Grok 4.7.
When the skill is worth the cost
Depending on the model, using the skill can add a considerable number of input tokens to each generation. As a result, the cost of each component goes up. Whether that's worth it depends on the model.
For cheaper models, it's an easy decision. GPT-6 Luna goes from 88.5% to 99% pass rate, that's the biggest uplift from our initial test set and still costs less than one cent per component. GPT-6.1 Sol goes from passing 96% of acceptance criteria up to 100% for just eight cents.
For more expensive models, it might be harder to justify. For example Claude Fable 5.1 meets about 96% of acceptance criteria without the skill. The skill takes it to 99.8% but triples the cost, to about $1.06 per component. Grok 4.7 improves by 6 points, but its cost goes from 12 cents to 78 cents per component, and each generation takes four times as long.
In both cases, a cheaper model with the skill matches or beats them for a fraction of the cost.
What the tests don’t cover
We know these results don't give the full picture yet, but they are a great proof of concept for the model.
We've done a detailed write up of our limitations, but here's two that we want to highlight.
- We started with common, well-documented components that have published guidance and many examples to learn from. Real websites often use more complex, creative designs, and that’s where we usually find issues in audits.
- Each test also builds one component from scratch as a single HTML file. Real projects add layers of complexity: frameworks, existing design systems and component libraries, each with its own conventions and its own accessibility problems. A model that writes an accessible component in isolation may not do the same inside a large codebase. Real-world results may be quite different.
What's next
Having started with tests that cover common components, we're now working on more complex tests, so results better reflect real-world performance. Real pages combine components, and that's where new problems can start to appear; focus moving between elements, duplicate landmarks and heading levels that don't reflect the content hierarchy.
We'll use the test results to improve the skill. If all of the models already get something right, the skill shouldn't need to include that information.
We'll also continue to test new models when they're released and publish the results here.
Need help with accessibility and AI?
AI is changing how digital products get built. We can help you work out what this means for the accessibility of your products and the experience for the people who use them.
