Intopia AI Accessibility Benchmark
How accessible is AI-generated code?
AI is increasingly used to build products, but it's hard to know how accessible the code it generates really is.
The Intopia AI Accessibility Benchmark is a repeatable way to measure how accessible AI-generated code is. We test each model with and without an accessibility skill, using advanced automated and browser tests as well as AI-judgement tests, with a light human review.
This is the proof of concept for the benchmark, testing the simplest components. We are working on more complex tests so results better reflect real-world performance.

What we've learned
Where accessibility failures occur
Across the models we test, accessibility failures tend to cluster in recurring areas.
Specialist guidance improves the results
Every model in our latest test improved with the accessibility skill loaded, although the size of the improvement varied.
Accessibility doesn’t have to be expensive
With our accessibility skill, a cheaper model reaches the same level of accessibility as the more expensive frontier model but at a fraction of the cost.
Hold up
What these numbers don’t mean
Before we get too carried away, a few caveats.
Read the full list in the Limitations section of Test Methodology.
100% here doesn’t mean WCAG conformant
We’re testing components, not complete pages. WCAG conformance is assessed at page level, so even a perfect score in this benchmark isn’t a conformance claim.
Passing 99% of the tests doesn’t mean ‘99% accessible’
A component can pass most of our test criteria and still have accessibility issues. A real page combines dozens of components, so even one issue per component can add up fast. Read why a 95% pass rate isn't necessarily an accessible website.
The costs are estimates
Costs are calculated from token counts using each provider’s published per-token prices.
Baseline vs With Skill
Cost in USD
Here are some summary stats from our most recent run.
Biggest uplift with skill
GPT-6 Luna
up 10.5points uplift
- Baseline
- 88.5%
- With Skill
- 99%
of acceptance criteria met
- Cost per gen
- before: under $0.01, after: $0.01
- Tokens in
- before: 3k, after: 94k
- Tokens out
- before: 2k, after: 5.6k
Best result with skill
GPT-6.1 Sol
up 4.1points uplift
- Baseline
- 95.9%
- With Skill
- 100%
of acceptance criteria met
- Cost per gen
- before: $0.02, after: $0.08
- Tokens in
- before: 2k, after: 94k
- Tokens out
- before: 1.7k, after: 3.3k
Best result without skill
Claude Opus 5.5
up 2.3points uplift
- Baseline
- 97.6%
- With Skill
- 99.9%
of acceptance criteria met
- Cost per gen
- before: $0.12, after: $0.36
- Tokens in
- before: 7.7k, after: 126k
- Tokens out
- before: 4.6k, after: 10k
Where issues are found
This chart shows how often each model failed the acceptance criteria in each category, without the skill and with the skill loaded. Each cell shows the percentage of acceptance criteria that failed and the counts behind it.
Darker cells have a higher failure rate, and both panels use the same colour scale (0% to 50% or more).
| Adaptive UI | Visual design | Semantic markup | Keyboard | Labels and messaging | |
|---|---|---|---|---|---|
| GPT-6 Luna | 19%, 8 of 42 failed | 27%, 59 of 222 failed | 8%, 38 of 501 failed | 9%, 35 of 384 failed | 0%, 0 of 78 failed |
| GPT-6.1 Terra | 5%, 2 of 42 failed | 17%, 37 of 222 failed | 3%, 14 of 501 failed | 4%, 14 of 384 failed | 0%, 0 of 78 failed |
| GPT-6.1 Sol | 12%, 5 of 42 failed | 11%, 24 of 222 failed | 2%, 10 of 501 failed | 3%, 12 of 384 failed | 0%, 0 of 78 failed |
| GPT-6 Astra | 14%, 6 of 42 failed | 4%, 9 of 222 failed | 3%, 13 of 501 failed | 3%, 10 of 384 failed | 0%, 0 of 78 failed |
| Claude Haiku 5.5 | 21%, 9 of 42 failed | 20%, 44 of 222 failed | 6%, 30 of 501 failed | 8%, 31 of 384 failed | 0%, 0 of 78 failed |
| Claude Sonnet 5.5 | 12%, 5 of 42 failed | 9%, 20 of 222 failed | 3%, 14 of 501 failed | 3%, 10 of 384 failed | 0%, 0 of 78 failed |
| Claude Opus 5.5 | 12%, 5 of 42 failed | 4%, 9 of 222 failed | 1%, 7 of 501 failed | 2%, 8 of 384 failed | 0%, 0 of 78 failed |
| Grok 4.7 | 12%, 5 of 42 failed | 15%, 33 of 222 failed | 7%, 33 of 501 failed | 4%, 14 of 384 failed | 0%, 0 of 78 failed |
| Gemini 4 Argon | 21%, 9 of 42 failed | 9%, 19 of 222 failed | 1%, 7 of 501 failed | 3%, 12 of 384 failed | 0%, 0 of 78 failed |
| Claude Fable 5.1 | 31%, 13 of 42 failed | 8%, 17 of 222 failed | 2%, 8 of 501 failed | 1%, 5 of 384 failed | 0%, 0 of 78 failed |
| Adaptive UI | Visual design | Semantic markup | Keyboard | Labels and messaging | |
|---|---|---|---|---|---|
| GPT-6 Luna | 5%, 2 of 42 failed | 3%, 7 of 222 failed | 0.2%, 1 of 501 failed | 0.5%, 2 of 384 failed | 0%, 0 of 78 failed |
| GPT-6.1 Terra | 0%, 0 of 42 failed | 0%, 0 of 222 failed | 0.6%, 3 of 501 failed | 0%, 0 of 384 failed | 0%, 0 of 78 failed |
| GPT-6.1 Sol | 0%, 0 of 42 failed | 0%, 0 of 222 failed | 0%, 0 of 501 failed | 0%, 0 of 384 failed | 0%, 0 of 78 failed |
| GPT-6 Astra | 0%, 0 of 42 failed | 0%, 0 of 222 failed | 0%, 0 of 501 failed | 0%, 0 of 384 failed | 0%, 0 of 78 failed |
| Claude Haiku 5.5 | 0%, 0 of 42 failed | 0.5%, 1 of 222 failed | 0.6%, 3 of 501 failed | 0%, 0 of 384 failed | 0%, 0 of 78 failed |
| Claude Sonnet 5.5 | 0%, 0 of 42 failed | 0%, 0 of 222 failed | 0.2%, 1 of 501 failed | 0%, 0 of 384 failed | 0%, 0 of 78 failed |
| Claude Opus 5.5 | 0%, 0 of 42 failed | 0%, 0 of 222 failed | 0%, 0 of 501 failed | 0.3%, 1 of 384 failed | 0%, 0 of 78 failed |
| Grok 4.7 | 7%, 3 of 42 failed | 2%, 5 of 222 failed | 0%, 0 of 501 failed | 0%, 0 of 384 failed | 1%, 1 of 78 failed |
| Gemini 4 Argon | 0%, 0 of 42 failed | 0%, 0 of 222 failed | 0%, 0 of 501 failed | 0%, 0 of 384 failed | 0%, 0 of 78 failed |
| Claude Fable 5.1 | 0%, 0 of 42 failed | 0.9%, 2 of 222 failed | 0%, 0 of 501 failed | 0%, 0 of 384 failed | 0%, 0 of 78 failed |
0%50% or more of criteria failed
Acceptance criteria met per run
This chart compares each model without the skill and with the skill loaded. It shows the percentage of acceptance criteria the model met across all its generated components. The axis starts at 80%. Further right is better.
- Baseline
- With Skill
Cost and accessibility performance
This chart compares each model with the skill loaded. It shows how many acceptance criteria the model failed across all its generated components, and its average cost to generate one component. Each criterion counts once, however many of its 3 generations failed. Models near the bottom left are the cheapest and have the fewest failures.
- OpenAI
- Anthropic
- xAI
Number of unique failed acceptance criteria
Results Summary
GPT models lead with the skill. Opus leads without it.
With the skill on, GPT-6.1 Sol and GPT-6 Astra met all of our acceptance criteria. GPT-6 Luna is the cheapest option: it meets 99% for about one cent per generated component. GPT-6.1 Sol meets 100% for about eight cents.
Without the skill, Claude Opus 5.5 scores highest. With the skill, it meets 99.9%, but the cost rises from 12 cents to 36 cents per generated component.
How the testing works
The suite has over 20 interactive component cases, from a plain button or text field up to a combobox, carousel, modal or tabs. Each component is checked against our acceptance criteria from WCAG 2.2, the ARIA Authoring Practices Guide and documented accessibility best practice.
We generate each component three times with the skill and three times without, then check each one with automated tests and human review. Every run records cost, turns, input and output tokens, and speed.
We make the skill used in these comparisons
Our skill gives a model instructions and reference guides to use while it writes code. The guides include acceptance criteria for common components and show the model how to build components that meet WCAG 2.2 Level AA. The benchmark tests against the same acceptance criteria.
Read the full geeky detail
Test methodologySign up for updates
Join our AI Accessibility Benchmarks mailing list for updates on the project and skill.
Need help with accessibility and AI?
AI is changing how digital products get built. We can help you work out what this means for the accessibility of your products and the experience for the people who use them.

