Skip to main content

Intopia AI Accessibility Benchmark

How accessible is AI-generated code?

AI is increasingly used to build products, but it's hard to know how accessible the code it generates really is.

The Intopia AI Accessibility Benchmark is a repeatable way to measure how accessible AI-generated code is. We test each model with and without an accessibility skill, using advanced automated and browser tests as well as AI-judgement tests, with a light human review.

This is the proof of concept for the benchmark, testing the simplest components. We are working on more complex tests so results better reflect real-world performance.

What we've learned

Where accessibility failures occur

Across the models we test, accessibility failures tend to cluster in recurring areas.

Specialist guidance improves the results

Every model in our latest test improved with the accessibility skill loaded, although the size of the improvement varied.

Accessibility doesn’t have to be expensive

With our accessibility skill, a cheaper model reaches the same level of accessibility as the more expensive frontier model but at a fraction of the cost.

Hold up

What these numbers don’t mean

Before we get too carried away, a few caveats.

  • 100% here doesn’t mean WCAG conformant

    We’re testing components, not complete pages. WCAG conformance is assessed at page level, so even a perfect score in this benchmark isn’t a conformance claim.

  • Passing 99% of the tests doesn’t mean ‘99% accessible’

    A component can pass most of our test criteria and still have accessibility issues. A real page combines dozens of components, so even one issue per component can add up fast. Read why a 95% pass rate isn't necessarily an accessible website.

  • The costs are estimates

    Costs are calculated from token counts using each provider’s published per-token prices.

Baseline vs With Skill

Cost in USD

Here are some summary stats from our most recent run.

  • Biggest uplift with skill

    GPT-6 Luna

    up 10.5points uplift

    Baseline
    88.5%
    With Skill
    99%

    of acceptance criteria met

    Cost per gen
    before: under $0.01, after: $0.01
    Tokens in
    before: 3k, after: 94k
    Tokens out
    before: 2k, after: 5.6k
  • Best result with skill

    GPT-6.1 Sol

    up 4.1points uplift

    Baseline
    95.9%
    With Skill
    100%

    of acceptance criteria met

    Cost per gen
    before: $0.02, after: $0.08
    Tokens in
    before: 2k, after: 94k
    Tokens out
    before: 1.7k, after: 3.3k
  • Best result without skill

    Claude Opus 5.5

    up 2.3points uplift

    Baseline
    97.6%
    With Skill
    99.9%

    of acceptance criteria met

    Cost per gen
    before: $0.12, after: $0.36
    Tokens in
    before: 7.7k, after: 126k
    Tokens out
    before: 4.6k, after: 10k

Where issues are found

This chart shows how often each model failed the acceptance criteria in each category, without the skill and with the skill loaded. Each cell shows the percentage of acceptance criteria that failed and the counts behind it.

Darker cells have a higher failure rate, and both panels use the same colour scale (0% to 50% or more).

Baseline
Adaptive UIVisual designSemantic markupKeyboardLabels and messaging
GPT-6 Luna19%, 8 of 42 failed27%, 59 of 222 failed8%, 38 of 501 failed9%, 35 of 384 failed0%, 0 of 78 failed
GPT-6.1 Terra5%, 2 of 42 failed17%, 37 of 222 failed3%, 14 of 501 failed4%, 14 of 384 failed0%, 0 of 78 failed
GPT-6.1 Sol12%, 5 of 42 failed11%, 24 of 222 failed2%, 10 of 501 failed3%, 12 of 384 failed0%, 0 of 78 failed
GPT-6 Astra14%, 6 of 42 failed4%, 9 of 222 failed3%, 13 of 501 failed3%, 10 of 384 failed0%, 0 of 78 failed
Claude Haiku 5.521%, 9 of 42 failed20%, 44 of 222 failed6%, 30 of 501 failed8%, 31 of 384 failed0%, 0 of 78 failed
Claude Sonnet 5.512%, 5 of 42 failed9%, 20 of 222 failed3%, 14 of 501 failed3%, 10 of 384 failed0%, 0 of 78 failed
Claude Opus 5.512%, 5 of 42 failed4%, 9 of 222 failed1%, 7 of 501 failed2%, 8 of 384 failed0%, 0 of 78 failed
Grok 4.712%, 5 of 42 failed15%, 33 of 222 failed7%, 33 of 501 failed4%, 14 of 384 failed0%, 0 of 78 failed
Gemini 4 Argon21%, 9 of 42 failed9%, 19 of 222 failed1%, 7 of 501 failed3%, 12 of 384 failed0%, 0 of 78 failed
Claude Fable 5.131%, 13 of 42 failed8%, 17 of 222 failed2%, 8 of 501 failed1%, 5 of 384 failed0%, 0 of 78 failed
With skill
Adaptive UIVisual designSemantic markupKeyboardLabels and messaging
GPT-6 Luna5%, 2 of 42 failed3%, 7 of 222 failed0.2%, 1 of 501 failed0.5%, 2 of 384 failed0%, 0 of 78 failed
GPT-6.1 Terra0%, 0 of 42 failed0%, 0 of 222 failed0.6%, 3 of 501 failed0%, 0 of 384 failed0%, 0 of 78 failed
GPT-6.1 Sol0%, 0 of 42 failed0%, 0 of 222 failed0%, 0 of 501 failed0%, 0 of 384 failed0%, 0 of 78 failed
GPT-6 Astra0%, 0 of 42 failed0%, 0 of 222 failed0%, 0 of 501 failed0%, 0 of 384 failed0%, 0 of 78 failed
Claude Haiku 5.50%, 0 of 42 failed0.5%, 1 of 222 failed0.6%, 3 of 501 failed0%, 0 of 384 failed0%, 0 of 78 failed
Claude Sonnet 5.50%, 0 of 42 failed0%, 0 of 222 failed0.2%, 1 of 501 failed0%, 0 of 384 failed0%, 0 of 78 failed
Claude Opus 5.50%, 0 of 42 failed0%, 0 of 222 failed0%, 0 of 501 failed0.3%, 1 of 384 failed0%, 0 of 78 failed
Grok 4.77%, 3 of 42 failed2%, 5 of 222 failed0%, 0 of 501 failed0%, 0 of 384 failed1%, 1 of 78 failed
Gemini 4 Argon0%, 0 of 42 failed0%, 0 of 222 failed0%, 0 of 501 failed0%, 0 of 384 failed0%, 0 of 78 failed
Claude Fable 5.10%, 0 of 42 failed0.9%, 2 of 222 failed0%, 0 of 501 failed0%, 0 of 384 failed0%, 0 of 78 failed

0%50% or more of criteria failed

Acceptance criteria met per run

This chart compares each model without the skill and with the skill loaded. It shows the percentage of acceptance criteria the model met across all its generated components. The axis starts at 80%. Further right is better.

  • Baseline
  • With Skill

Cost and accessibility performance

This chart compares each model with the skill loaded. It shows how many acceptance criteria the model failed across all its generated components, and its average cost to generate one component. Each criterion counts once, however many of its 3 generations failed. Models near the bottom left are the cheapest and have the fewest failures.

  • OpenAI
  • Anthropic
  • Google
  • xAI

Number of unique failed acceptance criteria

Results Summary

GPT models lead with the skill. Opus leads without it.

With the skill on, GPT-6.1 Sol and GPT-6 Astra met all of our acceptance criteria. GPT-6 Luna is the cheapest option: it meets 99% for about one cent per generated component. GPT-6.1 Sol meets 100% for about eight cents.

Without the skill, Claude Opus 5.5 scores highest. With the skill, it meets 99.9%, but the cost rises from 12 cents to 36 cents per generated component.

How the testing works

The suite has over 20 interactive component cases, from a plain button or text field up to a combobox, carousel, modal or tabs. Each component is checked against our acceptance criteria from WCAG 2.2, the ARIA Authoring Practices Guide and documented accessibility best practice.

We generate each component three times with the skill and three times without, then check each one with automated tests and human review. Every run records cost, turns, input and output tokens, and speed.

We make the skill used in these comparisons

Our skill gives a model instructions and reference guides to use while it writes code. The guides include acceptance criteria for common components and show the model how to build components that meet WCAG 2.2 Level AA. The benchmark tests against the same acceptance criteria.

Read the full geeky detail

Test methodology

Sign up for updates

Join our AI Accessibility Benchmarks mailing list for updates on the project and skill.

Where did you first learn about the AI Benchmark? (optional)

Need help with accessibility and AI?

AI is changing how digital products get built. We can help you work out what this means for the accessibility of your products and the experience for the people who use them.