About the AI Accessibility Benchmark
With AI models writing more and more front-end code, we wanted to know how accessible that code is, and where it goes wrong. We built this site to share what we find as we test each model. It shows which of our acceptance criteria each model passes and fails when it builds common components.
In 2026 we ran two long rounds of manual testing while we developed the Intopia web accessibility skill and the Figma Make skill. The testing was valuable as it showed us clearly how well models handle accessibility, but it took a long time to repeat for each model and new models are released too often for that to be sustainable on an ongoing basis.
We wanted a repeatable set of tests that let us compare models quickly and fairly, and track whether their output changes over time.
To keep testing models as they're released, we've built an automated test suite. Combined with light human review, it lets us check specific outputs from selected prompts, benchmark each new model on release, and measure the effect of changes to our skills.
Why components first
Components are the basic parts of every interface. If a model gets these wrong, everything built from them inherits the problem.
Our tests prompt the model to build common components, generating each one three times, because AI-generated code varies from run to run.
We're also actively developing template tests. These combine components into common page types, such as contact forms with validation, product pages with custom selectors, and web apps with complex features such as drag and drop. As the models continue to improve our tests need to challenge them and show where they break.
Acceptance criteria based on experience and best practice
For each component, we have written acceptance criteria that cover the most common accessibility issues we see in our work as accessibility professionals, plus the best practice techniques we recommend to clients.
Can you reach it with a keyboard? Does it have an accessible name? Does it tell screen reader users what state it's in? Does focus go to the right place?
We check each model's output against this criteria. At three generations per component, that's more than 1000 checks per model! A person reviews every result before we publish it.
Detailed test results help us develop the skill
Without tests, it's hard to know how well a skill helps a model. We've seen accessibility improve when we include guidance, but it hasn't been clear which information the model needs, or how much the guidance adds to the cost.
From now on, we’ll benchmark each model’s baseline to find where it fails, then give the skill the information it needs to fill those gaps. This keeps the skill lean and avoids unnecessary cost.
What's next
We'll test new models as they're released and publish what we find here. We'll keep developing our skills based on the results. We'll also keep adding components, and release the template tests when they're ready.
