[Background addon] a11y tests running after background is set

[ilucaslo]

Summary

My stack is Storybook 8.2.9 with Vue 3 TypeScript.

I have configured in my preview file background values for dark and light. For my component story i set the default background to dark since the component is designed to be used on dark backgrounds. I have accessibility addon configured. The issue is that the first time i select a story with the default dark background i get a false positive in the accessibility tests, this is a color contrast issue. I suspect the a11y tests are running before the background is load because if i rerun the tests manually by clicking the 'Tests completed' message the test pass. Also if i navigate to other stories and come back to the dark background, the test pass so i suspect the first time the background take some time to load. I also tried with the new globals api but i get same result.

storybook.mov

It also returns the same errors not only in the storybook UI but when i run test runner(v0.19.1) in the terminal.

Is this background loading time expected or there is a workaround to avoid the false positives? I would like to avoid the false positive specially in test runner since is used in the pipeline. I found a workaround for this setting some delay in the test runner config but is not ideal since is not scalable.

test-runner.ts

const config: TestRunnerConfig = {
  tags: {
    exclude: ['no-tests']
  },
  async preVisit(page) {
    await injectAxe(page)
  },
  async postVisit(page) {
    /*
      Workaround for an issue when the background is set
      after the a11y test runs resulting in false positives
    */
    await new Promise(resolve => setTimeout(resolve, 500))
    await checkA11y(page, '#storybook-root', {
      detailedReport: true,
      detailedReportOptions: {
        html: true
      }
    })
  }
}
 
export default config

Additional information

No response

Create a reproduction

https://stackblitz.com/edit/github-9hbveetc?file=.storybook%2Fmain.ts

[valentinpalkovic]

Hi @ilucaslo

Thank you for reaching out!

At Storybook, we are currently working on a revamp of @storybook/addon-a11y, which also affects when a particular axe run is started. Could you please try out the latest beta and give us feedback on whether it fixes the timing issue for you?

npx [email protected] upgrade

However, this will not affect when the test-runner executes an axe run.