I don't think I found what fixed this it might have been a magical combination of webpack and webpack cli / dev server version numbers because I was using Cypress component testing.

Also I moved all my commands using cypress-axe from support/command to support/index and everything seems to be working now.

Another tip is that if you update to the new webpack dev server, may be you are running tests with cypress component tests you probably want to set the webpack config with something like this:

  devServer: {
    host: "0.0.0.0",
    port: 8080,
    client: {
      overlay: {
        errors: true,
        warnings: false,
      },
    },

because these warns start to appear in an overlay which will cause your tests to fail.

I get this same warning with Webpack 4 and Cypress Component Testing. I wasn't encouraged much by the related thread on the axe-core repo but I can confirm the webpack config setting to disable overlay warnings works! Thanks for that. Here are the docs I used, if anyone else needs them: https://webpack.js.org/configuration/dev-server/#overlay

here is a PR that fixes this issue. #120

Seems this issue could be closed.