Support ESM when watch mode reruns files

[adamfuhrer]

Tell us about your runtime:

• QUnit version: 2.19.1
• Which environment are you using? (e.g., browser, Node): Node
• How are you running QUnit? (e.g., QUnit CLI, Grunt, Karma, manually in browser): Terminal

What are you trying to do?

Code that reproduces the problem:

Running qunit -w. On first run it successfully outputs the results of the tests, upon making a change to the test file, the error occurs:

What did you expect to happen?

Test should just re-run. Any help would be appreciated!

[adamfuhrer]

Also occurs with a files argument qunit --watch 'test/**/*.js':

[Krinkle]

@adamfuhrer Which Node.js version and OS do you use? (e.g. Node.js 14 on Debian Linux).

This appears to work for me with QUnit 2.19, so there's something a bit more subtle going on than the watch mode not working at all on subsequent runs. This is also covered by our CI jobs on Windows/Mac/Linux and on multiple Node versions.

Node.js 17 on macOS 10.15 (--watch, implied default test/ argument).

$ qunit --watch
TAP version 13
ok 1 First > 1
ok 2 Second > 1
1..2
# pass 2
# skip 0
# todo 0
# fail 0
File update: test/first.js
Restarting...
TAP version 13
ok 1 First > 100
ok 2 Second > 1
1..2
# pass 2
# skip 0
# todo 0
# fail 0
^CStopping QUnit...

Node.js 14 on Debian Linux (-w shortcut, and explicit argument).

nobody@c418f2e9a426:/example$ qunit -w test/
TAP version 13
ok 1 First > 100
ok 2 Second > 1
1..2
# pass 2
# skip 0
# todo 0
# fail 0
File update: test/first.js
Restarting...
TAP version 13
ok 1 First > 1
ok 2 Second > 1
1..2
# pass 2
# skip 0
# todo 0
# fail 0
^CStopping QUnit...

[adamfuhrer]

Appreciate the help. I'm on the LTS of Node 16.5.1. Also tried upgrading to the latest version of Node 18.3, and the issue still occurs unfortunately. I'm running macOS 12.3, so maybe that's where the issue is occuring

To give some more context I'm in a fresh project with qunit as the only devDependency and no plugins or config file overrides

[Krinkle]

@adamfuhrer Would you mind publishing your directory as a Git repo, a Gist, or attach as a ZIP file to a comment?

I don't make a habit of asking this, but I'm unable to reproduce it on the same Node and QUnit version and would like to rule out "everything else", before I suspect the macOS version difference as making an observable difference in how Node.js behaves.

Feel free to leave out node_modules, but do include package-lock.json.

[adamfuhrer]

Great, thanks so much! Attached the repo as a zip file here - core.zip

[Krinkle]

@adamfuhrer I've reproduced the issue and narrowed it down to the cause that "type": "module" is used in package.json, which means that upon re-running the tests and re-importing the same files from the test directory, Node.js is not actually executing them again and thus nothing happens.

It works fine when you project uses Node.js' default CJS/require mode, as the QUnit CLI clears Node.js' internal require cache between runs. However, Node.js does not yet offer a way for test runners to clear the ES6 module imports cache.

It looks like our friends at Mocha have run into the same limitation over at mochajs/mocha#4374.

[adamfuhrer]

Really appreciate the follow up, and looking into this! Good to know that there's a workaround available.

[Krinkle]

Eleventy has also run into the same issue. They worked around it by appending a random query string to the imported path. However, that still leaves the bug where any indirectly imported files remain re-used from the cache which means the watcher picks up changes in test files but not e.g. in your project source code.

https://www.zachleat.com/web/eleventy-v3-esm/ (from slide 67).

I think for QUnit 3.0, what I'll do is switch the QUnit CLI watch mode to spawn a child process and run the tests there, and then we can re-do that cleanly on each run, possibly spawning the "next" process eagerly after the last test (instead of before the first test) for faster response times during re-runs.