empty coverage for wasm on chrome

[konsultaner]

In my project https://github.com/konsultaner/connectanum-dart I use a vm and a wasm ci chain to produce a coverage output. I started to split up the two coverage outputs and found out that the wasm output is empty.

{"type":"CodeCoverage","coverage":[]}

I use this command to start the wasm coverage:

dart run test -p chrome --coverage=./coverage_chrome --timeout=5m --concurrency=1

I could reproduce it im my github action and locally on my linux system:

Dart SDK version: 3.5.2 (stable) (Wed Aug 28 10:01:20 2024 +0000) on "linux_x64"

Flutter 3.24.2 • channel stable • https://github.com/flutter/flutter.git
Framework • revision 4cf269e36d (vor 7 Tagen) • 2024-09-03 14:30:00 -0700
Engine • revision a6bd3f1de1
Tools • Dart 3.5.2 • DevTools 2.37.2

[konsultaner]

by the way, the test preformance difference without coverage and with coverage for wasm is 4x

[jakemac53]

I don't think this is expected to work at this time, I am not sure we have the information that would be needed to collect coverage on wasm. cc @kevmoo do you know who to ask about this?

dart run test -p chrome --coverage=./coverage_chrome --timeout=5m --concurrency=1

Are you separately configuring wasm as the compiler in your dart_test.yaml? I just want to make sure this is actually a wasm thing and not a dart2js thing, that looks like a dart2js test.

by the way, the test preformance difference without coverage and with coverage for wasm is 4x

It definitely is expected to take longer, how much longer I don't know, but essentially every piece of code that is executed has to be tracked etc, coverage is very expensive to collect.

[konsultaner]

platforms: [vm, chrome]
compilers: [dart2wasm]
on_platform:
  chrome: {timeout: 10x}

This really is a dart2wasm problem.

[jakemac53]

Open questions/issues:

• Does chrome support coverage for wasm? They didn't previously.
• Does dart2wasm produce source maps? cc @kevmoo who do we ask about this?
• We probably need support from source_map_stack_trace (Add support for stack traces containing wasm frames in package:source_map_stack_trace tools#586).
• How much work will it be to plumb all of those together? Hopefully we can borrow/share code with the JS coverage support.

[kevmoo]

We do produce source maps. Need to chat with @osa1 and @mkustermann for details.

[osa1]

Does chrome support coverage for wasm? They didn't previously.

I'm not sure. I've asked this to the relevant team in chat. I'll update. I've confirmed manually that Chrome currently generates coverage for Wasm. Correction: it seems like the answer is no, the coverage info I was seeing were all .js files. Chromium issue: https://g-issues.chromium.org/issues/354020953.

Does dart2wasm produce source maps?

Yes.

We probably need support from source_map_stack_trace

I'm working on implementing a Wasm stack trace mapper in test. If needed I'll update source_map_stack_trace as well.

How much work will it be to plumb all of those together? Hopefully we can borrow/share code with the JS coverage support

I don't know the details of test package, but I've been reading the dart2js code and it looks like it should be a few lines of changes after implementing the stack trace mapper. I'm working on it.

[konsultaner]

@osa1 so for now the best would be to get test coverage by dart2js or is it realistic to hope for a fix on the chrome side any soon?

[osa1]

@konsultaner I've reached out to the relevant team to get the Chromium issue triaged, but they seem to be away until the 18th. I will update when I hear back from them.

[osa1]

@konsultaner see https://g-issues.chromium.org/issues/354020953#comment4. It looks like Chrome won't be supporting coverage for Wasm code any time soon.

I briefly looked at coverage handling of this package and there doesn't seem to be anything specific to JS, it should handle Wasm coverage output as well. I've updated the stack_frame package in the meantime but it seems like it wasn't necessary for coverage support in dart test.

In any case, we can get back to this once Chrome starts generating coverage info of Wasm code. I suspect it will just work without any changes.

[konsultaner]

@osa1 thanks for your investigation. Seems like we have to wait for the feature. Thx 😇

[konsultaner]

To anyone reading this issue: I guess showing to the chrome team, that there is demand for wasm code coverage support, is the best way to let it happen. Go to:

https://issues.chromium.org/issues/354020953

sign in with your google account and hit the +1 on the top right corner, if you are interested into this feature.

[kevmoo]

To anyone reading this issue: I guess showing to the chrome team, that there is demand for wasm code coverage support, is the best way to let it happen. Go to:

https://issues.chromium.org/issues/354020953

sign in with your google account and hit the +1 on the top right corner, if you are interested into this feature.

See the guy who filed that issue? I wonder who [email protected] is 😉