Skip to content

Latest commit

 

History

History
244 lines (196 loc) · 16.9 KB

CG-10-17.md

File metadata and controls

244 lines (196 loc) · 16.9 KB
Raw

WebAssembly logo

Agenda for the October 17th video call of WebAssembly's Community Group

  • Host: Google Hangouts
  • Dates: Tuesday October 17th, 2017
  • Times: 9:00am–10:00am Pacific Time
  • Location: same Google Hangouts link as before
  • Contact:

Registration

None required if you've attended before. Email JF Bastien to sign up if it's your first time. The meeting is open to CG members only.

Logistics

The meeting will be a Google Hangouts call.

Agenda items

  1. Opening, welcome and roll call
    1. Opening of the meeting
    2. Introduction of attendees
  2. Find volunteers for note taking (acting chair to volunteer)
  3. Adoption of the agenda
  4. Proposals and discussions
    1. Versioning of external standard dependencies (Andreas Rossberg).
    2. Create new github.com/webassembly/workloads repo?
      1. We're all probably over-specializing on Tanks just because it's the easiest workload to run a quick experiment on.
      2. It seems useful to provide a common place to collect large test workloads that we keep up-to-date:
        1. Should test realistic paths through real codebases
        2. Load-time and throughput measurements
        3. Include source files if possible and link to author
        4. Updated as toolchain or codebase is updated, so always on the recommended path
        5. Primarily for engine and toolchain engineers to test and guide optimizations
        6. Non-goal: official "wasm benchmark" with overall "score"
      3. Two initial workloads to seed the repo:
        1. Unity: https://files.unity3d.com/marcot/benchmarks/
        2. Godot: http://godot.eska.me/pub/wasm-benchmark/
      4. Polls:
        1. POLL: create 'workload' repo with abovementioned goals?
        2. POLL: seed with above two workloads?
    3. Web platform test repository, discussion, and poll (Ben Titzer).
      1. Background: The Web Platform Tests repository is W3C-coordinated attempt to build a cross-browser testsuite for the Web-platform stack. It incorporates tests from many sources and offers, for example, a two-way sync feature between the Chromium repository and WPT in an effort to make it much easier to expand the WPT tests. The WPT effort would like to incorporate as many WASM tests as possible and run them in a cross-browser fashion, and make it easy to add tests. The issue is complicated by a build step that is necessary, since WASM spec tests are almost all written using the text format, which requires translation to binaries and JS files for running in engines that have a JS embedding.
      2. Discussion of proposals
        1. Preserve status quo, add additional mirroring in WPT of WASM spec repository
          1. WASM spec repo continues to be source of truth
          2. 1-way sync WASM spec -> WPT (automated)
          3. Enforce no-editing in WPT mirror of subdirectory
          4. Wasm participants can land spec tests. PRs welcome from browser engines.
        2. Change status quo, spec repo continues to be the source of truth
          1. 2-way sync of spec repo into stand-alone engine repos V8/Chakra, and browser engines Firefox/JSC.
          2. Running Ocaml conversion etc. requires a pre-submit step done by committer, with output included in the commit.
          3. V8/Chakra/SpiderMonkey/JSC contributors can land spec tests (in their home project repos, or in the spec repo directly)
        3. Change status quo, WPT becomes the source of truth
          1. Build a 2-way sync between V8 + Chakra for a slice of WPT (reuse existing mirror for others)
          2. V8/Chromium/Chakra/SpiderMonkey/JSC contributors can land spec tests (in their home project repos, or in WPT directly)
          3. WASM spec changes must include a spec branch/PR and a WPT branch/PR
        4. Detailed (v8 specific) doc
      3. Poll on proposals
        1. POLL: Tests contributed to the WASM spec repo should receive review from core contributors.
        2. POLL: WASM should maintain the source of truth by having compliance tests in the spec repo, as today.
  5. Closure

Agenda items for future meetings

None

Schedule constraints

None

Dates and locations of future meetings

Dates Location Host
2017-11-01 to 2017-11-02 Santa Clara, CA Intel
2017-11-06 to 2017-11-07 Burlingame, CA TPAC

Meeting notes

Roll Call

  • Arun Purushan
  • Ben L. Titzer
  • Benjamin Bouvier
  • Bradley Nelson
  • Dan Ehrenberg
  • Deepti Gandluri
  • Derek Schuff
  • Dru Knox
  • Eric Holk
  • Heejin Ahn
  • JF Bastien
  • Jacob Gravelle
  • Keith Miller
  • Kyle Buzby
  • Limin Zhu
  • Luke Wagner
  • Malcolm White
  • Mark Miller
  • Michael Ferris
  • Michael Hablich
  • Michael Holman
  • Paolo Serverini
  • Patrick Hickey
  • Peter Jensen
  • Sergey Rubanov
  • Tyler McMullen
  • Winterton Richard
  • Yury Delendik

Opening

Adoption of the agenda

Seconded by Brad.

Proposals and discussions

Versioning of external standard dependencies

Andreas Rossberg presenting

Issue

We depend on IEEE 754 and Unicode. How do we reference external standards that we depend on, and which have versions? For both we don’t really rely on what has changed or is likely to change in the future.

  1. Without a specific version, always applying the latest.
  2. Pinned to an exact version.
  3. Pinned to a minimum version.

JavaScript recently changed from minimum to latest version of Unicode. AWB told Andreas some history, see issue for details. It’s easier than updating Unicode every year for new emojis. There’s a feeling that referencing a specific version means it should be up to date. There’s a recommendation by Unicode and ISO that recommends no version, but AWB didn’t provide the reference. Andreas found a reference in Unicode, but it seems different and they seem to say that for a case like ours we want the minimum version approach. But it’s also very vague.

For Unicode we only depend on the range of code points (21 bits plus the gap), and UTF-8. For the text format we depend on the ASCII subset of Unicode. Andreas thinks 21 bit and UTF-8 as added in Unicode version 3 as far as he can tell, maybe that should be our minimum. Specifically, if new emojis come in or any new codepoint gets a meaning is totally irrelevant for WebAssembly (other than for the text format). Dan submitted pretty substantial tests for Unicode and UTF-8 to the spec test suite.

IEEE 754 sees less churn. Current version is 2008. Andreas doesn’t think that we depend on anything new from 2008. Dan points out that a new one might come in 2018, Andreas doesn’t think we’ll need anything from it. Dan had mentioned some changes in minnum / maxnum, had we adopted them that would require more thought from us.

The group doesn’t think that breaking changes for WebAssembly are likely in either standard (Andreas / Ben and others).

Consensus to float the Unicode version to latest, and pin IEEE 754-2008.

We’ll revisit when IEEE 754-2018 comes out.

New github.com/webassembly/workloads repo

Luke Wagner presenting

   1. We're all probably over-specializing on Tanks just because it's the easiest workload to run a quick experiment on.
   1. It seems useful to provide a common place to collect large test workloads that we keep up-to-date:
       1. Should test realistic paths through real codebases
       1. Load-time and throughput measurements
       1. Include source files if possible and link to author
       1. Updated as toolchain or codebase is updated, so always on the recommended path
       1. Primarily for engine and toolchain engineers to test and guide optimizations
       1. Non-goal: official "wasm benchmark" with overall "score"
   1. Two initial workloads to seed the repo:
       1. Unity: https://files.unity3d.com/marcot/benchmarks/
       1. Godot: http://godot.eska.me/pub/wasm-benchmark/
   1. Polls:
       1. POLL: create 'workload' repo with abovementioned goals?
       1. POLL: seed with above two workloads?
  • JF played with Unity benchmarks. Very modal, high variance, and measures WebGL 2 with fallback to WebGL 1. Luke says Jukka has a stripped down version, agrees we’d want to not draw.
  • JF asks if we want all of these to be runnable without a browser, in say the reference interpreter. Is that a goal, or do we rely on non-browser people to contribute such measurements? Luke thinks this would take much work, and might reduce the thing we’re measuring.
  • Titzer asks if we’re trying to attract workloads. Open the floodgates. If they’re not relevant or too say WebGL then we’d remove.
  • Derek: should we require source? Once we change toolchains, have threads, etc, we’ll want to reflect the new compilation patterns. I want to use it to benchmark my toolchains too.
  • Titzer: we can even check in older toolchain-compiled versions.
  • Derek: we can also allow some binary things that are critically important.
  • Brad: should we care about licenses?
  • Luke: we should ask for sufficiently permissive license. Sub-directories, separate licenses, readme to build and point to author, etc.
  • JF: any non-browser people interested in measuring or uploading workloads?
  • Pat (Fastly): doing non-browser, maybe 6 months from being able to contribute workloads.

Unanimous consent for Luke to create repo, create PRs for README, Godot, and Unity-lite

Web platform test repository, discussion, and poll

Ben Titzer presenting

   1. Background: The [Web Platform Tests repository](https://github.com/w3c/web-platform-tests) is W3C-coordinated attempt to build a cross-browser testsuite for the Web-platform stack. It incorporates tests from many sources and offers, for example, a two-way sync feature between the Chromium repository and WPT in an effort to make it much easier to expand the WPT tests. The WPT effort would like to incorporate as many WASM tests as possible and run them in a cross-browser fashion, and make it easy to add tests.
   The issue is complicated by a build step that is necessary, since WASM spec tests are almost all written using the text format, which requires translation to binaries and JS files for running in engines that have a JS embedding.
   1. Discussion of proposals
       1. Preserve status quo, add additional mirroring in WPT of WASM spec repository
          1. WASM spec repo continues to be source of truth
          1. 1-way sync WASM spec -> WPT (automated)
          1. Enforce no-editing in WPT mirror of subdirectory
          1. Wasm participants can land spec tests. PRs welcome from browser engines.
       1. Change status quo, spec repo continues to be the source of truth
          1. 2-way sync of spec repo into stand-alone engine repos V8/Chakra, and browser engines Firefox/JSC.
          1. Running Ocaml conversion etc. requires a pre-submit step done by committer, with output included in the commit.
          1. V8/Chakra/SpiderMonkey/JSC contributors can land spec tests (in their home project repos, or in the spec repo directly)
       1. Change status quo, WPT becomes the source of truth
          1. Build a 2-way sync between V8 + Chakra for a slice of WPT (reuse existing mirror for others)
          1. V8/Chromium/Chakra/SpiderMonkey/JSC contributors can land spec tests (in their home project repos, or in WPT directly)
          1. WASM spec changes must include a spec branch/PR and a WPT branch/PR
       1. [Detailed (v8 specific) doc](https://docs.google.com/document/d/1FEjBpsIj1vu5ZUJI6ki6xKBFIijxlPj_rQmsucMMQrg/edit#)
   1. Poll on proposals
       1. POLL: Tests contributed to the WASM spec repo should receive review from core contributors. 
       1. POLL: WASM should maintain the source of truth by having compliance tests in the spec repo, as today.
  • JF: Core and JS tests must still be runnable in a JS shell, as well as in a non-JS embedding. This is critical to fast VM iteration (both for our VM and non-web embedders). That doesn’t seem to be the case if we move to WPT. If this is the intent, the polls should be predicated on it.
  • Titzer / Brad: agree on V8 side.
  • JF: We’d like to understand how much slower tests will run with the WPT harness. These costs must remain low. Can you provide such comparison?
  • Titzer: JS tests in wasm spec repo use same framework as WPT (was added by Mozilla). Wouldn’t be slower. For HTML it’s a whole-browser, so of course slower.
  • JF: We want to reduce friction when adding WebAssembly tests and when importing them. It’s unclear at this time that WPT friction is low enough.
  • Brad: intent is: commit to your own repo, and automation just pushes to WPT. Bots do the work.
  • Andreas: that assumes infrastructure which isn’t there, policies, etc.
  • Brad: some willingness on Google side for support. Attempting to use infrastructure to fix the problem.
  • Dan Ehrenberg: the assumption is that there is infrastructure, otherwise two-way sync makes no sense.
  • Mike Holman: I’m against 2-way sync into browser engines. I don’t want random tests to break Chakra CI. Adds overhead. Sounds way simpler if source of truth is spec repo. We don’t need all this sync.
  • Brad: context for where it is used: a commit can come in to one browser and break others. Automation updates test expectation files so new, failing tests are disabled. Downside is auto turn-off, upside is no breakage. You do have to stay vigilant.
  • JF: seems premature adoption of WPT a bit. Heard concerns about random people adding WebAssembly tests. Such a situation would indeed be very troubling. Is there a concept of "owner"?
  • Titzer: I’m personally for status-quo. If we need to pay people to write compliance tests I think it’s better. I don’t think tests define WebAssembly. Tests are often bugs that were found, not really defining compliance.
  • Andreas: affects proposal process for evolving WebAssembly. Our current process involves forking spec repo, and adding tests there. It’s all worked on and committed together.
  • Brad: maybe we can rule out discussing source of truth, and instead focus on sync?
  • JF: It would be informative to hear why JavaScript’s own tests aren’t on WPT. WebAssembly being the new standard, We would rather follow the lead of JavaScript here instead of re-hash discussions that occurred in TC39. If anything WebAssembly is less “Web” than JavaScript is, so I’d think we shouldn’t be on the “Web” platform tests before JS? The WebKit team’s current position w.r.t. WPT applies to Web tests only. All things being equal we think being on WPT is good, but we also don’t consider WebAssembly and JavaScript as equal to other Web things.
  • Dan Ehrenberg: JS would probably benefit from two-way sync; even though we have similar requirements to have conformance tests with proposals, these tests have historically had holes in them, which were filled in by individual browser tests and could’ve been upstreamed sooner with fixing other browser bugs faster, if we had two-way sync.
  • Andreas: we can do things one at a time. Try to downstream to WPT, and see how that goes. Right now 2-way doesn’t solve a real problem, solution looking for a problem.
  • Titzer: spec repo has reference impl and tests so that by design changes and bug fixes are tested as part of the process.
  • Andreas: historically test262 was quite verbose, and people didn’t contribute to it. It’s not a good comparison, not about automation, just that test262 had too much bureaucracy.
  • Luke: questions about 1-way sync. There’s wast, JS, and HTML tests? We already mocked wast running in WPT harness, that would still work?
  • Andreas: yes. We also could just build the JS version in the test repo, and push it over the fence.
  • Luke: we certainly need to keep wast tests, and the JS one. HTML ones I can imagine in different place, but I can imagine problems with a split source of truth.
  • Brad: seems strong consensus to keep source of truth in spec repo.

Unanimous consent to keep spec repo as source of truth.

  • Brad: I like that 1-way sync help us learn more.
  • JF: review is already a thing we need to do for spec repo. Presumably new tests are covering missing things that are already spec’d, or something new being spec’d. Ownership is already part of our proposal process.
  • Brad: maybe we want curated as well as un-curated tests?
  • Titzer: we don’t have a reference interpreter for the JS embedding, so there’s a higher chance that engine quirks could leak into tests for JS embedding.
  • JF: we’re over time, let’s continue that discussion at the in-person meeting hosted by Intel.

Closure

Adjourned