Update API to latest IDL (#369)

* Tweak codegen docs a bit

* Update codegen's idl parser

* More tweaks to codegen

* new IDL

* Codegen plus manual updates

* Fix that features are not aligned between webgpu and wgpu-native

* make new public prop available

* update changelog

* Raise rather than assert, also in a few other places

CHANGELOG.md

@@ -17,6 +17,15 @@ Possible sections in each release:
 * Fixed: for any bug fixes.
 * Security: in case of vulnerabilities.
 
+### [v0.10.1] - tdb
+
+In this release the API is aligned with the latest webgpu.idl.
+
+Added:
+
+* New `wgpu.wgsl_language_features` property, which for now always returns an empty set.
+* The `GPUShaderModule.compilation_info` property (and its async version) are replaced with a `get_compilation_info()` method.
+
 
 ### [v0.9.5] - 02-10-2023
 

README.md

@@ -112,12 +112,12 @@ This code is distributed under the 2-clause BSD license.
 * Use `pip wheel --no-deps .` to build a wheel.
 
 
-### Changing the upstream wgpu-native version
+### Updating to a later version of WebGPU or wgpu-native
+
+To update to upstream changes, we use a combination of automatic code
+generation and manual updating. See [the codegen utility](codegen/README.md)
+for more information.
 
-* Use the optional arguments to `python download-wgpu-native.py --help` to
-  download a different version of the upstream wgpu-native binaries.
-* The file `wgpu/resources/wgpu_native-version` will be updated by the script to
-  track which version we depend upon.
 
 ## Testing
 

codegen/README.md

@@ -10,11 +10,12 @@ The purpose of this helper package is to:
 * To validate that our calls into wgpu-native are correct.
 
 We try to hit a balance between automatic code generation and providing
-hints to help with manual updating. It should *not* be necessarry to check
-the diffs of `webgpu.idl` or `wgpu.h`; any relevant differences should
+hints to help with manual updating. It should *not* be necessarry to
+check the diffs of `webgpu.idl` or `wgpu.h`. Instead, by running the
+codegen, any relevant differences (in webgpu.idl and wgpu.h) should
 result in changes (of code or annotations) in the respective `.py`
-files. That said, during development it can be helpful to use the
-WebGPU spec and the header file as a reference.
+files. That said, during development it can be helpful to use the WebGPU
+spec and the header file as a reference.
 
 This package is *not* part of the wgpu-lib - it is a tool to help
 maintain it. It has its own tests, which try to cover the utils well,
@@ -39,7 +40,7 @@ tests, because we are the only users. If it breaks, we fix it.
   flag/enum mismatches between IDL and wgpu.h.
 
 
-## Updating the base API
+## Updating the base API (`base.py`)
 
 The WebGPU API is specified by `webgpu.idl` (in the resources directory).
 We parse this file with a custom parser (`idlparser.py`) to obtain a description
@@ -55,48 +56,52 @@ Next, the Python base API (`base.py`) is updated:
 
 The update process to follow:
 
-* Download the latest `webgpu.idl`.
+* Download the latest `webgpu.idl` (see link above) and place in the resources folder.
 * Run `python codegen` to apply the automatic patches to the code.
-* Now go through all FIXME comments that were added, and apply any necessary
-  changes. Remove the FIXME comment if no further action is needed. Note that all
-  new classes/methods/properties (instead those marked as hidden) need a docstring.
-* Also check the diff of `flags.py`, `enums.py`, `structs.py` for any changes that might need manual work.
-* Run `python wgpu.codegen` again to validate that all is well.
+* It may be necessary to tweak the `idlparser.py` to adjust it to new formatting.
+* Check the diff of `flags.py`, `enums.py`, `structs.py` for any changes that might need manual work.
+* Go through all FIXME comments that were added in `base.py`:
+    * Apply any necessary changes.
+    * Remove the FIXME comment if no further action is needed, or turn into a TODO for later.
+    * Note that all new classes/methods/properties (instead those marked as hidden) need a docstring.
+* Run `python codegen` again to validate that all is well. Repeat the step above if necessary.
 * Make a summary of the API changes to put in the release notes.
 * Update downstream code, like our own tests and examples, but also e.g. pygfx.
 
-In some cases we may want to deviate from the WebGPU API, because well ... Python
-is not JavaScript. To tell the patcher logic how we deviate from the WebGPU spec:
+In some cases we may want to deviate from the WebGPU API, because well ...
+Python is not JavaScript. There is a simple system in place to mark any
+such differences, that also makes sure that these changes are listed
+in the docs. To mark how the py API deviates from the WebGPU spec:
 
 * Decorate a method with `@apidiff.hide` to mark it as not supported by our API.
 * Decorate a method with `@apidiff.add` to mark it as intended even though it does not
   match the WebGPU spec.
 * Decorate a method with `@apidiff.change` to mark that our method has a different signature.
 
+While `base.py` defines the API (and corresponding docstrings), the implementation
+of the majority of methods occurs in the backends.
+
 
 ## Updating the API of the backend implementations
 
 The backend implementations of the API (e.g. `rs.py`) are also patched.
-In this case the source is the base API (instead of the IDL).
-
-The update process is similar to the generation of the base API, except
-that methods are only added if they `raise NotImplementedError()` in
-the base implementation. Another difference is that this API should not
-deviate from the base API - only additions are allowed (which should
-be used sparingly).
+The backends are almost a copy of `base.py`: all methods in `base.py`
+that `raise NotImplementedError()` must be implemented.
+ The API of the backends should not
+deviate from the base API - only additions are allowed (and should be
+used sparingly).
 
-You'd typically update the backends while you're updating `base.py`.
+You'll typically need to update the backends while you're updating `base.py`.
 
 
 ## Updating the Rust backend (`rs.py`)
 
 The `rs.py` backend calls into a C library (wgpu-native). The codegen
 helps here, by parsing the corresponding `wgpu.h` and:
 
-* Detect and report missing flags and flag fields.
-* Detect and report missing enums and enum fields.
+* Detect and report missing flags and enum fields.
 * Generate mappings for enum field names to ints.
-* Validate and annotate struct creations.
+* Validate and annotate struct creations (missing struct fields are filled in).
 * Validate and annotate function calls into the lib.
 
 The update process to follow:
@@ -107,10 +112,10 @@ The update process to follow:
 * Diff `rs.py` to see what structs and functions have changed. Lines
   marked with a FIXME comment should be fixed. Others may or may not.
   Use `wgpu.h` as a reference to check available functions and structs.
-* Run `python wgpu.codegen` again to validate that all is well.
+* Run `python codegen` again to validate that all is well.
 * Make sure that the tests run and provide full coverage.
 * This process typically does not introduce changes to the API, but certain
-  features that were previously not supported could be after an update.
+  features that were previously not supported could now be withing reach.
 
 
 ## Further tips

codegen/apipatcher.py

@@ -313,11 +313,18 @@ def get_method_def(self, classname, methodname):
         # Get arg names and types
         args = idl_line.split("(", 1)[1].split(")", 1)[0].split(",")
         args = [arg.strip() for arg in args if arg.strip()]
-        defaults = [arg.partition("=")[2].strip() for arg in args]
-        defaults = [
-            default or (arg.startswith("optional ") and "None")
-            for default, arg in zip(defaults, args)
-        ]
+        raw_defaults = [arg.partition("=")[2].strip() for arg in args]
+        place_holder_default = False
+        defaults = []
+        for default, arg in zip(raw_defaults, args):
+            if default:
+                place_holder_default = "None"  # any next args must have a default
+            elif arg.startswith("optional "):
+                default = "None"
+            else:
+                default = place_holder_default
+            defaults.append(default)
+
         argnames = [arg.split("=")[0].split()[-1] for arg in args]
         argnames = [to_snake_case(argname) for argname in argnames]
         argnames = [(f"{n}={v}" if v else n) for n, v in zip(argnames, defaults)]

codegen/files.py

@@ -1,3 +1,7 @@
+"""
+Simple utilities to handle files, including a mini virtual file system.
+"""
+
 import os
 
 

codegen/idlparser.py

@@ -198,6 +198,8 @@ def resolve_type(self, typename):
             "boolean": "bool",
             "object": "dict",
             "ImageBitmap": "memoryview",
+            "ImageData": "memoryview",
+            "VideoFrame": "memoryview",
             "GPUPipelineConstantValue": "float",
         }
         name = pythonmap.get(name, name)
@@ -255,6 +257,8 @@ def _parse(self):
                     value = value.split("]")[-1]
                 # Parse
                 if value.startswith("("):  # Union type
+                    while ")" not in value:
+                        value = value.rstrip() + " " + self.read_line().lstrip()
                     assert value.count("(") == 1 and value.count(")") == 1
                     value = value.split("(")[1]
                     val, _, key = value.partition(")")
@@ -265,6 +269,8 @@ def _parse(self):
             elif line.startswith(("namespace ", "interface ", "partial interface ")):
                 # A class or a set of flags
                 # Collect lines that define this interface
+                while "{" not in line:
+                    line = line.rstrip() + " " + self.read_line().lstrip()
                 lines = [line]
                 while not line.startswith("};"):
                     line = self.read_line()
@@ -344,6 +350,8 @@ def _parse(self):
                     d[key] = val
                 self.enums[name] = d
             elif line.startswith("dictionary "):
+                while "{" not in line:
+                    line = line.rstrip() + self.read_line()
                 assert line.count("{") == 1 and line.count("}") == 0
                 lines = [line]
                 while not line.startswith("};"):
@@ -390,7 +398,12 @@ def _post_process(self):
         """
 
         # Drop some toplevel names
-        for name in ["NavigatorGPU", "GPUSupportedLimits", "GPUSupportedFeatures"]:
+        for name in [
+            "NavigatorGPU",
+            "GPUSupportedLimits",
+            "GPUSupportedFeatures",
+            "WGSLLanguageFeatures",
+        ]:
             self._interfaces.pop(name, None)
 
         # Divide flags and actual class definitions

tests/test_api.py

@@ -166,6 +166,7 @@ class GPU:
             wgpu._register_backend(GPU)
 
         GPU.request_adapter_async = lambda self: None
+        GPU.wgsl_language_features = set()
         wgpu._register_backend(GPU)
 
         assert wgpu.GPU is GPU

tests/test_rs_basics.py

@@ -156,7 +156,7 @@ def test_shader_module_creation_spirv():
     code4 = type("CodeObject", (object,), {})
 
     m1 = device.create_shader_module(code=code1)
-    assert m1.compilation_info() == []
+    assert m1.get_compilation_info() == []
 
     with raises(TypeError):
         device.create_shader_module(code=code4)

wgpu/__init__.py

@@ -33,6 +33,7 @@ def _register_backend(cls):
     globals()["GPU"] = GPU
     globals()["request_adapter"] = gpu.request_adapter
     globals()["request_adapter_async"] = gpu.request_adapter_async
+    globals()["wgsl_language_features"] = gpu.wgsl_language_features
     if hasattr(gpu, "print_report"):
         globals()["print_report"] = gpu.print_report
     else: