Automatic TypeScript definition generation from JSG RTTI

[mrbbot]

Hey! 👋 This PR adds support for generating TypeScript types from JSG RTTI, replacing the internal autodecl script. This forms the basis for the next workers-types version. These scripts are located in this repository rather than workers-types as they depend on Bazel outputs, and we'd like to be able to share the Bazel cache in CI. Going forward, DevProd should probably be the CODEOWNER for everything in the types directory.

To generate types, run:

$ bazel build //types:types

A gist containing the generated types can be found here. This also includes a copy of the types before TypeScript transformers are applied.

(fyi, about half of the additions are from pnpm-lock.yaml)

Implementation Notes and Questions

1. Added a jsg::fullyQualifiedTypeName() method. This behaves like jsg::typeName(), but includes namespaces and template arguments in the returned name. Namespaces are required to differentiate nested types with the same name (e.g. DurableObjectStorageOperations::GetOptions, KvNamespace::GetOptions and R2Bucket::GetOptions). Template arguments are required to differentiate jsg::IteratorBase... types generated by JSG_ITERATOR.
2. In workerd::jsg::rtti::Builder, replaced the symbol key from jsg::typeName() to jsg::fullyQualifiedTypeName() for the reasons above. This also affects which values should be passed as parameters to workerd::jsg::rtti::Builder::structure().
3. Changed the rtti.capnp schema. Generally, I tried to evolve the schema in a backwards-compatible way, but it would be cleaner if we're still able to make breaking changes. /cc @mikea
   1. Added fullyQualifiedName fields to Structure and StructureType schemas for the reasons above. I was hesitant to make name fully-qualified since I wasn't sure which other code depended on RTTI.
   2. Added support for jsg::LenientOptional to RTTI. When generating TypeScript definitions, we also need to be able to distinguish between optionals expecting null (kj::Maybe) and others expecting undefined. A new name field has been added to the MaybeType schema, similar to NumberType and StringType.
   3. Similarly, we need to be able to distinguish between kj::Array, kj::ArrayPtr and jsg::Sequence, so a new name field has been added to the ArrayType schema. It may be better to make this and the previous maybe field enums instead?
   4. Added a name field in a group with nested members. This ensures members coming from JSG_NESTED_TYPE_NAMED macros have the correct names.
   5. Added iterator and asyncIterator fields to the Structure schema. Whilst there are already iterable and asyncIterable boolean fields, we need to know the full method types (especially the returned (Async)Iterator type) for [Symbol.iterator]/[Symbol.asyncIterator].
4. Added a new api-encoder.c++ entrypoint that spits out RTTI to a file. All TypeScript generation scripts are written in TypeScript, so we can use the official TypeScript compiler API for creating/processing/printing AST nodes.
5. Setup Bazel to build/run JavaScript/TypeScript programs. I used https://github.com/aspect-build/rules_js instead of https://github.com/bazelbuild/rules_nodejs as this seems to be the actively maintained version. Note aspect-build/rules_js requires us to use pnpm. We should be able to use this setup for packaging/publishing workerd npm packages too. /cc @penalosa

TODOs

(to follow in later PRs)

1. Overrides: whilst correct TypeScript types are being generated without them (an improvement on autodecl 😅), they're not as accurate as they could be.
2. Parameter names: currently parameters are named param0, param1, ... . Ideally, we'd use the actual C++ parameter names here, so we need some way of getting these into the RTTI.
3. Compatibility dates: we currently generate a single set of types with all non-experimental compatibility flags enabled. We'd like types to depend on which flags users have enabled.
4. CI: we'd like to build/publish types automatically on PRs/releases.
5. Multiple files: we currently generate a large single file containing all types. We'd like to split this up into multiple purpose-related files, hence this implementation splits structures into named groups.

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[mrbbot]

I have read the CLA Document and I hereby sign the CLA

[penalosa]

This is very very cool!