Accept iterable and async iterable in functions?

[twiss]

What problem are you trying to solve?

In Web Crypto, we're considering to allow passing an iterable<BufferSource> or async iterable<BufferSource> in addition to just BufferSource to crypto.subtle.digest, in order to allow incremental hashing.

Currently, Web IDL only offers syntax for defining iterable interfaces (e.g. to return them), but not for accepting iterable objects (of any interface).

To be fair, it's not clear to me whether any other API will need this functionality as well, so I'm not sure whether this needs to be solved in Web IDL, or whether doing so is even easily possible. So, this issue is primarily to discuss what would be the best solution rather than immediately proposing an addition.

What solutions exist today?

A workaround could be something like

typedef object BufferSources; // (BufferSource or iterable<BufferSource> or async iterable<BufferSource>)

partial interface SubtleCrypto {
  Promise<ArrayBuffer> digest(
    AlgorithmIdentifier algorithm,
    BufferSources data
  );
}

or some variation of that, and then manually check in the spec prose that data implements BufferSource or iterable<BufferSource> or async iterable<BufferSource>.

Note that for the latter, this check is somewhat involved because we need to check that every value produced by the iterable is a BufferSource, and potentially do so asynchronously (and reject a promise rather than throw an exception, although Web Crypto already says to do so for all Web IDL conversion failures anyway).

How would you solve it?

We could allow syntax like

typedef (BufferSource or iterable<BufferSource> or async iterable<BufferSource>) BufferSources;

partial interface SubtleCrypto {
  Promise<ArrayBuffer> digest(
    AlgorithmIdentifier algorithm,
    BufferSources data
  );
}

And then offer some convenience methods to consume values from the iterables, convert the ECMAScript values to IDL values, and return an error if they can't be converted to the specified type, for example.

Anything else?

Just to be transparent, Web Crypto's use case is somewhat specific in that we explicitly want to allow the function to retain a reference to the passed object, and consume the values produced by the iterable incrementally, to conserve memory. In that sense, using sequence<BufferSource> wouldn't serve our use case, but perhaps it serves most other use cases.

All of that being said, allowing something like (BufferSource or iterable<BufferSource> or async iterable<BufferSource>) would clean up the syntax for this specific use case, and sort of fill out an asymmetry in the current spec in that it easily allows returning an iterable but not accepting one.

But, let me know what you think would be the best way to handle this use case :)

[annevk]

See also #1397 which I was just reminded of. You probably want to validate your approach against that PR and related discussion.

[twiss]

Ah, interesting, thanks for the pointer. That does look like exactly what we need, and it's good to know that there are other uses cases, as well :)

In that PR, async iterable<T> is used to refer to either a synchronous or asynchronous iterable, which does happen to be what we need, and I guess there's never any real reason to accept an async but not a sync iterable, so that simplifies things slightly as well.

[domenic]

but not for accepting iterable objects (of any interface).

sequence<InterfaceName> is the syntax for this. (It only accepts sync iterables, though.)

[twiss]

sequence<BufferSource> wouldn't serve our use case because it makes a copy of all the data, whereas

we explicitly want to allow the function to retain a reference to the passed object, and consume the values produced by the iterable incrementally, to conserve memory.

In other words, the differences described here are important for us (beyond accepting async iterables).

[annevk]

It doesn't make a copy of all the bytes involved. It just copies the list.

[twiss]

Right, OK. But it still requires all BufferSources to be in memory at the same time, so it doesn't enable "real" incremental hashing (e.g. imagine a generator which computes or retrieves the data to be hashed on demand, or so).
Perhaps just saying that we also need async iterables is a simpler argument though.