optional static typing for toolz

[Jasha10]

The type hints described in PEP-484 provide a convenient way to reason about code, and make it possible for static type checkers like mypy to catch type-mismatch errors.

Here is an example of how static type checking might be able to fit in with the toolz library:
Suppose f and g are typed functions defined as follows:

def f(x: A) -> B:
    ...
def g(y: B) -> C:
    ...

Since f is Callable[[A], B] and g is Callable[[B], C], one would expect the composite toolz.compose(g, f) to be Callable[[A], C]. (Here I am using the Callable protocol from the typing module).

Of course, implementing this could be tricky.

[eriknw]

Yeah, this trend is becoming a standard. I'm okay with adding static typing as long as it doesn't hinder readability too much. Hopefully it can be done in a way to improve readability.

Thanks for the suggestion @Jasha10! Are you willing to get this started? I found somebody created some typing information for cytoolz, which may be a good starting point: https://github.com/blindspot-ai/cytoolz-stubs

[Jiehong]

Very good idea!

Using stub files like cytoolz-stubs would not hinder readability much, but it would duplicate all definitions though (https://mypy.readthedocs.io/en/stable/stubs.html#stub-files).

Alternatively, this could be put in the source files directly, with minimal readability impact (IMO, it improves readability). As this uses python 3.5+, and types are supported in python 3.5+, this would not even break compatibility.

(mypy could be used to ensure all types are consistent within this lib as well, as a side effect).

Is there any preferred approach?

[eriknw]

I'd prefer to put annotations in the source files directly. Easier to maintain. It's clear that annotating functions has gained a lot of traction, so it's probably time to jump on the bandwagon.

[multimeric]

I note that toolz supports Python 3.5+. Would a dependency on typing_extensions be acceptable given that there are no other dependencies at the moment? This would expand how expressive we can make the typing.

[multimeric]

Also, there is the question of the right type checker to use. It seems that mypy actually doesn't support a lot of the more advanced PEPs that we might want for functional programming: Concatenate, Unpack etc

[gwerbin]

@multimeric Pyright seems popular enough, and tends to support new features sooner than Mypy. If you wanted to enable type checking in CI, Pyright would probably be fine. It's unfortunate that Mypy doesn't support the hot new features yet, but targeting one type checker is better than targeting none.

I think a dependency on typing-extensions is totally reasonable given that it's an "official" package.

[LincolnPuzey]

I have opened a PR #552 adding a pyright run to the CI.

It is important to note that even though we might use pyright to type-check toolz,
any type hints added to toolz will also be used by whatever type-checker dependent projects use.

So type hints should be written according to the rules of the standard library, not to whatever makes pyright pass.

[multimeric]

It's tricky though, because I started writing some type hints for toolz, and found that I needed to rely pretty heavily on new typing features like Unpack which mypy doesn't support (python/mypy#12280). So while these type hints would be correct according to the PEP specifications, I'm not sure mypy projects would see them as valid. What should be done in this case?

[LincolnPuzey]

I guess the simplest option would be to incrementally add type hints, and only use typing features once we are confident the major type checkers all support them.

Or if we know that a typing feature is not supported, but a type checker ignores it, rather than failing, we might go ahead and use it.

[LincolnPuzey]

I quickly ran into another question, do we add type hints that either (1) support ALL possible ways to call a function from toolz, or (2) just the 'best practice' or 'intended' ways?

For example, the remove function:

def remove(predicate, seq):
   return itertools.filterfalse(predicate, seq)

could be typed in two different ways;

(1) The following type hints for the function specify ALL the ways remove can currently be called without error

T = TypeVar("T")

def remove(predicate: Union[None, Callable[[T], Any]], seq: Iterable[T]) -> Iterator[T]:
   return itertools.filterfalse(predicate, seq)

(2) Or the following (more specific) type hints for the function declare the "intended" way to call the function (predicate can't be None and must return a bool). I would say that this way is better because it encourages callers to call functions as intended.

T = TypeVar("T")

def remove(predicate: Callable[[T], bool], seq: Iterable[T]) -> Iterator[T]:
   return itertools.filterfalse(predicate, seq)

I would choose option (2) if writing toolz from scratch.
But adding type hints like that may not be backwards-compatible with existing type-checked code that uses this function.

@eriknw Can I get your opinion on this?

[sid-kap]

Are there any plans to add an empty py.typed file in toolz, so that projects using toolz can use the type hints? (See https://mypy.readthedocs.io/en/stable/installed_packages.html)

[carlwr]

@eriknw is there an update on thoughts on + possible plans for bringing typing into toolz?

If I'm allowed to make a short pitch for that:

• There is a clear trend both within the Python community and software development in general to move towards typing (this is also mentioned in the conversation above)

• Python still lacks a high-quality, reasonably well-used, external library for functional programming that includes typing support [1]

A first step could be to bring some typing support to a few widely-used functions like curry, compose and pipe. Perhaps not necessarily covering all use cases for these, but a good part of them.

---

[1]: dry-python/returns would probably be the closest in that regard, but with its container-based approach it has a slightly different direction than toolz, also, its toolkit is far less comprehensive than toolz's and much less stable/reliable.

[Ilykuleshov]

Any update on this? It's the one thing stopping me from using toolz rn, would be great if this was fixed

[petergaultney]

Would y'all be open to a PR that starts with the easy stuff instead of the hard stuff?

itertoolz, for the most part, should be extremely easy to add type annotations to - nearly all functions would be some form of (seq: Sequence[T]) -> T or (seq: Sequence[T]) -> Sequence[T]. That, plus a py.typedfile in the root, would give our team what we need to start using at least theitertoolz` subset of functionality without writing a lot of untyped code.

returns has been mentioned, and it covers our typed compose/curry/pipe needs quite well, but I'm not aware of a library that does what itertoolz does and has static types. And typing compose and friends is not quite as simple - you need a lot of overload-type stuff usually as far as I understand it.