Loss of typing information on composed functions

[muxator]

Passing any function through compose.compose() seems to strip away all the information about the types of those functions.

Minimal example:

import compose


def f(x: int) -> str:
    return f"{x}"


# compose f with the identity function
g = compose.compose(f)

reveal_type(f(1))
reveal_type(g(1))

Testing on python 3.11 yields:

$ mypy --version
mypy 1.5.1 (compiled: yes)

$ mypy test_compose.py 
test_compose.py:11: note: Revealed type is "builtins.str"
test_compose.py:12: note: Revealed type is "Any"

I expected that the static type checker would have printed builtins.str even for g(1).

This is a stripped down example of a larger problem I had early on on a project of mine, where I wrongly assumed that composed functions would have brought over the type information about their signature. I ended up introducing a bug in my project because of this assumption.

This issue is a cross post of something I also found in another library (pytoolz/toolz#569).

Is this a limitation of the libraries? Of the language itself? Or the static analyzers?

Thank you.

[mentalisttraceur]

It's partially a library limitation (duplicate of #5 as far as that goes), and partially a language+analyzer limitation - Python has been slow to add type hints for combining/modifying callables (some relevant discussion in pytoolz/toolz#523 ), and then static analyzers are sometimes slow to support those few hints that we do get (as you might've seen in python/mypy#12280 ).

In particular, last I checked there was still no good way provided by Python to tell static type checkers that the return type of one function is supposed to be the same as the argument type of the next function in a variadic way. So compose type hints have to be hard-coded for each arity (number of composed functions).

I do apologize if the "can be type-checked" in my project description caused your assumption. When I wrote that, I was just talking about type-checking at runtime (i.e. isinstance checks to distinguish a compose instance from functions/lambdas), and I was able to convince myself that the misinterpretation potential was acceptably low because static type analysis wasn't yet so common in Python.

[muxator]

Thank you for your explanation, now it's more clear!

I suppose I'll have to wait some more time before the python ecosystem matures enough for this type of use case.

Just for information, the use case I had was a simple streaming parser for a text based file format. I wanted to return strongly typed values, I wanted to read the input file lazily, and I also wanted to separate the I/O from the pure parsing, so that the function was easily testable.

The composition would have been useful because I would write something like:

data: Iterable(SomeFormat) = stream_file(path) | filter_lines | parse

instead of:

data: Iterable(SomeFormat) = parse(filter_lines(stream_file(path)))

As long as I stay in python-land, I guess I'll stick to write g(f(h(x))) for the moment 😄; it's visually more difficult, but as expressive as the first form.

Thank you again!

[mentalisttraceur]

Do you mind installing https://pypi.org/project/compose-stubs along with compose and letting me know if type-checking starts to work for you?

Just note the 0.* version number and "alpha" development status classifier for now.

I've tested with MyPy, Pyre, and Pyright, and it works for compose and acompose (I'm still working on type-hinting sacompose - the simplest implementation loses statically-known is-this-async-or-not in some cases, but to avoid that requires a combinatoric explosion of overloads).

(MyPy struggles when the return type of one composed function and the argument type of the next composed function, MyPy's error is the rather unhelpful "Cannot infer type of argument ..." instead of the more helpful "... incompatible type ...; expected ..."; Pyre and Pyright don't have this problem, and in particular I want to compliment Pyright for actually saying what the concrete mismatch of types is.)

[muxator]

I'll try it tomorrow, thanks!

[mentalisttraceur]

Stubs package is now about as good as it's going to get, I think (until Python adds better type hint features). I've bumped it up from alpha to beta, and If we don't notice any serious issues in the near future I'll promote it to stable v1.0.0 soon.

[muxator]

Thanks @mentalisttraceur, after installing compose-stubs the simple example I posted in this issue now is OK!

$ mypy test_compose.py 
test_compose.py:11: note: Revealed type is "builtins.str"
test_compose.py:12: note: Revealed type is "builtins.str"

I only tested my simple use case of composition of two functions at a time, and works perfectly for me.
I do not mind the less informative error message from mypy, since that would come from a problem inside the module instead of on its APIs, I can live with that, or migrate to pyright.

At this point, in compose's README, I'd suggest to mention the usefulness of also installing compose-stubs if one wants to have non-surprising static typing.

Thanks again!

[mentalisttraceur]

At this point, in compose's README, I'd suggest to mention the usefulness of also installing compose-stubs if one wants to have non-surprising static typing.

Yep! That was my plan, I was just delaying to see if we noticed any deal-breaking issues. Since we haven't, compose-stubs is promoted to a stable v1.0.0 release and compose v1.5.0 refers to the stubs package in the description/README.