Autogenerate __text_signature__

[konstin]

As discussed in #305, we can't use __annotations__ in native code. We could however generate __text_signature__, using a similar implementation approach as described in #305.

[programmerjake]

would you accept a pull request that allows adding a user-provided signature using an annotation and have pyo3 automatically put it in the correct location in the docstring?

/// doc string 1
#[pyclass(signature = "MyClass(arg1, *, arg2=0)")]
/// doc string 2
struct MyClass {}

#[pymethods]
impl MyClass {
    #[new]
    /// doc string 1
    #[signature = "__new__($cls, arg1, *, arg2=0)"]
    /// doc string 2
    #[args(arg1, "*", arg2 = "0")]
    fn new(obj: &PyRawObject, arg1: i32, arg2: i32) {
        obj.init(Self {});
    }

    /// doc string 1
    #[signature = "my_method($self)"]
    /// doc string 2
    fn my_method(&self) {}

    #[staticmethod]
    #[signature = "method_without_docs()"]
    fn method_without_docs() {}
}

/// doc string 1
#[pyfunction(signature = "my_function(arg1, arg2)")]
/// doc string 2
fn my_function(arg1: i32, arg2: i32) -> i32 {
    0
}

The generated docstrings would be:

• for MyClass:

  "MyClass(arg1, *, arg2=0)\n--\n\ndoc string 1\ndoc string 2"

• for MyClass::new:

  "__new__($cls, arg1, *, arg2=0)\n--\n\ndoc string 1\ndoc string 2"

• for MyClass::my_method:

  "my_method($self)\n--\n\ndoc string 1\ndoc string 2"

• for MyClass::method_without_docs:

  "method_without_docs()\n--\n\n"

• for my_function:

  "my_function(arg1, arg2)\n--\n\ndoc string 1\ndoc string 2"

[kngwyu]

🤔
Is it related to type annotation?

[programmerjake]

Is it related to type annotation?

yes, it basically tells CPython what the function would look like if it were written in Python. According to CPython's docs, it is a part of Argument Clinic, which doesn't have the same stability guarantees as the rest of the C API, however, it is implemented in pretty much the same way in both CPython and PyPy.

Note that __text_signature__ doesn't seem to support type annotations; if those are desired, setting __signature__ is needed, which is much more complex to construct.

My proposal is relatively low implementation effort -- it just copies the provided string into the front of the doc string with the appropriate "\n--\n\n" separator. It may also be a good idea to warn when the given signature string doesn't match what CPython expects (which is very simple to check for), which would cause CPython to leave the given signature at the beginning of the docstring instead of putting it in __text_signature__.

[programmerjake]

I'm working on creating a pull request that implements the design in #310 (comment) except that I'm using text_signature as the attribute/parameter name instead of signature.

[programmerjake]

I'm working on creating a pull request that implements the design in #310 (comment) except that I'm using text_signature as the attribute/parameter name instead of signature.

Created: #675

[davidhewitt]

I plan to have a go at seeing how hard it would be to autogerate __signature__ (i.e. with type annotations) as part of the experiments I'll be trying with #684

[aldanor]

One good example of auto-generated signatures is in pybind11, where, for instance, you get signatures very close to being mypy-like, e.g. if your function takes (std::vector<int>, std::unordered_map<int, bool>, py::array_t<double>), you get something like (List[int], Dict[int, bool], np.ndarray[np.float64]). Maybe some ideas to steal so we don't have to reinvent the wheel here as it's very nice and readable to the end-user.

[davidhewitt]

A first implementation of this was completed in #2784

[merwok]

Two notes:

• function signatures are not necessarily related to type annotations: parameters can be annotated, but function signatures exist before and in parallel to the typing system; Signature objects and the inspect.signature function were developed on their own (with PEP 362)
• __text_signature__ is not officially supported – follow https://discuss.python.org/t/request-for-review-of-gh-81677-basic-support-for-annotations-in-text-signature-s/43914 to know if its status changes