Add basic support for PEP 695 type parameters in functions

[gentlegiantJGC]

PEP 695 added syntax for template arguments in classes, methods and functions.

Function example

def func[T](arg: T) -> T:
    return arg

This syntax is not generated by pybind11 but it can be added with custom type hints in a docstring like this.

    py::options options;
    options.disable_function_signatures();
    m.def(
        "passthrough1",
        [](py::object obj) { return obj; },
        py::doc("passthrough1[T](obj: T) -> T\n"));
    options.enable_function_signatures();

This pull requests adds basic support for this syntax.

It does not implement support for classes, variadics or constraints. I will leave this to someone who needs this.

[skarndev]

I think we should make this feature opt-in as a CLI switch as long as Python 3.9 and 3.11 have not reached their EOL. In many cases people want to generate a stub package that would cover all the Python versions they want to currently support. In this case, stubgen can act as linter, failing on parsing such docstrings if the flag is not passed.

[gentlegiantJGC]

Why couldn't we use the Python version the stubs are built with to dictate what features are valid?
Stubgen already outputs different stubs based on the Python version being used to generate them.
If a user wants stubs compatible with an old version they should probably be running stubgen on that old version.

I agree that this syntax should not appear in stubs generated by Python 3.11 and before.
I should add and exception or something to warn the user of the incompatibility. How is that usually done with this library?

Ps. Python 3.9 has already reached end of life.

[skarndev]

Why couldn't we use the Python version the stubs are built with to dictate what features are valid? Stubgen already outputs different stubs based on the Python version being used to generate them. If a user wants stubs compatible with an old version they should probably be running stubgen on that old version.

I agree that this syntax should not appear in stubs generated by Python 3.11 and before. I should add and exception or something to warn the user of the incompatibility. How is that usually done with this library?

Ps. Python 3.9 has already reached end of life.

I meant 3.10, sorry.

The reason that pinning a specific Python version to produce a different syntax stub does not work is that it would require building version-specific wheels of a stub package for the end user. If we do not produce backwards compatible syntax (for the range of supported interpreter versions), regardless of which Python version stubgen is run on, I think this should be fixed too, unless such incompatibility cannot be avoided e.g. with typing_extensions

[gentlegiantJGC]

I just want to clarify that the syntax I am adding support for is not automatically generated by pybind11. The library author needs to manually inject it into the docstring. It is down to them to ensure that this syntax only appears in libraries generated with python 3.12+. In older versions the syntax is invalid and stubgen should rightly error.

If we do not produce backwards compatible syntax (for the range of supported interpreter versions), regardless of which Python version stubgen is run on, I think this should be fixed too, unless such incompatibility cannot be avoided e.g. with typing_extensions

As you said, the stubs generated by pybind11 are aleady backwards incompatible.
The only example I can think of is the type hint for buffer (pybind/pybind11#5662) which uses collections.abc.Buffer since Python 3.12 and typing_extensions.Buffer before.

The reason that pinning a specific Python version to produce a different syntax stub does not work is that it would require building version-specific wheels of a stub package for the end user.

The user only needs to build stubs with the oldest python version they support. If they want to support the new syntax they would need to build with newer python versions.

[gentlegiantJGC]

I have fixed the code so that it does not generate the new syntax in versions prior to 3.12.
It will instead leave them as a docstring like the library currently does.

I have also added tests for 3.10 and 3.11 just to show how the library outputs it.
I wouldn't exactly say these outputs are correct but they are the most correct stubs that can be generated from invalid docstrings.

I also added an example to show how all versions can be correctly supported by the library author.