Demo: Testing pybind11-stubgen + mypy

[timohl]

Description

This PR shows how pybind11-stubgen and mypy could be used in pytest to check for correct type hints.
It stems from the discussion at #5663.

This is just a quick demo with one positive and one negative test.

Sources for calling pybind11-stubgen and mypy from Python:

• Integrating mypy into another Python application
• Calling pybind11-stubgen from Python:
  Add args to main sizmailov/pybind11-stubgen#252
  (thx @gentlegiantJGC for this 👍)

TODOs:

• Integrate test cases from Experimenting: Annotated[Any, pybind11.CppType("cpp_namespace::UserType")] #4888
• Maybe use this to generate stubs for all test modules (just to check if pybind11-stubgen and mypy run successful or find errors)
• Pin versions of mypy and pybind11-stubgen (or even use a version matrix in CI).
• Consider other type checkers (most notably Pyright/Pylance since this is the default in vscode).

---

📚 Documentation preview 📚: https://pybind11--5678.org.readthedocs.build/

[timohl]

Some errors I see so far:

• Python 3.14 seems like causing some mypy internal errors
• Some checks do not properly install pybind11-stubgen, although I added it to requirements.txt and pyproject.toml
• mypy report on https://github.com/pybind/pybind11/actions/runs/15149485803/job/42592785722?pr=5678 seems scrambled (maybe some formatting or windows specific stuff)

[rwgk]

The new test dependencies seem "small/lightweight" and "super lightweight":

$ time pip install mypy
Collecting mypy
  Using cached mypy-1.15.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl.metadata (2.1 kB)
Requirement already satisfied: typing_extensions>=4.6.0 in ./python/venvs/scratch/cpwheels/lib/python3.12/site-packages (from mypy) (4.13.2)
Requirement already satisfied: mypy_extensions>=1.0.0 in ./python/venvs/scratch/cpwheels/lib/python3.12/site-packages (from mypy) (1.1.0)
Using cached mypy-1.15.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl (12.4 MB)
Installing collected packages: mypy
Successfully installed mypy-1.15.0

real    0m1.098s
user    0m0.936s
sys     0m0.150s

$ time pip install pybind11-stubgen
Collecting pybind11-stubgen
  Using cached pybind11_stubgen-2.5.4-py3-none-any.whl.metadata (1.9 kB)
Using cached pybind11_stubgen-2.5.4-py3-none-any.whl (30 kB)
Installing collected packages: pybind11-stubgen
Successfully installed pybind11-stubgen-2.5.4

real    0m0.418s
user    0m0.242s
sys     0m0.048s

I think you're going in a great direction.

Maybe use this to generate stubs for all test modules (just to check if pybind11-stubgen and mypy run successful or find errors)

That sounds like a great idea, as long as you build in an easy way to turn that off during development iterations. I'd only want to turn that on again while I clean up for the next git push.

If you do that, I'd say forget the tests under #4888 (those came mostly from mypy). I believe they'll be 100% redundant.

Pyright/Pylance since this is the default in vscode

I'd look at that is the top priority. — I don't know a lot about typing, but last time I quizzed ChatGPT about it (a few months ago), it told me mypy is a good start, but — liberally reinterpreted by me — what really counts is LSP.

[henryiii]

If you haven't started using uv, you should try it, even if it's just uv venv and uv pip:

$ uv venv
Using CPython 3.13.3 interpreter at: /usr/local/opt/[email protected]/bin/python3.13
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate.fish

________________________________________________________
Executed in   28.56 millis    fish           external
   usr time   12.44 millis    0.25 millis   12.19 millis
   sys time   13.04 millis    1.91 millis   11.13 millis

$ time uv pip install mypy
Resolved 3 packages in 3ms
Installed 3 packages in 50ms
 + mypy==1.15.0
 + mypy-extensions==1.1.0
 + typing-extensions==4.13.2

________________________________________________________
Executed in   81.80 millis    fish           external
   usr time   18.27 millis    0.29 millis   17.98 millis
   sys time   54.27 millis    3.55 millis   50.73 millis

I think ty and pyrefly, the two new Python type checkers in Rust, are likely to have a big impact in the future, but for now, most projects work on passing mypy, and the existing type checkers like pylance know that and try to be compatible. I don't think I use anything other than mypy in CI on all my projects, though I use pyright locally in VSCode and via LSP in Vim.

The bug in Python 3.14.0b1 that we've reported also affects mypy, and it's been marked as a release blocker. So mypy will likely work better in beta 2.

[timohl]

I fixed most of the CI issues:

• Added xfail for Python 3.14
• Using --no-color-output fixes parsing the mypy report correctly
• I have changed CI at a couple of places to use tests/requirements.txt instead of manually installing pytest, numpy and so on. Since I added pybind11-stubgen and mypy there, the tests now run successfully (except for mingw because I am not sure how to change this the right was)

@henryiii
The CI is uses a mix of uv, pip, pipx, apt and the github actions setup-python and setup-uv.
Would it make sense to have this more consistently use uv, setup-python and setup-uv where possible (mingw seems to be an exception)?
Also is there a downside to using venv in CI checks?
In https://github.com/pybind/pybind11/actions/runs/15182964725/job/42696579100 uv complained about the system-wide environment being externally managed (while almost all other jobs did not complain).
Adding activate-environment: true to the setup-uv action solved this.

@rwgk

Maybe use this to generate stubs for all test modules (just to check if pybind11-stubgen and mypy run successful or find errors)

That sounds like a great idea, as long as you build in an easy way to turn that off during development iterations. I'd only want to turn that on again while I clean up for the next git push.

Probably best to use pytest markers for that (which could be excluded by default).
Custom markers (pytest docs)
Exclude custom marker by default (stackoverflow)

[timohl]

I have added a test that applies stubgen and mypy to the whole pybind11_tests module.

This is the stubgen error log:

error log

ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_nested : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_pair : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_tuple : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.nodefer_none_void : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_nested : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_pair : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_tuple : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ptr : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ref : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ref_wrap : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_move : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ptr : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ref : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ref_wrap : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks : Can't find/import 'pybind11_tests.callbacks.custom_function' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks : Can't find/import 'pybind11_tests.callbacks.custom_function2' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks.func_accepting_func_accepting_base : Invalid expression 'test_submodule_callbacks(module_&)::AbstractBase' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.ProtectedB.get_self : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.ProtectedB.void_foo : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_ : Invalid identifier '' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.get_mixed_gl : Invalid expression 'LocalBase<5>' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.get_mixed_lg : Invalid expression 'LocalBase<4>' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.load_external1 : Invalid expression 'LocalBase<6>' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.load_external2 : Invalid expression 'LocalBase<7>' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.methods_and_attributes.exercise_is_setter.Field.int_value. : Invalid expression 'pybind11_tests::exercise_is_setter::FieldBase' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.native_enum.pass_some_proto_enum : Invalid expression 'test_native_enum::some_proto_enum' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.native_enum.return_some_proto_enum : Invalid expression 'test_native_enum::some_proto_enum' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_array.index_using_ellipsis : Invalid expression 'detail::accessor' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_array_array : Can't find/import 'ArrayStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_complex_array : Can't find/import 'ComplexStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_enum_array : Can't find/import 'EnumStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_nested : Can't find/import 'NestedStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_partial : Can't find/import 'PartialStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_partial_nested : Can't find/import 'PartialNestedStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_string_array : Can't find/import 'StringStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.f_nested : Can't find/import 'NestedStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_array_array : Can't find/import 'ArrayStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_complex_array : Can't find/import 'ComplexStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_enum_array : Can't find/import 'EnumStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_rec_nested : Can't find/import 'NestedStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_string_array : Can't find/import 'StringStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.opaque_types.get_void_ptr_value : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.opaque_types.return_void_ptr : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.annotate_never : Can't find/import 'typing.Never' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.annotate_type_is : Can't find/import 'typing.TypeIs' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.check_type_is : Can't find/import 'typing.TypeIs' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor_2 : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor_3 : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_explicit_nullptr_dtor : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_name_and_destructor : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_renamed_capsule_with_destructor : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_renamed_capsule_with_destructor_2 : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.test_list_slicing : Invalid expression 'detail::accessor' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.test_list_slicing_default : Invalid expression 'detail::accessor' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_handle : Can't find/import 'weakref' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_handle_and_function : Can't find/import 'weakref' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_object : Can't find/import 'weakref' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_object_and_function : Can't find/import 'weakref' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.ContainerUsingPrivateESFT.ptr. : Invalid expression 'test_submodule_smart_ptr(module_&)::PrivateESFT' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.ContainerUsingPrivateESFT.ptr. : Invalid expression 'test_submodule_smart_ptr(module_&)::PrivateESFT' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.return_std_shared_ptr_example_drvd : Invalid expression 'holder_caster_traits_test::example_drvd' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.return_std_unique_ptr_example_drvd : Invalid expression 'holder_caster_traits_test::example_drvd' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_array : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_map : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_nested : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_vector : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_ptr_vector : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_array : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_map : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_nested : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_vector : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression 'std::vector' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression ':allocator >' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression ':allocator > > >' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_constr_optional : Can't find/import 'TplCtorClass' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_map : Can't find/import 'TplCtorClass' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_set : Can't find/import 'TplCtorClass' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_vector : Can't find/import 'TplCtorClass' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stubgen_error.identity_capsule : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.virtual_functions.Adder.__call__ : Invalid expression 'AdderBase::Data' WARNING pybind11_stubgen:error_handlers.py:150 Raw C++ types/values were found in signatures extracted from docstrings. Please check the corresponding sections of pybind11 documentation to avoid common mistakes in binding code: - https://pybind11.readthedocs.io/en/latest/advanced/misc.html#avoiding-cpp-types-in-docstrings - https://pybind11.readthedocs.io/en/latest/advanced/functions.html#default-arguments-revisited

This is the mypy report:

/tmp/pytest-of-timohl/pytest-5/test_stubgen_all0/pybind11_tests/stl.pyi:187: error: Ellipses cannot accompany other argument types in function type signature  [syntax]
/tmp/pytest-of-timohl/pytest-5/test_stubgen_all0/pybind11_tests/stl.pyi:187: error: Duplicate argument "std" in function definition
Found 2 errors in 1 file (errors prevented further checking)

I have extracted __doc__ of the function mentioned by mypy to see, what it actually contains:

func_with_string_or_vector_string_arg_overload(*args, **kwargs)
Overloaded function.

1. func_with_string_or_vector_string_arg_overload(arg0: std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) -> int

2. func_with_string_or_vector_string_arg_overload(arg0: collections.abc.Sequence[str]) -> int

3. func_with_string_or_vector_string_arg_overload(arg0: str) -> int

The function is bound here:

pybind11/tests/test_stl.cpp

Lines 577 to 582 in 98bd78f

	// #1258: pybind11/stl.h converts string to vector<string>
	m.def("func_with_string_or_vector_string_arg_overload",
	[](const std::vector<std::string> &) { return 1; });
	m.def("func_with_string_or_vector_string_arg_overload",
	[](const std::list<std::string> &) { return 2; });
	m.def("func_with_string_or_vector_string_arg_overload", [](const std::string &) { return 3; });

Not sure what is going on here.
I would have assumed that the list_caster would give collections.abc.Sequence:

pybind11/include/pybind11/stl.h

Lines 360 to 366 in 98bd78f

	PYBIND11_TYPE_CASTER(Type,
	io_name("collections.abc.Sequence", "list") + const_name("[")
	+ value_conv::name + const_name("]"));
	};
	template <typename Type, typename Alloc>
	struct type_caster<std::vector<Type, Alloc>> : list_caster<std::vector<Type, Alloc>, Type> {};

I will go through the other errors tomorrow.