Type Protobuf Requirement

[hmc-cs-mdrissi]

Summary

This is mainly a question. Why does mypy-protobuf dependent on types-protobuf, https://github.com/dropbox/mypy-protobuf/blob/main/setup.cfg#L19? Is there any need for types-protobuf to use mypy-protobuf to generate type stubs from .proto files? If not while the recommendation is reasonable, can it not be required? Current version of mypy I think already will warn users about missing type stub libraries.

My reason for wanting to avoid types-protobuf is that currently it interacts with other google cloud libraries in a buggy manner. The issue for that is here, python/typeshed#5800. When I install mypy-protobuf currently it brings in types-protobuf and then the google.cloud imports start to have errors. Until that issue is fixed currently best way to type check my codebase is install mypy-protobuf, use it to generate stubs for .proto files, and uninstall types-protobuf.

I can manually pip uninstall types-protobuf, but I'm avoidant to do that as it means I can't have my library depend on mypy-protobuf in developer requirements without making other developers also need to pip uninstall it and then having pip be a bit unhappy I'm ignoring requirements. Needing non-standard way to install library to hack around this is fairly inconvenient for new users to the library.

edit: Looking at older versions types-protobuf looks like a recent requirement, so an alternative is just locking mypy-protobuf to a slightly older version until google-cloud/types-protobuf are more compatible. I'm fine if the decision is to do nothing as root cause of this issue is really types-protobuf library and not mypy-protobuf.

[nipunn1313]

I can look into this in more detail, but one recommendation for short term is to run mypy-protobuf in a separate venv Thanks for the report! It’s appreciated —Nipunn

…

On Sat, Aug 28, 2021 at 11:15 PM Mehdi Drissi ***@***.***> wrote: Summary This is mainly a question. Why does mypy-protobuf dependent on types-protobuf, https://github.com/dropbox/mypy-protobuf/blob/main/setup.cfg#L19? Is there any need for types-protobuf to use mypy-protobuf to generate type stubs from .proto files? If not while the recommendation is reasonable, can it not be required? Current version of mypy I think already will warn users about missing type stub libraries. My reason for wanting to avoid types-protobuf is that currently it interacts with other google cloud libraries in a buggy manner. The issue for that is here, python/typeshed#5800 <python/typeshed#5800>. When I install mypy-protobuf currently it brings in types-protobuf and then the google.cloud imports start to have errors. Until that issue is fixed currently best way to type check my codebase is install mypy-protobuf, use it to generate stubs for .proto files, and uninstall types-protobuf. I can manually pip uninstall types-protobuf, but I'm avoidant to do that as it means I can't have my library depend on mypy-protobuf in developer requirements without making other developers also need to pip uninstall it and then having pip be a bit unhappy I'm ignoring requirements. Needing non-standard way to install library to hack around this is fairly inconvenient for new users to the library. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#283>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ5PI3GNVMQV3VC34RUDXTT7HGAPANCNFSM5C74UHEA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nipunn1313]

I think we can do something similar to #271 to remove this dependency to help make it easier to use without the separate venv.

Types-protobuf must be installed for mypy to run properly unfortunately, but not for mypy-protobuf to run.

it may be even more confusing to new users to have to install types-protobuf manually. Have to look more into the specifics of the bug you mentioned when back at computer

[hmc-cs-mdrissi]

My way of handling mypy issue with types-protobuf is adding,

[mypy-google.protobuf.*]
ignore_missing_imports = True

to my repo's mypy config. Undesirable yes, but import errors of other google.cloud libraries on pyright is also undesirable so I have to make a trade off.

[nipunn1313]

Cool. So I am able to repro this. I am almost certain this is a typeshed issue - as the google.protobuf stubs in typeshed are incomplete (per python/typeshed#5800).

I'm hesitant to remove the dependency on types-protobuf, because the generated stubs actually need them for mypy to work. For example, they all routinely import from google.protobuf.descriptor. Technically it's not required for mypy-protobuf to run, but it is required to run mypy on the generated output.

I'll look into proper solutions to python/typeshed#5800, but as a bandaid, we might be able to stuff some stubs for google-cloud-core in typeshed

[nipunn1313]

I poked at this further and did testing with both mypy and pyright
Check out python/typeshed#5800 (comment)

I don't think there's a way to have both types-protobuf and google.cloud installed simultaneously and get types for both. Unfortunately, types-protobuf is required to make use of generated code. Hoping to drive toward a solution to this issue over there, as I believe the issue bug is somewhere in the land of typeshed/mypy/pyright.

In the meantime, if you use google.cloud - I recommend a workaround of uninstalling types-protobuf and dealing with the subsequent errors w/ the workaround described by #283 (comment). You'll lose a bit of typing on your protobufs in exchange for gaining typing in google.cloud.

I will leave this as-is to support developers using protobuf w/o using google.cloud.

I truly apologize that things haven't been working recently in tandem with these libraries. I believe it's a regression associated with modular typeshed and hope it can be resolved. Growing pains are expected. Modular typeshed is truly amazing in many ways, but doesn't seem to have accounted for this case.

For now follow along in python/typeshed#5800

[nipunn1313]

For context - read https://mypy-lang.blogspot.com/2021/05/the-upcoming-switch-to-modular-typeshed.html

Recently, typeshed switched to generating many stub packages - rather than being a monolithic home for stubs hardcoded into type checkers.

Appears unresolved how to handle namespace packages (eg google) where there are multiple pip packages within the same namespace package (eg google.protobuf and google.cloud.firestore)

[hmc-cs-mdrissi]

Thank you for the investigation and decision is reasonable. It’s always a challenge when a bug in your library is closely linked to an upstream issue.

mypy protobuf has been great simple to use library and a definite value add for my team. Thanks for all the work on it.