Support for PEP 561

[JoeyBG]

Hi!

This looks great. Currently, there seems to be no way to use this properly with a type checker (such as mypy) to validate code.

From their documentation (https://mypy.readthedocs.io/en/latest/installed_packages.html and the linked PEP) it seems that there is a couple options available:

1. Include a py.typed blank file in the package
2. Rename the package to boto3-stubs or something similar and generate .pyi files instead of .py

Not sure how 2) would play with IDEs and stuff...

If it's something that you would be willing to add support for, I could be interested in helping out a bit.

[alliefitter]

Thanks for the interest! I've personally never used mypy or another type checker, but my first intuition for this package was to use .pyi files. However, I wasn't able to get it working. I think it's related to the package structure of boto3_type_annotations (the sub-packages and modules) not actually existing in boto3.

I've had some success using .pyi files to type methods that are created at runtime in one of my other projects iterable_collections. In this case, though the module and class for which I'm documenting methods, collection.Collection, do actually exist. I'm just adding methods that don't exist before runtime. (Though atm there's a bug to where the stub files will only work with Python 3.6) With that in mind, I tried to take a similar approach in this package. I couldn't get it working, but I didn't invest much time in that approach because this is a side project and I knew from pyboto3 that this approach would work.

All of that said, I'd MUCH prefer to get stub files working than to have these mock classes. It's at best a workaround and at worst a hack (probably closer to a hack, but still a pretty convenient one). In a perfect world, this would just be a part of typeshed. I could probably devote some time on the weekends to experimenting with it, and if you want to give it shot, you're more than welcome to! Only contributor guideline is any new code should adhere to the PEP8 (obviously that constraint doesn't apply to the boto3 code, just the build scripts and such).

[alliefitter]

Hmmm, actually..... As a quick fix for static type checking, do you think just adding .pyi files for the existing mock classes would work? Something like this:

boto3_type_annotations/
  s3/
    client.py
    client.pyi
    paginator.py
    paginator.pyi
    service_resource.py
    service_resource.pyi
    waiter.py
    waiter.pyi

What are your thoughts on this? Like I said, I haven't ever used mypy. One caveat I can see here is that you're not actually instantiating the mock objects. Not sure how that would play with mypy.

[alliefitter]

@cowboygneox submitted pull request #9 which demonstrates how support for mypy would work. After a little bit of experimentation, it looks like this should play well with IDEs as well. Supporting mypy with the current build scripts would just take a new build configuration. It would look something like this:

services: all
with_docs: true
with_clients: true
with_service_resources: true
with_paginators: true
with_waiters: true
package_name: boto3-stubs
module_name: boto3-stubs
version: 0.3.1
readme: "../README.md"
license: "../LICENSE

That should be good enough for this too work with mypy. There do appear to be some problems with IDEs, though. After checking out the pull request and install the package, I made the following script:

import boto3
from boto3 import sts

sts_client: sts.Client = boto3.client(service_name='sts')

PyCharm recognizes that sts_client is of type sts.Client and provides autocompletion, but it does not recognize boto3.client as a function in boto3. Meaning that all of boto3's functions, sub packages, and classes will need to be parsed and added to the stub package.

Also, I just want to say that I feel pretty dumb for not looking up PEP 561 when this issue was opened. I think I just assume it was PEP 484. This all should have been done back in May.

[cowboygneox]

@alliefitter I recommend that we get your existing stubs to work as a boto3-stubs without adding support for the missing boto3 functions. We can add those next.

For now, if you want to be mypy compliant with what you've already produced, just do the following:

import boto3  # type: ignore

[cowboygneox]

And just to be super clear:

import boto3  # type: ignore
from boto3 import sts

sts_client: boto3.sts.Client = boto3.client(service_name='sts')

^ That is MyPy compliant.

[alliefitter]

Welp, @JoeyBG, @cowboygneox, @hevy-CC4477, I've got some good news for ya'll.

[afarbos]

Does this mean you won't move forward with #9 changes ?
I still think this is really valuable, it could work right now, with no need of python 3.8 and it helps people moving forward with types in python.

[cowboygneox]

I agree that we should make some decision on this. I don't see why we can't merge it and then just bail if AWS' version is better. This could be made available pretty quickly if we decide to execute.

[alliefitter]

Sorry about the long silence. Haven't had much time for side projects over the last few months. As an explanation of my reasoning for not going forward with the boto3-stubs package. PEP-561 says:

Third parties seeking to distribute stub files are encouraged to contact the maintainer of the package about distribution alongside the package. If the maintainer does not wish to maintain or package stub files or type information inline, then a third party stub-only package can be created.

In the boto3 issue I tagged above, they didn't give me permission to do so. However, someone else has forked the repo and released a boto3-stubs package on PyPI. I'm gonna officially deprecate this project and include a link to the fork.