Structural subtype XX of X not recognized as subclass of X in class Z(Generic[TypeVar("XT", bound=X)]) and class ZZ(Z[XX])

[jmehnle]

Bug Report

It seems that whereas Python's structural subtyping works (using Python 3.7 and typing_extensions.Protocol) at runtime (see below), mypy doesn't recognize the subtype relationship in the following scenario:

1. There is a protocol class pep249.Connection (I'm creating this).
2. There is a class pymysql.connections.Connection that conforms to this protocol, as evidenced by issubclass(pymysql.connections.Connection, pep249.Connection) == True.
3. When defining a generic base class DatabaseAdapter that's generic in a ConnectionT type variable that has pep249.Connection as an upper bound (DatabaseAdapter(Generic[ConnectionT])), and defining a subclass MySQLAdapter
   that parameterizes the generic base class with pymysql.connections.Connection (MySQLAdapter(DatabaseAdapter[pymysql.connections.Connection])), mypy complains that pymysql.connections.Connection is not a subtype of pep249.Connection.

To Reproduce

https://gist.github.com/jmehnle/37ece52f9dbbec439ed8d0c11db47f41

# pep249.py

from typing import Any
from typing_extensions import Protocol, runtime_checkable

@runtime_checkable
class Connection(Protocol):
    def close(self) -> None: ...
    def commit(self) -> None: ...
    def rollback(self) -> None: ...
    def cursor(self, *__args: Any, **__kwargs: Any) -> Any: ...

# main.py

from typing import Generic, TypeVar
import pep249
import pymysql

ConnectionT = TypeVar("ConnectionT", bound=pep249.Connection)

class DatabaseAdapter(Generic[ConnectionT]): ...

class MySQLAdapter(DatabaseAdapter[pymysql.connections.Connection]): ...

Expected Behavior

pymysql.connections.Connection should be recognized as a (structural) subtype of pep249.Connection.

Actual Behavior

(py3.7-typing-protocol) jbook:~/scratch> mypy --version
mypy 0.790+dev.7273e9ab1664b59a74d9bd1d2361bbeb9864b7ab

(py3.7-typing-protocol) jbook:~/scratch> mypy .
main.py:11: error: Type argument "pymysql.connections.Connection" of "DatabaseAdapter" must be a subtype of "pep249.Connection"
Found 1 error in 1 file (checked 2 source files)

(py3.7-typing-protocol) jbook:~/scratch> python
Python 3.7.9 (default, Sep  6 2020, 13:20:25)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pep249
>>> import pymysql
>>> pymysql.connections.Connection
<class 'pymysql.connections.Connection'>
>>> issubclass(pymysql.connections.Connection, pep249.Connection)
True

Your Environment

• Mypy version used: 0.790+dev.7273e9ab1664b59a74d9bd1d2361bbeb9864b7ab (latest master)
• Mypy command-line flags: (none)
• Mypy configuration options from mypy.ini (and other config files): (none)
• Python version used: 3.7.9
• Operating system and version: macOS 10.15.6 with MacPorts

[gvanrossum]

Could this maybe be because the typeshed stubs for pymysql are incomplete?

[jmehnle]

I'm happy to investigate, but what exactly would I need to look for? Also, what about the fact that issubclass (in conjunction with typing_extensions.runtime_checkable) thinks that pymysql.connections.Connection is a subclass of pep249.Connection?

[hauntsaninja]

issubclass is actually pretty unsophisticated; it only checks presence of an attribute, whereas mypy checks a lot more stuff.

You could take a look at the pymysql stubs: https://github.com/python/typeshed/tree/master/third_party/2and3/pymysql If you can construct a self-contained repro, that would narrow down whether there's an issue with pymysql stubs or a mypy bug. I'd also confirm whether something simpler like: x: pep249.Connection = pymysql.connections.Connection(...) passes.

There are a couple other open issues for mypy to be more helpful about diagnosing why something isn't an instance of a Protocol — I think improving this is a pretty high priority.

[jmehnle]

@hauntsaninja, the advice to try a plain assignment was gold. It gets me some useful output:

main.py:7: error: Incompatible types in assignment (expression has type "pymysql.connections.Connection", variable has type "pep249.Connection")
main.py:7: note: Following member(s) of "Connection" have conflicts:
main.py:7: note:     Expected:
main.py:7: note:         def cursor(self, **Any) -> Any
main.py:7: note:     Got:
main.py:7: note:         def cursor(self, cursor: Optional[Type[Cursor]] = ...) -> Cursor

So if I change the method definition like so:

--- pep249.py	2020-10-09 01:14:44.000000000 +0000
+++ pep249.py.new	2020-10-09 01:14:48.000000000 +0000
@@ -8,4 +8,4 @@
     def close(self) -> None: ...
     def commit(self) -> None: ...
     def rollback(self) -> None: ...
-    def cursor(self, *__args: Any, **__kwargs: Any) -> Any: ...
+    def cursor(self, __cursor: Any = ...) -> Any: ...

then it all works. But this puts me in a dilemma, because PEP 249 doesn't specify the signature of Connection.cursor(), and I can't assume there's a single argument (with a default), because other, non-pymysql DBI drivers have different arguments. Based on #5876 there's no way to type an arbitrary argument list. Evidently my first attempt doesn't match the actual signature of pymysql.connections.Connection.cursor.

[gvanrossum]

Here's a horrible hack that might help: declare cursor as a property (really!) that returns a Callable:

from typing import *

class Pro(Protocol):
    @property
    def foo(self) -> Callable[..., Any]:
        ...

class C:
    def foo(self) -> None:
        pass

x: Pro = C()

No errors there.

[jmehnle]

Hot damn, that works. Thanks, @gvanrossum. I never, ever would've come up with that trick. So where does this leave us? I see at least a couple deficiencies:

• When Type argument "XX" of "ZZ" must be a subtype of "X" is triggered, mypy doesn't give the same helpful information about why XX isn't recognized as a subtype of X as it does when Incompatible types in assignment is triggered.
• There is no way of specifying a loosely defined function/method signature (here: self required, all further arguments unspecified) in a protocol class that structural subtypes can satisfy (Make any callable compatible with (*args: Any, **kwargs: Any)? #5876). The property trick basically ignores all arguments, which happens to work well enough in this case.

[hauntsaninja]

Closing, since #5876 has been fixed