dmypy crash on followup check of file importing multiprocessing

[lwfitzgerald]

Crash Report

The dmypy daemon crashes if a check is performed on a file importing multiprocessing.util after an earlier check of any other set of files.

Traceback (most recent call last):
  File "/Users/lwfitzgerald/mypy/mypy/dmypy_server.py", line 229, in serve
    resp = self.run_command(command, data)
  File "/Users/lwfitzgerald/mypy/mypy/dmypy_server.py", line 272, in run_command
    return method(self, **data)
  File "/Users/lwfitzgerald/mypy/mypy/dmypy_server.py", line 331, in cmd_run
    return self.check(sources, is_tty, terminal_width)
  File "/Users/lwfitzgerald/mypy/mypy/dmypy_server.py", line 393, in check
    messages = self.fine_grained_increment_follow_imports(sources)
  File "/Users/lwfitzgerald/mypy/mypy/dmypy_server.py", line 568, in fine_grained_increment_follow_imports
    messages = fine_grained_manager.update(changed, [])
  File "/Users/lwfitzgerald/mypy/mypy/server/update.py", line 245, in update
    result = self.update_one(changed_modules, initial_set, removed_set, blocking_error)
  File "/Users/lwfitzgerald/mypy/mypy/server/update.py", line 328, in update_one
    result = self.update_module(next_id, next_path, next_id in removed_set)
  File "/Users/lwfitzgerald/mypy/mypy/server/update.py", line 414, in update_module
    remaining += propagate_changes_using_dependencies(
  File "/Users/lwfitzgerald/mypy/mypy/server/update.py", line 808, in propagate_changes_using_dependencies
    raise RuntimeError('Max number of iterations (%d) reached (endless loop?)' % MAX_ITER)
RuntimeError: Max number of iterations (1000) reached (endless loop?)

If a check of a file importing of multiprocessing.util is the first check run using the daemon, the daemon does not crash and correctly errors that the util module is not present in the multiprocessing typeshed stubs:

Daemon started
b.py:1: error: Cannot find implementation or library stub for module named "multiprocessing.util"
b.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports

To Reproduce

# Create the files
touch a.py
echo "import multiprocessing.util" > b.py

# Clean up any old daemon
dmypy kill || true

# Perform the first check
dmypy run -- --show-traceback a.py

# Perform the second check on a file containing "import multiprocessing.util"
dmypy run -- --show-traceback b.py

Your Environment

Dependency versions:

mypy-extensions==0.4.3
tomli==2.0.0
typing_extensions==4.0.1

• Mypy version used: mypy 0.940+dev.63abf4b5b26edb2c7db2a9ff50dfa0e63a43434a. Also reproduced on 0.920.
• Mypy command-line flags: --show-traceback
• Mypy configuration options from mypy.ini (and other config files): None
• Python version used: 3.9.8
• Operating system and version: MacOS 11.6 (Big Sur)

[lwfitzgerald]

Having looked at this again, it's actually worse. The crash appears to happen simply with an import of multiprocessing:

# Create the files
touch a.py
echo "import multiprocessing" > b.py

# Clean up any old daemon
dmypy kill || true

# Perform the first check
dmypy run -- --show-traceback a.py

# Perform the second check on a file containing "import multiprocessing"
dmypy run -- --show-traceback b.py

Daemon crashed!
Traceback (most recent call last):
  File "/Users/lwfitzgerald/mypy/mypy/dmypy_server.py", line 229, in serve
    resp = self.run_command(command, data)
  File "/Users/lwfitzgerald/mypy/mypy/dmypy_server.py", line 272, in run_command
    return method(self, **data)
  File "/Users/lwfitzgerald/mypy/mypy/dmypy_server.py", line 331, in cmd_run
    return self.check(sources, is_tty, terminal_width)
  File "/Users/lwfitzgerald/mypy/mypy/dmypy_server.py", line 393, in check
    messages = self.fine_grained_increment_follow_imports(sources)
  File "/Users/lwfitzgerald/mypy/mypy/dmypy_server.py", line 568, in fine_grained_increment_follow_imports
    messages = fine_grained_manager.update(changed, [])
  File "/Users/lwfitzgerald/mypy/mypy/server/update.py", line 245, in update
    result = self.update_one(changed_modules, initial_set, removed_set, blocking_error)
  File "/Users/lwfitzgerald/mypy/mypy/server/update.py", line 328, in update_one
    result = self.update_module(next_id, next_path, next_id in removed_set)
  File "/Users/lwfitzgerald/mypy/mypy/server/update.py", line 414, in update_module
    remaining += propagate_changes_using_dependencies(
  File "/Users/lwfitzgerald/mypy/mypy/server/update.py", line 808, in propagate_changes_using_dependencies
    raise RuntimeError('Max number of iterations (%d) reached (endless loop?)' % MAX_ITER)
RuntimeError: Max number of iterations (1000) reached (endless loop?)

I will update the issue title accordingly.

[JukkaL]

I can reproduce the issue. This seems to have been around for some time, at least from 0.900.

[kjagiello]

I've just run into this issue and did some initial debugging. Mypy seems to get stuck inside of propagate_changes_using_dependencies on the following nodes:

LOG:  fine-grained: triggered: ['<multiprocessing.Value>', '<multiprocessing.RawArray>', '<multiprocessing.RawValue>', '<multiprocessing.Array>']
LOG:  fine-grained: triggered: ['<multiprocessing.Value>', '<multiprocessing.RawArray>', '<multiprocessing.RawValue>', '<multiprocessing.Array>']
LOG:  fine-grained: triggered: ['<multiprocessing.Value>', '<multiprocessing.RawArray>', '<multiprocessing.RawValue>', '<multiprocessing.Array>']
LOG:  fine-grained: triggered: ['<multiprocessing.Value>', '<multiprocessing.RawArray>', '<multiprocessing.RawValue>', '<multiprocessing.Array>']
...

Setting a breakpoint within compare_symbol_table_snapshots reveals the following:

> /Users/kjagiello/Dev/private/mypy/mypy/server/astdiff.py(121)compare_symbol_table_snapshots()
-> triggers.add(item_name)
(Pdb) item_name
'multiprocessing.Array'
(Pdb) snapshot1[name]
('Var', ('multiprocessing.Array', 1, True), ('Overloaded', (('CallableType', (('TypeType', ('TypeVar', '_CT', 'multiprocessing.context._CT', 107, 0, (), ('Instance', 'ctypes._CData', (), ('None',)), 0)), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('LiteralType', ('Instance', 'builtins.bool', (), ('None',)), False)), ('TypeVar', '_CT', 'multiprocessing.context._CT', 107, 0, (), ('Instance', 'ctypes._CData', (), ('None',)), 0), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED: 3>), False, False), ('CallableType', (('TypeType', ('TypeVar', '_CT', 'multiprocessing.context._CT', 108, 0, (), ('Instance', 'ctypes._CData', (), ('None',)), 0)), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('UnionType', (('AnyType',), ('LiteralType', ('Instance', 'builtins.bool', (), ('None',)), True)))), ('AnyType',), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED: 3>), False, False), ('CallableType', (('Instance', 'builtins.str', (), ('None',)), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('UnionType', (('LiteralType', ('Instance', 'builtins.bool', (), ('None',)), True), ('TypeAliasType', 'multiprocessing.context._LockLike', ())))), ('AnyType',), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED: 3>), False, False), ('CallableType', (('UnionType', (('Instance', 'builtins.str', (), ('None',)), ('TypeType', ('Instance', 'ctypes._CData', (), ('None',))))), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('UnionType', (('Instance', 'builtins.bool', (), ('None',)), ('TypeAliasType', 'multiprocessing.context._LockLike', ())))), ('AnyType',), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED_OPT: 5>), False, False))), False)
(Pdb) snapshot2[name]
('Var', ('multiprocessing.Array', 1, True), ('Overloaded', (('CallableType', (('TypeType', ('TypeVar', '_CT', 'multiprocessing.context._CT', 113, 0, (), ('Instance', 'ctypes._CData', (), ('None',)), 0)), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('LiteralType', ('Instance', 'builtins.bool', (), ('None',)), False)), ('TypeVar', '_CT', 'multiprocessing.context._CT', 113, 0, (), ('Instance', 'ctypes._CData', (), ('None',)), 0), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED: 3>), False, False), ('CallableType', (('TypeType', ('TypeVar', '_CT', 'multiprocessing.context._CT', 114, 0, (), ('Instance', 'ctypes._CData', (), ('None',)), 0)), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('UnionType', (('AnyType',), ('LiteralType', ('Instance', 'builtins.bool', (), ('None',)), True)))), ('AnyType',), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED: 3>), False, False), ('CallableType', (('Instance', 'builtins.str', (), ('None',)), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('UnionType', (('LiteralType', ('Instance', 'builtins.bool', (), ('None',)), True), ('TypeAliasType', 'multiprocessing.context._LockLike', ())))), ('AnyType',), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED: 3>), False, False), ('CallableType', (('UnionType', (('Instance', 'builtins.str', (), ('None',)), ('TypeType', ('Instance', 'ctypes._CData', (), ('None',))))), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('UnionType', (('Instance', 'builtins.bool', (), ('None',)), ('TypeAliasType', 'multiprocessing.context._LockLike', ())))), ('AnyType',), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED_OPT: 5>), False, False))), False)

... next occurence:

-> triggers.add(item_name)
(Pdb) item_name
'multiprocessing.Array'
(Pdb) snapshot1[name]
('Var', ('multiprocessing.Array', 1, True), ('Overloaded', (('CallableType', (('TypeType', ('TypeVar', '_CT', 'multiprocessing.context._CT', 113, 0, (), ('Instance', 'ctypes._CData', (), ('None',)), 0)), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('LiteralType', ('Instance', 'builtins.bool', (), ('None',)), False)), ('TypeVar', '_CT', 'multiprocessing.context._CT', 113, 0, (), ('Instance', 'ctypes._CData', (), ('None',)), 0), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED: 3>), False, False), ('CallableType', (('TypeType', ('TypeVar', '_CT', 'multiprocessing.context._CT', 114, 0, (), ('Instance', 'ctypes._CData', (), ('None',)), 0)), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('UnionType', (('AnyType',), ('LiteralType', ('Instance', 'builtins.bool', (), ('None',)), True)))), ('AnyType',), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED: 3>), False, False), ('CallableType', (('Instance', 'builtins.str', (), ('None',)), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('UnionType', (('LiteralType', ('Instance', 'builtins.bool', (), ('None',)), True), ('TypeAliasType', 'multiprocessing.context._LockLike', ())))), ('AnyType',), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED: 3>), False, False), ('CallableType', (('UnionType', (('Instance', 'builtins.str', (), ('None',)), ('TypeType', ('Instance', 'ctypes._CData', (), ('None',))))), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('UnionType', (('Instance', 'builtins.bool', (), ('None',)), ('TypeAliasType', 'multiprocessing.context._LockLike', ())))), ('AnyType',), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED_OPT: 5>), False, False))), False)
(Pdb) snapshot2[name]
('Var', ('multiprocessing.Array', 1, True), ('Overloaded', (('CallableType', (('TypeType', ('TypeVar', '_CT', 'multiprocessing.context._CT', 119, 0, (), ('Instance', 'ctypes._CData', (), ('None',)), 0)), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('LiteralType', ('Instance', 'builtins.bool', (), ('None',)), False)), ('TypeVar', '_CT', 'multiprocessing.context._CT', 119, 0, (), ('Instance', 'ctypes._CData', (), ('None',)), 0), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED: 3>), False, False), ('CallableType', (('TypeType', ('TypeVar', '_CT', 'multiprocessing.context._CT', 120, 0, (), ('Instance', 'ctypes._CData', (), ('None',)), 0)), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('UnionType', (('AnyType',), ('LiteralType', ('Instance', 'builtins.bool', (), ('None',)), True)))), ('AnyType',), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED: 3>), False, False), ('CallableType', (('Instance', 'builtins.str', (), ('None',)), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('UnionType', (('LiteralType', ('Instance', 'builtins.bool', (), ('None',)), True), ('TypeAliasType', 'multiprocessing.context._LockLike', ())))), ('AnyType',), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED: 3>), False, False), ('CallableType', (('UnionType', (('Instance', 'builtins.str', (), ('None',)), ('TypeType', ('Instance', 'ctypes._CData', (), ('None',))))), ('UnionType', (('Instance', 'builtins.int', (), ('None',)), ('Instance', 'typing.Sequence', (('AnyType',),), ('None',)))), ('UnionType', (('Instance', 'builtins.bool', (), ('None',)), ('TypeAliasType', 'multiprocessing.context._LockLike', ())))), ('AnyType',), ('typecode_or_type', 'size_or_initializer', 'lock'), (<ArgKind.ARG_POS: 0>, <ArgKind.ARG_POS: 0>, <ArgKind.ARG_NAMED_OPT: 5>), False, False))), False)

As you can see the ID of the multiprocessing.context._CT TypeVar seems to always differ between the snapshots.

[ilevkivskyi]

A self-contained test case:

[case testBoundGenericMethodFine]
import lib
[file lib/__init__.pyi]
from lib import context
foo = context.test.foo
[file lib/context.pyi]
from typing import TypeVar
import lib.other

T = TypeVar("T")
class Test:
    def foo(self, x: T, n: lib.other.C) -> T: ...
test: Test

[file lib/other.pyi]
class C: ...
[file lib/other.pyi.2]
class B: ...
class C(B): ...
[out]
==

Unfortunately, this is a fundamental problem. Each time we look up a method, we call freshen_function_type_vars() on it, because otherwise there may be type variable clashes. Therefore, there may be situations where we never reach a fixed point.
I think our type variable identity system needs to be seriously cleaned up/re-implemented. There is a meta issue on this #4814, and it may be a quite large project (and requires good understanding of mypy internals).

[ilevkivskyi]

Actually I think there may be a simple short term solution for this. We can "anonimize" generic callables before comparisons. Anyway, their type variables are purely internal, and can leak only if there is a bug (like famous #1317)