aioitertools.zip is doing strange things with exceptions

[filip-zyzniewski]

Description

What happens:

$ python3 -m venv aioitertools_zip_exception
$ cat aioitertools_zip_exception.py 
import asyncio
import platform
import sys

import aioitertools


async def gen():
    for i in range(10):
        if i == 9:
            assert False
        await asyncio.sleep(1)
        yield i


async def gen_pairs():
    for i in range(10):
        if i == 9:
            assert False
        await asyncio.sleep(1)
        yield i, "a"


async def main():
    print("platform version", platform.version())
    print("python version", sys.version)
    print("aioitertools version", aioitertools.__version__)

    print("the exception is getting swallowed:")
    async for (i, j) in aioitertools.zip(gen(), gen()):
        print(i, j)

    print("this attempts to unpack the exception:")
    async for (i, j), (k, l) in aioitertools.zip(gen_pairs(), gen_pairs()):
        print(i, j, k, l)


asyncio.run(main())
$ . ./aioitertools_zip_exception/bin/activate
(aioitertools_zip_exception) $ python3 -m pip install aioitertools
Collecting aioitertools
  Downloading aioitertools-0.11.0-py3-none-any.whl (23 kB)
Installing collected packages: aioitertools
Successfully installed aioitertools-0.11.0
(aioitertools_zip_exception) $ python3 aioitertools_zip_exception.py 
platform version Darwin Kernel Version 22.6.0: Wed Jul  5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000
python version 3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)]
aioitertools version 0.11.0
the exception is getting swallowed:
0 0
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
 
this attempts to unpack the exception:
0 a 0 a
1 a 1 a
2 a 2 a
3 a 3 a
4 a 4 a
5 a 5 a
6 a 6 a
7 a 7 a
8 a 8 a
Traceback (most recent call last):
  File "/Users/Filip.Zyzniewski/aioitertools_zip_exception.py", line 38, in <module>
    asyncio.run(main())
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/Filip.Zyzniewski/aioitertools_zip_exception.py", line 34, in main
    async for (i, j), (k, l) in aioitertools.zip(gen_pairs(), gen_pairs()):
              ^^^^^^
TypeError: cannot unpack non-iterable AssertionError object
(aioitertools_zip_exception) $ 

What I expected to happen: I expected the exception to percolate up the call stack and be raised at the top level.

Details

• OS: Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000
• Python version: 3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)]
• aioitertools version: 0.11.0
• Can you repro on master? yes: https://gist.github.com/filip-zyzniewski/1bb9c2dbde5da1dfeea5154f6e8a81f7
• Can you repro in a clean virtualenv? done above

[filip-zyzniewski]

It seems like this behavior got introduced in 835bad7 .

[amyreese]

@filip-zyzniewski can you confirm if #157 matches your expected behavior?

[filip-zyzniewski]

@amyreese not exactly, please see this:

$ pwd
/Users/Filip.Zyzniewski/git/github.com/amyreese/aioitertools
$ git status
On branch zip-exception
Your branch is up to date with 'origin/zip-exception'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	aioitertools_zip_exceptions.py

nothing added to commit but untracked files present (use "git add" to track)
$ git rev-parse HEAD                
6285944f42395ffdca96aa388d2db1e4f545c7bb
$ cat aioitertools_zip_exceptions.py    
import asyncio
import platform
import sys

import aioitertools

def gen():
    for i in range(2):
        yield ValueError(f"ValueError({i})")

async def agen():
    for i in range(2):
        yield ValueError(f"ValueError({i})")


async def main():
    print("platform version", platform.version())
    print("python version", sys.version)
    print("aioitertools:", aioitertools)
    print("aioitertools version", aioitertools.__version__)

    print("zip builtin behaves as expected:")
    for (i, j) in zip(range(10), gen()):
        print(i, j)

    print("aioitertools.zip raises an exception passed as a value:")
    async for (i, j) in aioitertools.zip(range(2), agen()):
        print(i, j)


asyncio.run(main())
$ python3 aioitertools_zip_exceptions.py
platform version Darwin Kernel Version 23.1.0: Mon Oct  9 21:27:24 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T6000
python version 3.11.7 (main, Dec  4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)]
aioitertools: <module 'aioitertools' from '/Users/Filip.Zyzniewski/dev/git/github.com/amyreese/aioitertools/aioitertools/__init__.py'>
aioitertools version 0.11.0
zip builtin behaves as expected:
0 ValueError(0)
1 ValueError(1)
aioitertools.zip raises an exception passed as a value:
Traceback (most recent call last):
  File "/Users/Filip.Zyzniewski/dev/git/github.com/amyreese/aioitertools/aioitertools_zip_exceptions.py", line 31, in <module>
    asyncio.run(main())
  File "/opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/Filip.Zyzniewski/dev/git/github.com/amyreese/aioitertools/aioitertools_zip_exceptions.py", line 27, in main
    async for (i, j) in aioitertools.zip(range(2), agen()):
  File "/Users/Filip.Zyzniewski/dev/git/github.com/amyreese/aioitertools/aioitertools/builtins.py", line 441, in zip
    raise v
ValueError: ValueError(0)
$

[4l1fe]

It seems i have the same issue of zip not raising an exception, but returning it, i guess due to the parameter return_exceptions=True in the implematation code.

The code i'm using zip in looks like

    sublists_data_chain: list[tuple[int, str, dict]] = (
        aioitertools.chain(
            aioitertools.zip(
                aioitertools.count(1),
                aioitertools.repeat(subentity.value),
                api_client.irequest_sublist_pages(config.account_id, subentity, entity_id))
            for subentity in subentities)
    )

    async for step, entity_name, data in sublists_data_chain:
        do_some_stuff(step, entity_name, data)

I'm making a rich iterator extending requesting data with variables for furher chaining all the responses from a resource api and would like an exception to be raised ASAP while iterating the defined chain with ioitertools, but zip as @filip-zyzniewski said swallows an expection.

[tabrezm]

@filip-zyzniewski can you confirm if #157 matches your expected behavior?

I believe this would also need to be patched for zip_longest too.