Adding support for skipping jobs?

[plutaniano]

Would it be possible to add a way for a job to be skipped? I was thinking of something like this:

async def inverse(num: float) -> float:
    try:
        return 1/num
    except ZeroDivisionError as exc:
        raise aiometer.SkipException from exc


async def main():
    async with aiometer.amap(
        async_fn=inverse,
        args=range(-5, 5),
    ) as inverses:
        async for num in inverses:
            print(f"Inverse: {num}")


if __name__ == "__main__":
    asyncio.run(main())

[florimondmanca]

Hey there @plutaniano,

Would you expect a "skip" behavior to not include results for the skipped jobs in the result async iterator?

This might be tricky to implement properly while maintaining the type checking niceties we currently have.

Also it could make exception handling more complicated and we would have to handle the SkipException as a special case in our code.

For now perhaps this could be a how-to pattern resorting to a flag object?

class Skipped:
    pass

Then have your jobs return Skipped(), using str|Skipped for the type hint, and check for is instance(Skipped, value) when iterating.

[plutaniano]

Would you expect a "skip" behavior to not include results for the skipped jobs in the result async iterator?

Yes. It's a way for a job to say that it didn't produce a result, so aiometer.amap and others shouldn't include its return value in the results iterator. (technically a function that raises doesn't return anything, but that's kind of the point)

This might be tricky to implement properly while maintaining the type checking niceties we currently have.

I don't see how that would affect typing. A job's type isn't changed by what exceptions it raises.

If anything it makes typing easier for the user, because it keeps the return type of a Awaitable[T] job limited to T rather than T | Sentinel, like you suggested.

Also it could make exception handling more complicated and we would have to handle the SkipException as a special case in our code.

Yes, that's true, but imo it's a good trade off for the added functionality. Airflow does a very similar thing with AirflowSkipException