Flaky datetime test

[Alexander-N]

The test test_datetime_check fails from time to time with an error like

---- test_datetime_check stdout ----
thread 'test_datetime_check' panicked at 'called `Result::unwrap()` on an `Err` value: PyErr { type: Py(0x60d15300, PhantomData) }', src\libcore\result.rs:1192:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

It could be that this occurs only on Windows with Python 3.6: https://github.com/PyO3/pyo3/pull/719/checks?check_run_id=384307686 and #651 (comment).

@pganssle Do you have any idea what could cause this?

[davidhewitt]

When developing pyO3 on Windows with 3.8 the test wobbles for me occasionally too.

[pganssle]

This is a real head-scratcher. I would expect possible "flakiness" for the hypothesis-based tests based on rare edge-cases, but test_datetime_check should be entirely deterministic (and not dependent on platform, either). That test just constructs a few datetime subclasses and calls the "check" functions on them, pretty dead simple.

Maybe we should start running the tests with RUST_BACKTRACE=1, so we can get a bit more info in hard-to-repro cases like this?

My immediate possible thoughts are some sort of thread-safety issue or possibly another test setting the Python exception state and returning without failing to check it (which would then raise an exception in whatever the next thing to run is that checks the exception state).

One possibility is that this is bleeding over from some issue with these out-of-bounds tests, but that shouldn't be Windows-specific.

[davidhewitt]

I had this fail on me on CI earlier today. I think it was in a Linux environment?!

https://github.com/davidhewitt/pyo3/commit/43df3299abc3cfdf006914c44d5b7f3d0a987f6c/checks?check_suite_id=414584878

[pganssle]

I had this fail on me on CI earlier today. I think it was in a Linux environment?!

I don't think so - maybe WSL or something, but a lot of the paths use \ as a separator.

I can't really make heads or tails of that backtrace. I guess what's happening is that something in _get_subclasses is breaking, but PyResult is Result<..., ()>, so ? doesn't seem to propagate anything about the inner error (not even the line it's on :().

I assume it's some sort of thread-safety issue. Maybe py.import("datetime")? isn't as thread-safe as we thought? That's the only thing in there that would obviously mutate any global variable.

Anyone with a Windows machine want to try and make this slightly more reproducible? It might help to use a Barrier in all the different test functions to try and more readily induce race conditions.

We might be able to make this less flaky by locking around _get_subclasses or around whatever subset of the test has the race condition, but it's a bit hard to confidently make a change like that without being able to reproduce the bug. Plus it would be good to know if the root cause is something to be addressed in PyO3 or something to warn people about in user code or something.

[davidhewitt]

but PyResult is Result<..., ()>

PyResult is Result<..., PyErr>, but the test has a map_err(|e| e.print(py)) which is printing and throwing away the PyErr.

Frustratingly I can't see anywhere in that output the result of e.print() - maybe Python wrote to its output buffer but did not flush?!

[davidhewitt]

Anyone with a Windows machine want to try and make this slightly more reproducible?

I'm tempted to run the test suite in a loop on my machine this morning and see how long it takes to get a failure 😅

[davidhewitt]

I ran it for a bit and added some debug printing too.

There seems to be a race going on where multiple tests are making progress at the same time. This seems very odd given that all of the tests are supposedly acquiring the GIL... ?

running 4 tests
{'timedelta': <class 'datetime.timedelta'>}
{'date': <class 'datetime.date'>}
{'datetime': <class 'datetime.datetime'>}
{'time': <class 'datetime.time'>}
{'timedelta': <class 'datetime.timedelta'>, 'Subklass': <class '__main__.Subklass'>}
{'date': <class 'datetime.date'>, 'Subklass': <class '__main__.Subklass'>}
test test_delta_check ... {'datetime': <class 'datetime.datetime'>, 'Subklass': <class '__main__.Subklass'>}
ok
{'time': <class 'datetime.time'>, 'Subklass': <class '__main__.Subklass'>}
test test_date_check ... FAILED
test test_datetime_check ... FAILED
test test_time_check ... FAILED

Gotta head to work now, but I might pick this up again at a later date...

[davidhewitt]

...might be related to #288?

[pganssle]

There seems to be a race going on where multiple tests are making progress at the same time. This seems very odd given that all of the tests are supposedly acquiring the GIL... ?

This is an easy mistake to make (I made it when first building the datetime support), but when you create a GILGuard object with acquire_gil, you don't actually hold the GIL for the lifetime of object - what is guaranteed is that whenever your Rust code is executing you hold the GIL. Python and C functions are free to release the GIL, and in fact they will do so any time they, among other things, do an I/O operation like attempting to import a module.

This is important to keep in mind because it can lead to deadlocks if you attempt to layer on additional locking mechanisms - see this comment about how we avoid deadlocks in datetime.

With regard to "multiple tests making progress at the same time", I think that's perfectly fine, but it seems like maybe the issue is that in _get_subclasses we're overwriting what Subklass means (I'm not sure what namespace independent calls to py.run use, but it looks like maybe it's __main__)?

If that's the problem, I'm kinda surprised we haven't seen this problem before now, and that it's only happening on Windows machines. If that turns out to be the issue, it should be pretty simple to either give these things unique names or put them in a unique namespace.

[Alexander-N]

but it seems like maybe the issue is that in _get_subclasses we're overwriting what Subklass means (I'm not sure what namespace independent calls to py.run use, but it looks like maybe it's __main__)?

If that's the problem, I'm kinda surprised we haven't seen this problem before now, and that it's only happening on Windows machines. If that turns out to be the issue, it should be pretty simple to either give these things unique names or put them in a unique namespace.

I tried to make the names unique for each test: https://github.com/Alexander-N/pyo3/pull/3/files but the test still fails https://github.com/Alexander-N/pyo3/pull/3/checks?check_run_id=420934780.

[davidhewitt]

I tried to make the names unique for each test: https://github.com/Alexander-N/pyo3/pull/3/files but the test still fails https://github.com/Alexander-N/pyo3/pull/3/checks?check_run_id=420934780.

Strange. Also in the exception message I can't see the traceback being printed out, might need to run the tests with PYTHONUNBUFFERED=1 environment variable?

[Alexander-N]

Not sure if this is related to buffering, sometimes I get a NameError like:

NameError: name 'SubSubklassdatetime' is not defined

https://github.com/Alexander-N/pyo3/pull/3/checks?check_run_id=421093043

And sometimes not: https://github.com/Alexander-N/pyo3/pull/3/checks?check_run_id=421092950

[davidhewitt]

Ah yeah, probably not buffering. It's test_date_check which failed in the one without a traceback, which doesn't have an e.print() statement in it.

I wonder, could it be some kind of ownership issue where these things have a zero reference count and are getting GC'd out from underneath us? I don't see how that could be possible though...