Skip lmdb compat tests #2314
Closed
Skip lmdb compat tests #2314
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously in several places we were calling `old_ac.clear()` to clear all arctic/libraries created by an compatibility tests. This however was producing issues when running pytest with multiprocessing as we do on the CI: - Each pytest worker sets up its own instance of the session scoped fixtures - When one process finishes with a fixture it runs the cleanup on an arctic instance which is potentially actively used by another pytest worker. Thus the spurious mongo library not found errors. To mitigate this we introduce a new context manager `CompatLibrary`. It is responsible for setting up the `Venv`, `VenvArctic` and `VenvLib` and also creates the library on `__enter__` and cleans the library on `__exit__`. So we rework all compat tests to use the new `CompatLibrary`
For `test_storage_mover_clone_old_library` we need to create multiple libraries from the same arctic instance. (because we can't have multiple active arctic instances to lmdb)
For LMDB on windows deleting a directory will fail if a file in it is still in use. Likely the short lived venv process or an active arctic instance remains alive for a while longer and prevents the delete. In this case we just catch the storage exception and leave the deletion to the temporary direcory used for the root of lmdb.
We have seen spuirous very rare segfaults during cleanup of lmdb compat tests. In general lmdb compat tests are not that useful as not much old data is stored on lmdb. And it is hard to correctly handle lmdb for compat tests which require muliple processes acessing the same LMDB storage (which LMDB doesn't fully support)
IvoDD
requested review from
alexowens90,
willdealtry and
poodlewars
as code owners
G-D-Petrov
approved these changes
Apr 10, 2025
|
LMDB is important to a lot of hobbyists and people first trying out arcticdb. We need to make sure it's a good experience and that includes cross version compatibility so it would be better to not completely drop these. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have seen spuirous very rare segfaults during cleanup of lmdb compat tests.
In general lmdb compat tests are not that useful as not much old data is stored on lmdb. And it is hard to correctly handle lmdb for compat tests which require muliple processes acessing the same LMDB storage (which LMDB doesn't fully support)
Reference Issues/PRs
What does this implement or fix?
Any other comments?
Checklist
Checklist for code changes...