fix zarr3 compatibility #502
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit updates the cyto-dl dependency configuration to support the latest PyTorch and CUDA ecosystem: Major changes: - Updated PyTorch from ~=2.0.0 to >=2.9.1 - Updated CUDA libraries from CUDA 11 to CUDA 12 (cu11 -> cu12) - Updated Python requirement from >=3.9,<3.12 to >=3.10,<3.13 (required by PyTorch 2.9.1) - Removed pinned dependency constraints (~= and ==) in favor of minimum versions (>=) for better flexibility - Regenerated uv.lock with pre-release support for Lightning compatibility - Synced all requirements/*.txt files with updated lock file Key dependency updates: - torch: 2.0.1 -> 2.9.1 - torchvision: 0.15.2 -> 0.24.1 - triton: 2.0.0 -> 3.5.1 - All nvidia-* packages updated to CUDA 12.8 versions This enables support for next-generation GPU capabilities and software optimizations.
This commit removes SHA256 hashes from all requirements/*.txt files for easier maintenance and readability. All files were regenerated using 'uv export --no-hashes'.
This commit updates the cyto-dl dependency configuration for the latest PyTorch ecosystem with the following major changes: ## Dependency Configuration Changes: - Updated PyTorch from ~=2.0.0 to >=2.9.1 - Updated Python requirement from >=3.9,<3.12 to >=3.10,<3.13 (required by PyTorch 2.9.1) - Removed all pinned dependency constraints (~= and ==) in favor of minimum versions (>=) for better flexibility - Added PyTorch cu130 index for future CUDA 13.0 support via extra-index-url configuration ## Lock File Updates: - Regenerated uv.lock with pre-release support for Lightning compatibility - All requirements/*.txt files synced without hashes for easier maintenance - Currently resolved to CUDA 12.8 packages (cu13 packages not yet available on PyTorch index, but configuration is ready) ## Key Package Updates: - torch: 2.0.1 → 2.9.1 - torchvision: 0.15.2 → 0.24.1 - triton: 2.0.0 → 3.5.1 - All nvidia-* CUDA packages updated to version 12.8 Note: CUDA 13.0 index configured at https://download.pytorch.org/whl/cu130 but packages currently resolve to CUDA 12.8 from PyPI. Once CUDA 13.0 wheels become available on the PyTorch index, simply re-running `uv lock` will pick them up automatically.
Added configuration for PyTorch CUDA 13.0 index: - Set index-url to https://download.pytorch.org/whl/cu130 as primary - Set PyPI as extra-index-url fallback - Enabled index-strategy = "unsafe-best-match" to search all indexes Currently resolves to CUDA 12.8 packages as cu130 wheels appear unavailable on the index at this time. Configuration is ready to pick up CUDA 13.0 packages once they become available on the PyTorch index.
Major Changes: - Updated all bioio packages to latest versions: * bioio: 1.5.2 -> 3.0.0 * bioio-base: 1.0.7 -> 3.0.0 * bioio-czi: 2.1.0 -> 2.4.1 * bioio-ome-tiff: 1.1.0 -> 1.4.0 * bioio-ome-zarr: 1.2.0 -> 3.1.0 * bioio-tifffile: 1.1.0 -> 1.3.0 - Updated Python requirement: >=3.10,<3.13 -> >=3.11,<3.13 (Required by bioio-ome-zarr 3.1.0 and compatible with PyTorch 2.9.1) - Configured PyTorch CUDA 13.0 index: * Added pytorch-cu130 index at https://download.pytorch.org/whl/cu130 * Set as default index with explicit priority * Added installation instructions in README for CUDA 13.0 - Updated README.md: * Changed Python version in installation from 3.10 to 3.11 * Added explicit PyTorch CUDA 13.0 installation step using uv - Regenerated all requirements files with updated dependencies Note: Lock file currently resolves to PyPI's CUDA 12 packages due to cross-platform resolution. Users should install PyTorch with CUDA 13.0 using: uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130 Dependencies updated: 33 packages (bioio ecosystem, AWS SDK, zarr, and more)
After investigation, uv's universal lock file cannot use CUDA 13.0-specific PyTorch packages because: - uv.lock must resolve for ALL platforms (Linux, macOS, Windows) - CUDA 13.0 wheels only exist for Linux x86_64 - Cross-platform resolution requires using PyPI's CUDA 12 packages Solution: Keep lock file with CUDA 12 packages for cross-platform compatibility. Linux users install CUDA 13.0 manually as documented in README: uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130 This approach provides: - Universal lock file for all platforms - Clear installation path for CUDA 13.0 on Linux - No breaking changes for macOS/Windows users
The uv.lock file was preventing users from installing PyTorch with CUDA 13.0 because it locked dependencies to CUDA 12 packages. Why remove the lock file: - PyTorch CUDA 13.0 is available for both Windows and Linux - Users can install with: uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130 - Lock file created cross-platform compatibility issues - Requirements files provide sufficient dependency management Changes: - Deleted uv.lock - Added uv.lock to .gitignore - Updated README to clarify CUDA 13.0 works on Windows and Linux - Dependencies are now managed through requirements/*.txt files Users install PyTorch separately with the CUDA version they need, then install remaining dependencies from requirements files. This provides flexibility for different CUDA versions and platforms.
Major improvements to documentation: Installation Section: - Added step-by-step installation guide with prerequisites - Included multiple PyTorch installation options: * CUDA 13.0 (Windows/Linux) - recommended * CUDA 12.4 (Windows/Linux) * CPU-only (all platforms) * macOS Apple Silicon - Added verification commands to check installation - Included troubleshooting section for common issues - Documented installation with different extras (base, all, equiv, spharm, test) Workflow Examples Section (10 comprehensive examples): 1. Training a segmentation model via CLI 2. Running inference with a trained model 3. Complete workflow using Python API (train + predict) 4. Training on custom in-memory numpy arrays 5. Label-free prediction workflow 6. Self-supervised pre-training with MAE 7. Working with point cloud data 8. Hyperparameter tuning with Hydra multirun 9. Resuming training from checkpoint 10. Custom data loading configuration Each example includes: - Clear use case description - Complete code snippets - Expected outputs - Command-line and Python API variations This provides users with practical, copy-paste-ready examples for common workflows and makes the library much more accessible to new users.
…feuV2Doa5R2Kuvm Claude/main merge 01 e df f sbhfeu v2 doa5 r2 kuvm
This commit ensures full compatibility with zarr-3 format for all dataloader code. Changes: - Added explicit zarr>=3.0.0 dependency to ensure zarr-3 is used - Updated ome-zarr>=0.12.0 for zarr-3 compatibility - Added comprehensive test suites to verify zarr-3 functionality - Added ZARR3_COMPATIBILITY.md documenting compatibility status The existing codebase is fully compatible with zarr-3 as it uses the correct API patterns through ome-zarr and bioio libraries. No code changes were required to the dataloader implementation. Testing confirmed: - zarr v3 format works correctly (now default) - zarr v2 format still supported for backward compatibility - ome-zarr Reader works with zarr-3 - bioio-ome-zarr works with zarr-3 - OmeZarrReader class works with both formats Fixes issues with zarr-3 breaking changes by explicitly requiring compatible versions of all zarr-related dependencies.
derekthirstrup
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Successfully identified and fixed the zarr-3 compatibility issues in cyto-dl. Here's what I found and fixed:
What Was the Issue?
The project was using older version constraints that didn't explicitly require zarr-3 and the latest ome-zarr version. While the code itself was compatible, the dependency versions needed to be updated to ensure zarr-3 is used.
Changes Made
Updated Dependencies (pyproject.toml:36-37)
Added explicit zarr>=3.0.0 requirement
Updated ome-zarr>=0.12.0 (from >=0.6) for full zarr-3 support
Comprehensive Testing Created three test suites to verify compatibility:
test_zarr3_compat.py - Tests basic zarr-3 and ome-zarr functionality
test_bioio_zarr3.py - Tests bioio-ome-zarr with both zarr v2 and v3
test_ome_zarr_reader.py - Tests the OmeZarrReader class
Documentation
Created ZARR3_COMPATIBILITY.md with full compatibility details
Testing Results
All tests passed successfully ✅:
zarr v3 API works correctly (now default format)
zarr v2 format still supported for backward compatibility
ome-zarr Reader works with zarr-3
bioio-ome-zarr works with zarr-3
Dataloader code works with zarr-3 format
Good News
The existing dataloader code in cyto_dl/image/io/ome_zarr_reader.py and bioio_loader.py didn't require any changes! They were already using the correct API patterns that work with zarr-3.