Skip to content

Commit

Permalink
Merge branch 'main' into main-kerchunk
Browse files Browse the repository at this point in the history
  • Loading branch information
jsignell authored Jan 31, 2025
2 parents 77b1088 + 9c3d0f9 commit d4dfb6c
Show file tree
Hide file tree
Showing 7 changed files with 17 additions and 416 deletions.
2 changes: 1 addition & 1 deletion docs/releases.rst
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Documentation
Internal Changes
~~~~~~~~~~~~~~~~

- Vendor netCDF3 reader from kerchunk. (:pull:`397`) By `Tom Nicholas <https://github.com/TomNicholas>`_.
- Add netCDF3 test. (:pull:`397`) By `Tom Nicholas <https://github.com/TomNicholas>`_.

.. _v1.2.0:

Expand Down
17 changes: 15 additions & 2 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -431,14 +431,27 @@ We can also write these references out as an [IcechunkStore](https://icechunk.io
```python
# create an icechunk repository, session and write the virtual dataset to the session
from icechunk import Repository, Storage, VirtualChunkContainer, local_filesystem_storage
storage = local_filesystem_storage(str('combined'))
storage = local_filesystem_storage("./local/icechunk/store")

# By default, local virtual references and public remote virtual references can be read wihtout extra configuration.
repo = Repository.create(storage=storage)
session = repo.writeable_session("main")

# write the virtual dataset to the session with the IcechunkStore
combined_vds.virtualize.to_icechunk(session.store)
vds1.virtualize.to_icechunk(session.store)
session.commit("Wrote first dataset")
```

#### Append to an existing Icechunk Store

You can append a virtual dataset to an existing Icechunk store using the `append_dim` argument. This is especially useful for datasets that grow over time. Note that Zarr does not currently support concatenating datasets with different codecs or chunk shapes.

```python
session = repo.writeable_session("main")

# write the virtual dataset to the session with the IcechunkStore
vds2.virtualize.to_icechunk(session.store, append_dim="time")
session.commit("Appended second dataset")
```

See the [Icechunk documentation](https://icechunk.io/icechunk-python/virtual/#creating-a-virtual-dataset-with-virtualizarr) for more details.
Expand Down
5 changes: 1 addition & 4 deletions virtualizarr/readers/netcdf3.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ def open_virtual_dataset(
virtual_backend_kwargs: Optional[dict] = None,
reader_options: Optional[dict] = None,
) -> Dataset:
from virtualizarr.vendor.kerchunk.netCDF3 import NetCDF3ToZarr
from kerchunk.netCDF3 import NetCDF3ToZarr

if virtual_backend_kwargs:
raise NotImplementedError(
Expand All @@ -38,9 +38,6 @@ def open_virtual_dataset(
loadable_variables,
)

if reader_options is None:
reader_options = {}

refs = NetCDF3ToZarr(filepath, inline_threshold=0, **reader_options).translate()

# both group=None and group='' mean to read root group
Expand Down
21 changes: 0 additions & 21 deletions virtualizarr/vendor/kerchunk/LICENSE.txt

This file was deleted.

Empty file.
307 changes: 0 additions & 307 deletions virtualizarr/vendor/kerchunk/netCDF3.py

This file was deleted.

Loading

0 comments on commit d4dfb6c

Please sign in to comment.