Zarr-Python → Refactoring

[sanketverma1704]

Hi everyone! 👋🏻

Recently, we had meetings on July 6th and 7th, 2023 (meeting notes) led by @rabernat to discuss and decide the path forward for Zarr-Python development. After a good discussion with the attendees and gauging their interest, we decided to divide the larger group into two groups; they are:

• Benchmarking and performance working group 📈 → led by @JackKelly
• Refactoring working group 🧑🏻‍💻 → led by @jhamman

Thank you @JackKelly and @jhamman, for stepping up! 🙏🏻

What's next? 👀

This discussion thread aims to kick off the 🧑🏻‍💻 Refactoring 🧑🏻‍💻 working group and hold any top-level discussions related to the development work.

Here's the project board to organise and track progress: https://github.com/orgs/zarr-developers/projects/5

Thank you, everyone, for joining the meetings and sharing your insights. Please feel free to ask any questions.

@jhamman, please take it from here.

[cisaacstern]

Thanks for coordinating @MSanKeys963! Looking forward to contributing to this working group.

[jhamman]

Thanks @MSanKeys963 and @rabernat for getting this going!

Background and objectives

Zarr-python has traditionally been the canonical implementation of the Zarr spec. It has also been the place where a lot of experimentation has been done . Today, Zarr-python has a fair bit of technical debt, and importantly, an out-of-date implementation of the Zarr V3 spec.

Based on the conversations we had in July, I want to highlight the two main objectives for this focus group:

1. Bring Zarr-python up to date with the V3 spec
2. Pay down technical debt (focus on storage interface and test environment)

While these aren't particularly sexy objectives, they are poised to be highly impactful as most users of Zarr are currently relying on Zarr-python already.

If you are keen to get involved with either the V3 spec implementation or storage related refactors, this is the group for you.

Meetings

We are going to meet twice a month starting in mid-August. We will schedule the first meeting for 1-hr. Following meetings will be 30 minutes. Please fill out this poll to let us know what times you could attend the kickoff meeting.

update 8/4/23 - call in details

To join the video meeting, click this link: https://meet.google.com/mxq-jbnu-zka
Otherwise, to join by phone, dial +1 316-746-8152 and enter this PIN: 393 374 434#
To view more phone numbers, click this link: https://tel.meet/mxq-jbnu-zka?hs=5

During the kickoff meeting, we will attempt to map out the primary effort around getting the Zarr-V3 spec reimplemented in Zarr-python.

Reading

• Agenda for first meeting (please suggest changes in the doc, if you want!)
• Zarr V3 specification

Very quick intro to me

Today, I'm the CTO at @earth-mover, a software startup that uses Zarr extensively. I've been contributing to open source for many years. I got my start in open source in graduate school when I was fortunate enough to start contributing to Xarray. I've since spent a lot of time working on various problems around the Xarray, Dask, Zarr, and Jupyter ecosystem, often wearing the @pangeo-data evangelist hat. After about 6 years at @NCAR, I spent a few years at @carbonplan, where I was the technology director where we built some cool Zarr things like this and this.

[jhamman]

Timezones have now been activated in the poll.

[cisaacstern]

Thanks @jhamman, poll complete, looking forward!

[jhamman]

I've added call in details for the kickoff meeting scheduled for August 16, 2023 at 9a PT.

[jakirkham]

@MSanKeys963 could you please add the call details to the community calls calendar?

https://github.com/zarr-developers/community-calls

[sanketverma1704]

@MSanKeys963 could you please add the call details to the community calls calendar?

https://github.com/zarr-developers/community-calls

@jakirkham - Done. 💯
Check here: https://zarr.dev/community-calls/

[sanketverma1704]

Update: I've included the link for the project board in the description.

[sanketverma1704]

Update: The refactoring kick-off meeting is now on the Zarr Community Calendar.
Please check here: https://zarr.dev/community-calls/

[sanketverma1704]

Update: As discussed during the kick-off meeting, I've added bi-weekly standup meetings for the refactor group to the Zarr Community Calendar. See zarr-developers/community-calls#24.

Meeting details here: https://zarr.dev/community-calls/

[sanketverma1704]

Hi all. 👋🏻

We have our first standup meeting tomorrow. Meeting details are in the community calendar.
Check here: https://zarr.dev/community-calls/

CC: @jhamman @cisaacstern @tomwhite @d-v-b @maxrjones @norlandrhagen @zarr-developers/steering-council

[jhamman]

For those following along, I've updated the project board with an initial set of V3 compat tickets. More tickets will certainly need to be added but this seemed like a good first tranche.

[jhamman]

Meeting notes from today's meeting: https://hackmd.io/Rbh6oae8S7mNU-CPWWwPaw

[cisaacstern]

Given that Earthmover (presumably including @jhamman) are presenting at the zarr refactor meeting time this week, I'm assuming we will skip this week's meeting?

https://discourse.pangeo.io/t/pangeo-showcase-arraylake-a-cloud-native-data-lake-platform-for-earth-system-science/3701

[d-v-b]

works for me

[jhamman]

Came here to propose the same thing. Let's skip this week's meeting. @MSanKeys963 - can you cancel the calendar event for this week?

[sanketverma1704]

Done. 💯

[jhamman]

We'll have our next refactor meeting today (in 45 minutes). Today, I'm going to introduce a straw-man proposal that we integrate Zarrita's Array and Group classes (which today support both v2 and v3) into Zarr-Python.

cc @normanrz who has been working hard on this library lately (and unfortunately is out this week).

[jakirkham]

Are we keeping notes from this meeting? Would it be possible to publish them to the website (as we do with other Zarr meetings)?

cc @MSanKeys963 @jhamman (who may have thoughts here)

[jhamman]

We will have our next working group meeting tomorrow at 9a PT (@MSanKeys963 updated the calendar invite).

In advance of this meeting, I have prepared a draft design doc for Zarr-Python 3.0 which includes many of the ideas we have been discussing at this meeting. It also provides a way to bring Zarr-Python and Zarrita back together. If folks have time to look over this (and comment) ahead of the meeting tomorrow, that would be great.

[jhamman]

Thanks to @normanrz, @d-v-b, @dcherian, and @monodeldiablo for early comments on this doc.

[jhamman]

This doc has been shared more broadly now: #1569

Next week's meeting will also be focused on this topic. Time permitting, we'll turn the discussion toward early actions that we can take.

[jhamman]

At yesterday's, there was a request to document ways for others to jump in to the v3 fun. I've started to create a list of issues that would be good ways to help with the effort: https://github.com/zarr-developers/zarr-python/issues?q=is%3Aissue+is%3Aopen+label%3AV3+label%3A%22help+wanted%22

I'm sure we'll be opening more issues as we go but this is the list that we came up yesterday.

[jhamman]

We'll have a check in today at the regular time. Expecting it to be relatively low head count due to the upcoming holiday but I'll be there to discuss progress toward v3.

[sanketverma1704]

I'll be there.

[TomNicholas]

Are these meetings still occurring? I tried joining the zoom link just now but no-one here...

[d-v-b]

according to this calendar the next refactoring meeting is next week

[sanketverma1704]

@TomNicholas: https://zarr.dev/community-calls/ is one true source for all Zarr related meetings. (there's a Google calendar at the bottom)

[jhamman]

This meeting is happening today (in 25 minutes). Hope to see folks there!

[jhamman]

For folks following this issue... Reminding you that this meeting is still happening (now every Friday at 7a PT).

We're closing in on the 3.0 release and I want to make sure we're all aligned on the remaining release blockers. Some highlights in the past few weeks:

• Xarray and Dask support with Zarr-Python 2 and 3 are all but done (huge props to @TomAugspurger for driving the implementation of consolidated metadata)
• @rabernat did some perf testing with Zarr Python 3 and wow, async for the win.
• many bugs and rough edges were smoothed out in 3.0.0.beta

For tomorrow's meeting, here are some specific technical topics I'd like to discuss are:

• v2 codec pipeline vis a vis Kerchunk (cc @normanrz)
• feature flags for experimental features in v3 (codecs, dtypes, stores, consolidated metadata)
• store open mode semantics (cc @rabernat)

Hope folks can make the meeting but if not, see you around the GitHub!