Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] What combination of filters is equivalent to a bfloat16? #693

Closed
alxmrs opened this issue Jan 19, 2025 · 3 comments
Closed

[Question] What combination of filters is equivalent to a bfloat16? #693

alxmrs opened this issue Jan 19, 2025 · 3 comments

Comments

@alxmrs
Copy link

alxmrs commented Jan 19, 2025

My understanding is that Zarr is capable of storing ML model weights in distributed storage. Reading the documentation on filters, it seems like numcodecs already has all the requisite interfaces to support reading and writing bfloat16 datatypes (or really, any other crazy data type researchers find useful for modeling). I'm curious if anyone reading this knows the proper recipe of [codec/filter/etc]s to represent bfloat16 dtypes.

In think in practice, folks might use this library along with AsType() during encoding/decoding: https://github.com/jax-ml/ml_dtypes

@TomNicholas
Copy link
Member

See also zarr-developers/zarr-python#2656

@d-v-b
Copy link
Contributor

d-v-b commented Jan 19, 2025

I think we should just support bfloat16 as its own dtype, along with any other dtype that's useful to people.

@alxmrs
Copy link
Author

alxmrs commented Jan 27, 2025

Thanks, that makes sense to me.

@alxmrs alxmrs closed this as completed Jan 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants