https://github.com/ldeakin/zarrs

A rust library for the Zarr storage format for multidimensional arrays and metadata
https://github.com/ldeakin/zarrs

library rust zarr zarr-v2 zarr-v3 zarrs

Last synced: 2 months ago
JSON representation

A rust library for the Zarr storage format for multidimensional arrays and metadata

Host: GitHub
URL: https://github.com/ldeakin/zarrs
Owner: LDeakin
License: apache-2.0
Created: 2023-09-22T07:25:17.000Z (over 1 year ago)
Default Branch: main
Last Pushed: 2024-11-24T22:01:51.000Z (7 months ago)
Last Synced: 2024-11-27T09:06:39.231Z (7 months ago)
Topics: library, rust, zarr, zarr-v2, zarr-v3, zarrs
Language: Rust
Homepage: https://zarrs.dev
Size: 2.92 MB
Stars: 111
Watchers: 4
Forks: 9
Open Issues: 3
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Citation: CITATION.cff
- Codeowners: .github/CODEOWNERS

Awesome Lists containing this project

README

        # zarrs

[![Latest Version](https://img.shields.io/crates/v/zarrs.svg)](https://crates.io/crates/zarrs)

[![zarrs documentation](https://docs.rs/zarrs/badge.svg)][documentation]

![msrv](https://img.shields.io/crates/msrv/zarrs)

[![downloads](https://img.shields.io/crates/d/zarrs)](https://crates.io/crates/zarrs)

[![build](https://github.com/LDeakin/zarrs/actions/workflows/ci.yml/badge.svg)](https://github.com/LDeakin/zarrs/actions/workflows/ci.yml)

[![codecov](https://codecov.io/gh/LDeakin/zarrs/graph/badge.svg?token=OBKJQNAZPP)](https://codecov.io/gh/LDeakin/zarrs)

[![DOI](https://zenodo.org/badge/695021547.svg)](https://zenodo.org/badge/latestdoi/695021547)

`zarrs` is a Rust library for the [Zarr] storage format for multidimensional arrays and metadata.

`zarrs` supports [Zarr V3] and a V3 compatible subset of [Zarr V2].

It is fully up-to-date and conformant with the Zarr 3.0 specification with support for:

- all *core extensions* (data types, codecs, chunk grids, chunk key encodings, storage transformers),

- all accepted [Zarr Enhancement Proposals (ZEPs)](https://zarr.dev/zeps/) and several draft ZEPs:

  - ZEP 0003: Variable chunking

  - ZEP 0007: Strings

- experimental codecs and data types intended for standardisation, and

- user-defined custom extensions and stores.

> [!TIP]

> If you are a Python user, check out [`zarrs-python`].

> It includes a high-performance codec pipeline for the reference [`zarr-python`] implementation.

A changelog can be found [here][CHANGELOG].

Correctness issues with past versions are [detailed here][correctness_issues].

Developed at the [Department of Materials Physics, Australian National University, Canberra, Australia].

## Getting Started

- Review the [implementation status] ([zarr version support], [array support], [storage support], and the [`zarrs` ecosystem](#zarrs-ecosystem)).

- Read [The `zarrs` Book].

- View the [examples] and [the example below](#example).

- Read the [documentation].

## Example

```rust

use zarrs::group::GroupBuilder;

use zarrs::array::{ArrayBuilder, DataType, FillValue, ZARR_NAN_F32};

use zarrs::array::codec::GzipCodec; // requires gzip feature

use zarrs::array_subset::ArraySubset;

use zarrs::storage::ReadableWritableListableStorage;

use zarrs::filesystem::FilesystemStore; // requires filesystem feature

// Create a filesystem store

let store_path: PathBuf = "/path/to/hierarchy.zarr".into();

let store: ReadableWritableListableStorage =

    Arc::new(FilesystemStore::new(&store_path)?);

// Write the root group metadata

GroupBuilder::new()

    .build(store.clone(), "/")?

    // .attributes(...)

    .store_metadata()?;

// Create a new V3 array using the array builder

let array = ArrayBuilder::new(

    vec![3, 4], // array shape

    DataType::Float32,

    vec![2, 2].try_into()?, // regular chunk shape (non-zero elements)

    FillValue::from(ZARR_NAN_F32),

)

.bytes_to_bytes_codecs(vec![

    Arc::new(GzipCodec::new(5)?),

])

.dimension_names(["y", "x"].into())

.attributes(serde_json::json!({"Zarr V3": "is great"}).as_object().unwrap().clone())

.build(store.clone(), "/array")?; // /path/to/hierarchy.zarr/array

// Store the array metadata

array.store_metadata()?;

println!("{}", array.metadata().to_string_pretty());

// {

//     "zarr_format": 3,

//     "node_type": "array",

//     ...

// }

// Perform some operations on the chunks

array.store_chunk_elements::(

    &[0, 1], // chunk index

    &[0.2, 0.3, 1.2, 1.3]

)?;

array.store_array_subset_ndarray::(

    &[1, 1], // array index (start of subset)

    ndarray::array![[-1.1, -1.2], [-2.1, -2.2]]

)?;

array.erase_chunk(&[1, 1])?;

// Retrieve all array elements as an ndarray

let array_ndarray = array.retrieve_array_subset_ndarray::(&array.subset_all())?;

println!("{array_ndarray:4}");

// [[ NaN,  NaN,  0.2,  0.3],

//  [ NaN, -1.1, -1.2,  1.3],

//  [ NaN, -2.1,  NaN,  NaN]]

```

## `zarrs` Ecosystem

### Core

- [`zarrs`]: The core library for manipulating Zarr hierarchies.

- [`zarrs_data_type`]: Zarr data types (re-exported as `zarrs::data_type`).

- [`zarrs_metadata`]: Zarr metadata support (re-exported as `zarrs::metadata`).

- [`zarrs_plugin`]: The plugin API for `zarrs` (re-exported as `zarrs::plugin`).

- [`zarrs_storage`]: The storage API for `zarrs` (re-exported as `zarrs::storage`).

### Stores

- [`zarrs_filesystem`]: A filesystem store (re-exported as `zarrs::filesystem`).

- [`zarrs_object_store`]: [`object_store`] store support.

- [`zarrs_opendal`]: [`opendal`] store support.

- [`zarrs_http`]: A synchronous http store.

- [`zarrs_zip`]: A storage adapter for zip files.

- [`zarrs_icechunk`]: [`icechunk`] store support.

  - `git`-like version control for Zarr hierachies.

  - Read "virtual Zarr datacubes" of archival formats (e.g., [`netCDF4`](https://www.unidata.ucar.edu/software/netcdf/), [`HDF5`](https://www.hdfgroup.org/solutions/hdf5/), etc.) created by [`VirtualiZarr`](https://github.com/zarr-developers/VirtualiZarr) and backed by [`icechunk`].

### Bindings

- [`zarrs-python`]: A high-performance codec pipeline for [`zarr-python`].

- [`zarrs_ffi`]: A subset of `zarrs` exposed as a C/C++ API.

### Zarr Metadata Conventions

- [`ome_zarr_metadata`]: A library for OME-Zarr (previously OME-NGFF) metadata.

### Tools

- [`zarrs_tools`]: Various tools for creating and manipulating Zarr V3 data with the zarrs rust crate

  - A reencoder that can change codecs, chunk shape, convert Zarr V2 to V3, etc.

  - Create an [OME-Zarr] hierarchy from a Zarr array.

  - Transform arrays: crop, rescale, downsample, gradient magnitude, gaussian, noise filtering, etc.

  - Benchmarking tools and performance benchmarks of `zarrs`.

## Licence

`zarrs` is licensed under either of

 - the Apache License, Version 2.0 [LICENSE-APACHE](./LICENCE-APACHE) or  or

 - the MIT license [LICENSE-MIT](./LICENCE-MIT) or , at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

[CHANGELOG]: https://github.com/LDeakin/zarrs/blob/main/CHANGELOG.md

[correctness_issues]: https://github.com/LDeakin/zarrs/blob/main/doc/correctness_issues.md

[implementation status]: https://docs.rs/zarrs/latest/zarrs/#implementation-status

[zarr version support]: https://docs.rs/zarrs/latest/zarrs/#zarr-version-support

[array support]: https://docs.rs/zarrs/latest/zarrs/#array-support

[storage support]: https://docs.rs/zarrs/latest/zarrs/#storage-support

[examples]: https://github.com/LDeakin/zarrs/tree/main/zarrs/examples

[documentation]: https://docs.rs/zarrs/latest/zarrs/

[The `zarrs` Book]: https://book.zarrs.dev

[`zarrs`]: https://github.com/LDeakin/zarrs/tree/main/zarrs

[`zarrs_data_type`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_data_type

[`zarrs_metadata`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_metadata

[`zarrs_plugin`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_plugin

[`zarrs_storage`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_storage

[`zarrs_filesystem`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_filesystem

[`zarrs_http`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_http

[`zarrs_object_store`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_object_store

[`zarrs_opendal`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_opendal

[`zarrs_zip`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_zip

[`zarrs_icechunk`]: https://github.com/LDeakin/zarrs_icechunk

[`zarrs_ffi`]: https://github.com/LDeakin/zarrs_ffi

[`zarrs-python`]: https://github.com/ilan-gold/zarrs-python

[`zarr-python`]: https://github.com/zarr-developers/zarr-python

[`zarrs_tools`]: https://github.com/LDeakin/zarrs_tools

[`ome_zarr_metadata`]: https://github.com/LDeakin/rust_ome_zarr_metadata

[`object_store`]: https://github.com/apache/arrow-rs/tree/main/object_store

[`opendal`]: https://github.com/apache/OpenDAL

[`icechunk`]: https://github.com/earth-mover/icechunk

[Zarr]: https://zarr.dev

[Zarr V3]: https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html

[Zarr V2]: https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html

[OME-Zarr]: https://ngff.openmicroscopy.org/latest/

[Department of Materials Physics, Australian National University, Canberra, Australia]: https://physics.anu.edu.au/research/mp/

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/ldeakin/zarrs

Awesome Lists containing this project

README