{"id":17549094,"url":"https://github.com/ldeakin/zarrs","last_synced_at":"2025-04-08T17:17:56.345Z","repository":{"id":196426645,"uuid":"695021547","full_name":"LDeakin/zarrs","owner":"LDeakin","description":"A rust library for the Zarr storage format for multidimensional arrays and metadata","archived":false,"fork":false,"pushed_at":"2024-11-24T22:01:51.000Z","size":3057,"stargazers_count":111,"open_issues_count":3,"forks_count":9,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-11-27T09:06:39.231Z","etag":null,"topics":["library","rust","zarr","zarr-v2","zarr-v3","zarrs"],"latest_commit_sha":null,"homepage":"https://zarrs.dev","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LDeakin.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-22T07:25:17.000Z","updated_at":"2024-11-24T23:50:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"6c4d8747-900c-4bb1-872c-56c29d710167","html_url":"https://github.com/LDeakin/zarrs","commit_stats":{"total_commits":891,"total_committers":4,"mean_commits":222.75,"dds":0.02581369248035914,"last_synced_commit":"b5667ae46474b519074ea542cd4b2360981bc038"},"previous_names":["ldeakin/zarrs"],"tags_count":74,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LDeakin%2Fzarrs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LDeakin%2Fzarrs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LDeakin%2Fzarrs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LDeakin%2Fzarrs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LDeakin","download_url":"https://codeload.github.com/LDeakin/zarrs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247440271,"owners_count":20939213,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["library","rust","zarr","zarr-v2","zarr-v3","zarrs"],"created_at":"2024-10-21T02:49:48.796Z","updated_at":"2025-04-08T17:17:56.322Z","avatar_url":"https://github.com/LDeakin.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# zar\u003cins\u003ers\u003c/ins\u003e\n\n[![Latest Version](https://img.shields.io/crates/v/zarrs.svg)](https://crates.io/crates/zarrs)\n[![zarrs documentation](https://docs.rs/zarrs/badge.svg)][documentation]\n![msrv](https://img.shields.io/crates/msrv/zarrs)\n[![downloads](https://img.shields.io/crates/d/zarrs)](https://crates.io/crates/zarrs)\n[![build](https://github.com/LDeakin/zarrs/actions/workflows/ci.yml/badge.svg)](https://github.com/LDeakin/zarrs/actions/workflows/ci.yml)\n[![codecov](https://codecov.io/gh/LDeakin/zarrs/graph/badge.svg?token=OBKJQNAZPP)](https://codecov.io/gh/LDeakin/zarrs)\n[![DOI](https://zenodo.org/badge/695021547.svg)](https://zenodo.org/badge/latestdoi/695021547)\n\n`zarrs` is a Rust library for the [Zarr] storage format for multidimensional arrays and metadata.\n\n`zarrs` supports [Zarr V3] and a V3 compatible subset of [Zarr V2].\nIt is fully up-to-date and conformant with the Zarr 3.0 specification with support for:\n- all *core extensions* (data types, codecs, chunk grids, chunk key encodings, storage transformers),\n- all accepted [Zarr Enhancement Proposals (ZEPs)](https://zarr.dev/zeps/) and several draft ZEPs:\n  - ZEP 0003: Variable chunking\n  - ZEP 0007: Strings\n- experimental codecs and data types intended for standardisation, and\n- user-defined custom extensions and stores.\n\n\u003e [!TIP]\n\u003e If you are a Python user, check out [`zarrs-python`].\n\u003e It includes a high-performance codec pipeline for the reference [`zarr-python`] implementation.\n\nA changelog can be found [here][CHANGELOG].\nCorrectness issues with past versions are [detailed here][correctness_issues].\n\nDeveloped at the [Department of Materials Physics, Australian National University, Canberra, Australia].\n\n## Getting Started\n- Review the [implementation status] ([zarr version support], [array support], [storage support], and the [`zarrs` ecosystem](#zarrs-ecosystem)).\n- Read [The `zarrs` Book].\n- View the [examples] and [the example below](#example).\n- Read the [documentation].\n\n## Example\n```rust\nuse zarrs::group::GroupBuilder;\nuse zarrs::array::{ArrayBuilder, DataType, FillValue, ZARR_NAN_F32};\nuse zarrs::array::codec::GzipCodec; // requires gzip feature\nuse zarrs::array_subset::ArraySubset;\nuse zarrs::storage::ReadableWritableListableStorage;\nuse zarrs::filesystem::FilesystemStore; // requires filesystem feature\n\n// Create a filesystem store\nlet store_path: PathBuf = \"/path/to/hierarchy.zarr\".into();\nlet store: ReadableWritableListableStorage =\n    Arc::new(FilesystemStore::new(\u0026store_path)?);\n\n// Write the root group metadata\nGroupBuilder::new()\n    .build(store.clone(), \"/\")?\n    // .attributes(...)\n    .store_metadata()?;\n\n// Create a new V3 array using the array builder\nlet array = ArrayBuilder::new(\n    vec![3, 4], // array shape\n    DataType::Float32,\n    vec![2, 2].try_into()?, // regular chunk shape (non-zero elements)\n    FillValue::from(ZARR_NAN_F32),\n)\n.bytes_to_bytes_codecs(vec![\n    Arc::new(GzipCodec::new(5)?),\n])\n.dimension_names([\"y\", \"x\"].into())\n.attributes(serde_json::json!({\"Zarr V3\": \"is great\"}).as_object().unwrap().clone())\n.build(store.clone(), \"/array\")?; // /path/to/hierarchy.zarr/array\n\n// Store the array metadata\narray.store_metadata()?;\nprintln!(\"{}\", array.metadata().to_string_pretty());\n// {\n//     \"zarr_format\": 3,\n//     \"node_type\": \"array\",\n//     ...\n// }\n\n// Perform some operations on the chunks\narray.store_chunk_elements::\u003cf32\u003e(\n    \u0026[0, 1], // chunk index\n    \u0026[0.2, 0.3, 1.2, 1.3]\n)?;\narray.store_array_subset_ndarray::\u003cf32, _\u003e(\n    \u0026[1, 1], // array index (start of subset)\n    ndarray::array![[-1.1, -1.2], [-2.1, -2.2]]\n)?;\narray.erase_chunk(\u0026[1, 1])?;\n\n// Retrieve all array elements as an ndarray\nlet array_ndarray = array.retrieve_array_subset_ndarray::\u003cf32\u003e(\u0026array.subset_all())?;\nprintln!(\"{array_ndarray:4}\");\n// [[ NaN,  NaN,  0.2,  0.3],\n//  [ NaN, -1.1, -1.2,  1.3],\n//  [ NaN, -2.1,  NaN,  NaN]]\n```\n\n## `zarrs` Ecosystem\n\n### Core\n- [`zarrs`]: The core library for manipulating Zarr hierarchies.\n- [`zarrs_data_type`]: Zarr data types (re-exported as `zarrs::data_type`).\n- [`zarrs_metadata`]: Zarr metadata support (re-exported as `zarrs::metadata`).\n- [`zarrs_plugin`]: The plugin API for `zarrs` (re-exported as `zarrs::plugin`).\n- [`zarrs_storage`]: The storage API for `zarrs` (re-exported as `zarrs::storage`).\n\n### Stores\n- [`zarrs_filesystem`]: A filesystem store (re-exported as `zarrs::filesystem`).\n- [`zarrs_object_store`]: [`object_store`] store support.\n- [`zarrs_opendal`]: [`opendal`] store support.\n- [`zarrs_http`]: A synchronous http store.\n- [`zarrs_zip`]: A storage adapter for zip files.\n- [`zarrs_icechunk`]: [`icechunk`] store support.\n  - `git`-like version control for Zarr hierachies.\n  - Read \"virtual Zarr datacubes\" of archival formats (e.g., [`netCDF4`](https://www.unidata.ucar.edu/software/netcdf/), [`HDF5`](https://www.hdfgroup.org/solutions/hdf5/), etc.) created by [`VirtualiZarr`](https://github.com/zarr-developers/VirtualiZarr) and backed by [`icechunk`].\n\n### Bindings\n- [`zarrs-python`]: A high-performance codec pipeline for [`zarr-python`].\n- [`zarrs_ffi`]: A subset of `zarrs` exposed as a C/C++ API.\n\n### Zarr Metadata Conventions\n- [`ome_zarr_metadata`]: A library for OME-Zarr (previously OME-NGFF) metadata.\n\n### Tools\n- [`zarrs_tools`]: Various tools for creating and manipulating Zarr V3 data with the zarrs rust crate\n  - A reencoder that can change codecs, chunk shape, convert Zarr V2 to V3, etc.\n  - Create an [OME-Zarr] hierarchy from a Zarr array.\n  - Transform arrays: crop, rescale, downsample, gradient magnitude, gaussian, noise filtering, etc.\n  - Benchmarking tools and performance benchmarks of `zarrs`.\n\n## Licence\n`zarrs` is licensed under either of\n - the Apache License, Version 2.0 [LICENSE-APACHE](./LICENCE-APACHE) or \u003chttp://www.apache.org/licenses/LICENSE-2.0\u003e or\n - the MIT license [LICENSE-MIT](./LICENCE-MIT) or \u003chttp://opensource.org/licenses/MIT\u003e, at your option.\n\nUnless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.\n\n[CHANGELOG]: https://github.com/LDeakin/zarrs/blob/main/CHANGELOG.md\n[correctness_issues]: https://github.com/LDeakin/zarrs/blob/main/doc/correctness_issues.md\n[implementation status]: https://docs.rs/zarrs/latest/zarrs/#implementation-status\n[zarr version support]: https://docs.rs/zarrs/latest/zarrs/#zarr-version-support\n[array support]: https://docs.rs/zarrs/latest/zarrs/#array-support\n[storage support]: https://docs.rs/zarrs/latest/zarrs/#storage-support\n[examples]: https://github.com/LDeakin/zarrs/tree/main/zarrs/examples\n[documentation]: https://docs.rs/zarrs/latest/zarrs/\n[The `zarrs` Book]: https://book.zarrs.dev\n\n[`zarrs`]: https://github.com/LDeakin/zarrs/tree/main/zarrs\n[`zarrs_data_type`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_data_type\n[`zarrs_metadata`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_metadata\n[`zarrs_plugin`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_plugin\n[`zarrs_storage`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_storage\n[`zarrs_filesystem`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_filesystem\n[`zarrs_http`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_http\n[`zarrs_object_store`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_object_store\n[`zarrs_opendal`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_opendal\n[`zarrs_zip`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_zip\n[`zarrs_icechunk`]: https://github.com/LDeakin/zarrs_icechunk\n[`zarrs_ffi`]: https://github.com/LDeakin/zarrs_ffi\n[`zarrs-python`]: https://github.com/ilan-gold/zarrs-python\n[`zarr-python`]: https://github.com/zarr-developers/zarr-python\n[`zarrs_tools`]: https://github.com/LDeakin/zarrs_tools\n[`ome_zarr_metadata`]: https://github.com/LDeakin/rust_ome_zarr_metadata\n[`object_store`]: https://github.com/apache/arrow-rs/tree/main/object_store\n[`opendal`]: https://github.com/apache/OpenDAL\n[`icechunk`]: https://github.com/earth-mover/icechunk\n\n[Zarr]: https://zarr.dev\n[Zarr V3]: https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html\n[Zarr V2]: https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html\n[OME-Zarr]: https://ngff.openmicroscopy.org/latest/\n\n[Department of Materials Physics, Australian National University, Canberra, Australia]: https://physics.anu.edu.au/research/mp/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fldeakin%2Fzarrs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fldeakin%2Fzarrs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fldeakin%2Fzarrs/lists"}