{"id":16518971,"url":"https://github.com/zoj613/zarr-ml","last_synced_at":"2025-03-21T08:31:59.105Z","repository":{"id":244679940,"uuid":"795734169","full_name":"zoj613/zarr-ml","owner":"zoj613","description":"An implementation of the Zarr storage format specification for chunked \u0026 compressed multidimensional arrays.","archived":false,"fork":false,"pushed_at":"2025-03-12T22:03:27.000Z","size":1604,"stargazers_count":17,"open_issues_count":8,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-17T23:37:47.468Z","etag":null,"topics":["hdf5","n5","ncdf","zarr","zarr-v3"],"latest_commit_sha":null,"homepage":"https://zoj613.github.io/zarr-ml/","language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zoj613.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-03T23:17:09.000Z","updated_at":"2025-02-27T21:13:49.000Z","dependencies_parsed_at":"2024-06-16T18:11:49.465Z","dependency_job_id":"52a9d62c-1703-43b6-843f-2597f9f15ed7","html_url":"https://github.com/zoj613/zarr-ml","commit_stats":null,"previous_names":["zoj613/zarr-ml"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zoj613%2Fzarr-ml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zoj613%2Fzarr-ml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zoj613%2Fzarr-ml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zoj613%2Fzarr-ml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zoj613","download_url":"https://codeload.github.com/zoj613/zarr-ml/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244765548,"owners_count":20506831,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hdf5","n5","ncdf","zarr","zarr-v3"],"created_at":"2024-10-11T16:44:25.452Z","updated_at":"2025-03-21T08:31:59.096Z","avatar_url":"https://github.com/zoj613.png","language":"OCaml","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![codecov][1]](https://codecov.io/gh/zoj613/zarr-ml)\n[![CI][2]](https://github.com/zoj613/zarr-ml/actions/workflows/)\n[![license][3]](https://github.com/zoj613/zarr-ml/blob/main/LICENSE)\n\n# zarr-ml\nThis library provides an OCaml implementation of the Zarr version 3\nstorage format specification for chunked \u0026 compressed multi-dimensional\narrays, designed for use in parallel computing.\n\n## Features\n- Supports creating n-dimensional Zarr arrays and chunking them along any dimension.\n- Compresses chunks using a variety of supported compression codecs.\n- Supports indexing operations to read/write views of a Zarr array.\n- Supports storing arrays in-memory or the local filesystem. It is also\n  extensible, allowing users to easily create and use their own custom storage\n  backends. See the example implementing an [In-memory Zip archive store][9] for more details.\n- Supports both synchronous and asynchronous I/O via [Lwt][4] and [Eio][8]. The user can\n  easily use their own scheduler of choice. See the [example][10] implementing\n  a filesystem store that uses the [Picos][11] concurrency library for non-blocking I/O.\n- Leverages the strong type system of Ocaml to create a type-safe API; making\n  it impossible to create, read or write malformed arrays.\n- Supports organizing arrays into heirarchies via groups.\n\n## Documentation\nAPI documentation can be found [here][5]. The full specification of the storage\nformat can be found [there][6].\n\n## Installation\nThe library comes in several flavors dependending on the synchronous/asynchronous\nbackend of choice. To install the synchronous API, use\n```shell\n$ opam install zarr-sync\n```\nTo install zarr with an asynchronous API powered by `Lwt` or `Eio`, use\n```shell\n$ opam install zarr-lwt\n$ opam install zarr-eio\n```\nTo install the development version using the latest git commit, do\n```\n# for zarr-sync\n opam pin add zarr-sync git+https://github.com/zoj613/zarr-ml \n# for zarr-lwt\n opam pin add zarr-lwt git+https://github.com/zoj613/zarr-ml \n# for zarr-eio\n opam pin add zarr-eio git+https://github.com/zoj613/zarr-ml \n ```\n\n## Quick start\nBelow is a demonstration of the library's API for synchronous reads/writes.\nA similar example using the `Lwt`-backed Asynchronous API can be found [here][7]\n### setup\n```ocaml\nopen Zarr\nopen Zarr.Codecs\nopen Zarr.Indexing\nopen Zarr_sync.Storage\nopen IO.Infix  (* opens infix operators \u003e\u003e= and \u003e\u003e| for monadic bind \u0026 map *)\n\nlet store = FilesystemStore.create \"testdata.zarr\";;\n```\n### create group\n```ocaml\nlet group_node = Node.Group.of_path \"/some/group\";;\nFilesystemStore.Group.create store group_node;;\n```\n### create an array\n```ocaml\nlet array_node = Node.Array.(group_node / \"name\");;\n(* creates an array with char data type and fill value '?' *)\nFilesystemStore.Array.create\n  ~codecs:[`Transpose [2; 0; 1]; `Bytes BE; `Gzip L2]\n  ~shape:[100; 100; 50]\n  ~chunks:[10; 15; 20]\n  Ndarray.Char \n  '?'\n  array_node\n  store;;\n```\n### read/write from/to an array\n```ocaml\nlet slice = [R (0, 20); I 10; F];;\nlet x = FilesystemStore.Array.read store array_node slice Ndarray.Char;;\n(* Do some computation on the array slice *)\nlet x' = Zarr.Ndarray.map (fun _ -\u003e Random.int 256 |\u003e Char.chr) x;;\nFilesystemStore.Array.write store array_node slice x';;\nlet y = FilesystemStore.Array.read store array_node slice Ndarray.Char;;\nassert (Ndarray.equal x' y);;\n```\n### create an array with sharding\n```ocaml\nlet config =\n  {chunk_shape = [5; 3; 5]\n  ;codecs = [`Transpose [2; 0; 1]; `Bytes LE; `Zstd (0, true)]\n  ;index_codecs = [`Bytes BE; `Crc32c]\n  ;index_location = Start};;\n\nlet shard_node = Node.Array.(group_node / \"another\");;\n\nFilesystemStore.Array.create\n  ~codecs:[`ShardingIndexed config]\n  ~shape:[100; 100; 50]\n  ~chunks:[10; 15; 20]\n  Ndarray.Complex32\n  Complex.zero\n  shard_node\n  store;;\n```\n### exploratory functions\n```ocaml\nlet a, g = FilesystemStore.hierarchy store;;\nList.map Node.Array.to_path a;;\n(*- : string list = [\"/some/group/name\"; \"/some/group/another\"] *)\nList.map Node.Group.to_path g;;\n(*- : string list = [\"/\"; \"/some\"; \"/some/group\"] *)\n\nFilesystemStore.Array.reshape store array_node [25; 32; 10];;\n\nlet meta = FilesystemStore.Group.metadata store group_node;;\nMetadata.Group.show meta;; (* pretty prints the contents of the metadata *)\n\nFilesystemStore.Array.exists store shard_node;;\nFilesystemStore.Group.exists store group_node;;\n\nlet a, g = FilesystemStore.Group.children store group_node;;\nList.map Node.Array.to_path a;;\n(*- : string list = [\"/some/group/name\"; \"/some/group/another\"] *)\nList.map Node.Group.to_path g;;\n(*- : string list = [] *)\n\nFilesystemStore.Group.delete store group_node;;\nFilesystemStore.clear store;; (* clears the store *)\nFilesystemStore.Group.rename store group_node \"new_name\";;\nFilesystemStore.Array.rename store anode \"new_name\";;\n```\n\n[1]: https://codecov.io/gh/zoj613/zarr-ml/graph/badge.svg?token=KOOG2Y1SH5\n[2]: https://img.shields.io/github/actions/workflow/status/zoj613/zarr-ml/build-and-test.yml?branch=main\n[3]: https://img.shields.io/github/license/zoj613/zarr-ml\n[4]: https://ocsigen.org/lwt/latest/manual/manual\n[5]: https://zoj613.github.io/zarr-ml\n[6]: https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html\n[7]: https://zoj613.github.io/zarr-ml/zarr/Zarr/index.html#examples\n[8]: https://github.com/ocaml-multicore/eio\n[9]: https://github.com/zoj613/zarr-ml/tree/main/examples/zipstore.ml\n[10]: https://github.com/zoj613/zarr-ml/tree/main/examples/picos_fs_store.ml\n[11]: https://ocaml-multicore.github.io/picos/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzoj613%2Fzarr-ml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzoj613%2Fzarr-ml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzoj613%2Fzarr-ml/lists"}