{"id":18104178,"url":"https://github.com/gauteh/hidefix","last_synced_at":"2025-08-10T03:08:35.269Z","repository":{"id":40363945,"uuid":"261138515","full_name":"gauteh/hidefix","owner":"gauteh","description":"🐕 Concurrent HDF5 and NetCDF4 reader","archived":false,"fork":false,"pushed_at":"2024-10-21T13:12:19.000Z","size":8516,"stargazers_count":57,"open_issues_count":9,"forks_count":7,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-07-18T13:07:57.677Z","etag":null,"topics":["concurrency","experimental","hdf5","multi-threaded","netcdf","performance","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gauteh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-04T10:03:00.000Z","updated_at":"2025-06-11T20:18:06.000Z","dependencies_parsed_at":"2024-01-12T04:44:45.207Z","dependency_job_id":"15372ce3-df14-4bcd-a98e-ef15313f8fc6","html_url":"https://github.com/gauteh/hidefix","commit_stats":{"total_commits":315,"total_committers":3,"mean_commits":105.0,"dds":0.06349206349206349,"last_synced_commit":"643e9c68a4b4f55484053a4b738a7c76412a9e9c"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"purl":"pkg:github/gauteh/hidefix","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gauteh%2Fhidefix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gauteh%2Fhidefix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gauteh%2Fhidefix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gauteh%2Fhidefix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gauteh","download_url":"https://codeload.github.com/gauteh/hidefix/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gauteh%2Fhidefix/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269669010,"owners_count":24456582,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-10T02:00:08.965Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","experimental","hdf5","multi-threaded","netcdf","performance","rust"],"created_at":"2024-10-31T22:14:09.263Z","updated_at":"2025-08-10T03:08:35.242Z","avatar_url":"https://github.com/gauteh.png","language":"Rust","readme":"[![Crates.io](https://img.shields.io/crates/v/hidefix.svg)](https://crates.io/crates/hidefix)\n[![PyPI](https://img.shields.io/pypi/v/hidefix.svg)](https://pypi.org/project/hidefix/)\n[![Documentation](https://docs.rs/hidefix/badge.svg)](https://docs.rs/hidefix/)\n[![Build (rust)](https://github.com/gauteh/hidefix/workflows/Rust/badge.svg)](https://github.com/gauteh/hidefix/actions?query=branch%3Amain)\n[![Build (python)](https://github.com/gauteh/hidefix/workflows/Python/badge.svg)](https://github.com/gauteh/hidefix/actions?query=branch%3Amain)\n[![codecov](https://codecov.io/gh/gauteh/hidefix/branch/main/graph/badge.svg)](https://codecov.io/gh/gauteh/hidefix)\n[![Rust nightly](https://img.shields.io/badge/rustc-nightly-orange)](https://rust-lang.github.io/rustup/installation/other.html)\n\n\u003cimg src=\"https://raw.githubusercontent.com/gauteh/hidefix/main/idefix.png\"\u003e\n\n# HIDEFIX\n\nThis Rust and Python library provides an alternative reader for the\n[HDF5](https://support.hdfgroup.org/HDF5/doc/H5.format.html) file or [NetCDF4\nfile](https://www.unidata.ucar.edu/software/netcdf/docs/file_format_specifications.html)\n(which uses HDF5) which supports concurrent access to data. This is achieved by\nbuilding an index of the chunks, allowing a thread to use many file handles to\nread the file. The original (native) HDF5 library is used to build the index,\nbut once it has been created it is no longer needed. The index can be\nserialized to disk so that performing the indexing is not necessary.\n\nIn Rust:\n\n```rust\nuse hidefix::prelude::*;\n\nlet idx = Index::index(\"tests/data/coads_climatology.nc4\").unwrap();\nlet mut r = idx.reader(\"SST\").unwrap();\n\nlet values = r.values::\u003cf32\u003e(None, None).unwrap();\n\nprintln!(\"SST: {:?}\", values);\n```\n\nor with Python using Xarray:\n```python\nimport xarray as xr\nimport hidefix\n\nds = xr.open_dataset('file.nc', engine='hidefix')\nprint(ds)\n```\n\nSee the [example](examples/) for how to use hidefix for\n[regular](examples/read_hfx_cache.rs), [parallel](examples/read_hfx_parallel.rs) or\n[concurrent](examples/read_hfx_concurrent.rs) reads.\n\n## Motivation\n\nThe HDF5 library requires internal locks to be _thread-safe_ since it relies on\ninternal buffers which cannot be safely accessed/written to from multiple\nthreads. This effectively causes multi-threaded applications to use sequential\nreads, while competing for the locks. And also apparently cause each other\ntrouble, perhaps through dropping cached chunks which other threads still need.\nIt can be safely used from different processes, but that requires potentially\nmuch more overhead than multi-threaded or asynchronous code.\n\n## Some basic benchmarks\n\n`hidefix` is intended to perform better when concurrent reads are made either\nto the same dataset, same file or to different files from a single process. For\nbasic benchmarks the performance is on-par or slightly better compared to doing\nstandard *sequential* reads than the native HDF5 library (through its\n[rust-bindings](https://github.com/aldanor/hdf5-rust)). Where `hidefix` shines\nis once the _multiple threads_ in the _same process_ tries to read in _any way_\nfrom a HDF5 file simultaneously.\n\nThis simple benchmark tries to read a small dataset sequentially or\nconcurrently using the `cached` reader from `hidefix` and the native reader\nfrom HDF5. The dataset is chunked, shuffled and compressed (using gzip):\n\n```sh\n$ cargo bench --bench concurrency -- --ignored\n\ntest shuffled_compressed::cache_concurrent_reads  ... bench:  15,903,406 ns/iter (+/- 220,824)\ntest shuffled_compressed::cache_sequential        ... bench:  59,778,761 ns/iter (+/- 602,316)\ntest shuffled_compressed::native_concurrent_reads ... bench: 411,605,868 ns/iter (+/- 35,346,233)\ntest shuffled_compressed::native_sequential       ... bench: 103,457,237 ns/iter (+/- 7,703,936)\n```\n\n## Inspiration and other projects\n\nThis work is based in part on the [DMR++\nmodule](https://github.com/OPENDAP/bes/tree/master/modules/dmrpp_module) of the\n[OPeNDAP](https://www.opendap.org/) [Hyrax\nserver](https://www.opendap.org/software/hyrax-data-server). The\n[zarr](https://zarr.readthedocs.io/en/stable/) format does something similar,\nand the same approach has been [tested out on\nHDF5](https://medium.com/pangeo/cloud-performant-reading-of-netcdf4-hdf5-data-using-the-zarr-library-1a95c5c92314)\nas swell.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgauteh%2Fhidefix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgauteh%2Fhidefix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgauteh%2Fhidefix/lists"}