{"id":21407292,"url":"https://github.com/mkst/hdf5-aes","last_synced_at":"2025-03-16T17:18:27.668Z","repository":{"id":114862574,"uuid":"278410229","full_name":"mkst/hdf5-aes","owner":"mkst","description":"AES Encryption/Decription Filter for HDF5","archived":false,"fork":false,"pushed_at":"2020-07-12T10:47:22.000Z","size":8,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-23T04:13:15.660Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mkst.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-09T16:02:02.000Z","updated_at":"2023-08-23T12:27:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"9a065d99-68ea-4306-9dae-067a44e5bcfb","html_url":"https://github.com/mkst/hdf5-aes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkst%2Fhdf5-aes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkst%2Fhdf5-aes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkst%2Fhdf5-aes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkst%2Fhdf5-aes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mkst","download_url":"https://codeload.github.com/mkst/hdf5-aes/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243902320,"owners_count":20366262,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-22T16:45:57.335Z","updated_at":"2025-03-16T17:18:27.645Z","avatar_url":"https://github.com/mkst.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HDF5 AES Filter\n\nThis is a work-in-progress HDF5 filter that enables [AES256](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) encryption/decryption of HDF5 datasets.\n\n## Quickstart\n\nAssumes you're running on Ubuntu 20.04 with HDF5 v1.10.4.\n\n### Build\n```sh\n$ docker build . -t aes\n$ mkdir -p build\n$ docker run --rm -ti -v $(pwd):/app aes sh -c \"cd /app/build \u0026\u0026 cmake ../src \u0026\u0026 make\"\n```\n### Run\n```sh\n$ export HDF5_PLUGIN_PATH=$(pwd)/build\n$ python3 examples/dataset.py w your_secret_aes256_key_goes_here\n$ python3 examples/dataset.py r your_secret_aes256_key_goes_here\n```\n\n## Performance\n\nTesting with a 1GB file against `/dev/shm` via the [dataset.py](/examples/dataset.py) example. It takes `~0.35` seconds to create the dataset before the write; this has been subtracted in the times recorded below.\n\n| Filter  | Operation | Time (S) | Notes         |\n|:-------:|:---------:|:--------:|:--------------|\n| None    | Write     | 0.50     |               |\n| None    | Read      | 0.60     |               |\n| **AES** | **Write** | **1.65** |               |\n| **AES** | **Read**  | **0.88** |               |\n| gzip    | Write     | 12.30    |               |\n| gzip    | Read      | 2.52     | Default Level |\n| lzf     | Write     | 2.30     |               |\n| lzf     | Read      | 1.72     |               |\n\nWrites are `~3.3x` slower than with no filter, reads are `~1.5x` slower. If you are already using a compression filter, the impact of also encrypting/decrypting is fairly low.\n\n## Notes\n\nThe 16-byte [IV](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Initialization_vector_(IV)) is created randomly via calls to `rand()` for each *chunk* that is encrypted. Each chunk is prepended with the IV during encryption; this 'dummy' block is then removed as part of the decryption process. This ensures that encrypting the same data with the same key does not generate the same ciphertext.\n\n## TODO\n\n  - [ ] Set KEY via helper function, not environment vars\n  - [ ] Allow encryption flavour to be specified via configuration\n  - [ ] Add support to h5py to support encryption + compression\n\n## Acknowledgements\n\nWith thanks to [derobins](https://github.com/derobins) for their [filter template](https://github.com/derobins/random_hdf5_filters/tree/master/filter_template).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmkst%2Fhdf5-aes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmkst%2Fhdf5-aes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmkst%2Fhdf5-aes/lists"}