{"id":25895535,"url":"https://github.com/diamondlightsource/hdf5filters","last_synced_at":"2025-03-02T22:31:52.103Z","repository":{"id":147427237,"uuid":"89781867","full_name":"DiamondLightSource/hdf5filters","owner":"DiamondLightSource","description":"A collection of HDF5 filters, wrapped in a cmake build system","archived":false,"fork":false,"pushed_at":"2024-07-15T17:55:30.000Z","size":122,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-07-16T13:44:17.143Z","etag":null,"topics":["compression","from-dls-controls","hdf5","lz4"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DiamondLightSource.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-29T11:42:40.000Z","updated_at":"2024-07-15T17:55:31.000Z","dependencies_parsed_at":"2023-06-13T21:09:26.694Z","dependency_job_id":null,"html_url":"https://github.com/DiamondLightSource/hdf5filters","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fhdf5filters","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fhdf5filters/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fhdf5filters/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiamondLightSource%2Fhdf5filters/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DiamondLightSource","download_url":"https://codeload.github.com/DiamondLightSource/hdf5filters/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241582515,"owners_count":19985845,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","from-dls-controls","hdf5","lz4"],"created_at":"2025-03-02T22:31:51.637Z","updated_at":"2025-03-02T22:31:52.093Z","avatar_url":"https://github.com/DiamondLightSource.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/DiamondLightSource/hdf5filters.svg?branch=master)](https://travis-ci.org/DiamondLightSource/hdf5filters)\n\nhdf5filters\n===========\n\nA collection of HDF5 compression filters, wrapped in a cmake build system.\n\nThe implementation of the compression algorithms and filter code are imported\nfrom various open-source projects, without modifications.\n\nThe purpose of this module is to pull together a selection of high performance\ncompression algorithms with implementations of [HDF5 dynamically loadable filters](https://support.hdfgroup.org/HDF5/doc/Advanced/DynamicallyLoadedFilters/)\nand provide a sensible (cmake based) build system.\n\nLZ4\n---\n\nExtremely Fast Compression algorithm: http://www.lz4.org\n\nCode: https://github.com/lz4/lz4\n\nLog:\n\n| Date           | version | SHA                                      | \n| -------------- | ------- | ---------------------------------------- |\n| 29 April 2017  | v1.7.5  | 7bb64ff2b69a9f8367de9ab483cdadf42b4c1b65 |\n\n\nh5lzfilter\n----------\n\nDynamically loadable HDF5 filter using LZ4 compression.\n\nCode: https://github.com/nexusformat/HDF5-External-Filter-Plugins/tree/master/LZ4/src\n\nLog:\n\n| Date           | version | SHA                                      | \n| -------------- | ------- | ---------------------------------------- |\n| 29 April 2017  | N/A     | 863db280bcb3a120849bcedd75426af6f55dce12 |\n\n\nbitshuffle\n----------\n\nFilter for improving compression of typed binary data. Includes implementation\nof a HDF5 dynamically loadable filter.\n\nCode: https://github.com/kiyo-masui/bitshuffle\n\nLog:\n\n| Date           | version    | SHA                                      | \n| -------------- | ---------- | ---------------------------------------- |\n| 29 April 2017  | 0.3.3.dev1 | 762e5d7ef27ccc3d975546cc281609fb6464b563 |\n\n\nBlosc\n-----\n\nFilter utilizing the Blosc meta compressor. Blosc is an external dependency\nand this module only includes the HDF5 dynamically loadable filter code.\nBlosc combines a number of popular compression algorithms like LZ4, Snappy,\nzlib, etc and enable multi-threaded and highly optimized use of these.\n\nCode: https://github.com/blosc/hdf5-blosc\n\nLog:\n\n| Date           | version    | SHA                                      | \n| -------------- | ---------- | ---------------------------------------- |\n| 12 July 2018   | N/A        | efa7653f0735cd03e7e9efb94f3ebcbcbec42889 |\n\n\nBuild and install\n=================\n\nRequirements: HDF5 C library and headers version \u003e= 1.8.11\n\nRecommendation: build on/for a processor with Intel AVX2 support for best \nperformance. The following flags can be used to control what optimisations are \nenabled:\n\n* CMAKE_BUILD_TYPE=Release will enable compiler optimizations: -O3\n* CMAKE_BUILD_TYPE=RelWithDebInfo will enable compiler optimizations: -O2\n* USE_SSE2=ON will enable SSE2 extensions: -msse2 (default: ON)\n* USE_AVX2=ON will enable AVX2 extensions: -mavx2 (default: OFF - use gcc \u003e= 4.8)\n\n```\ncd hdf5filters\nmkdir -p cmake-build\ncd cmake-build\ncmake -DHDF5_ROOT=/path/to/hdf5/installation/ \\\n      -DBLOSC_ROOT_DIR=/path/to/blosc/installation \\\n      -DCMAKE_INSTALL_PREFIX=/path/to/install/destination \\\n      -DCMAKE_BUILD_TYPE=Release \\ \n      -DUSE_AVX2=ON \\\n      ..\nmake\nmake install\n```\n\nThe compression libs are installed into the `CMAKE_INSTALL_PREFIX/lib` dir and the HDF5 filters\nare installed into the `CMAKE_INSTALL_PREFIX/h5plugin` dir.\n\nUsage\n=====\n\nSet the environment variable HDF5_PLUGIN_PATH to point to the `CMAKE_INSTALL_PREFIX/h5plugin` \ndir. Use semicolon to list multiple dirs. Then use HDF5 (\u003e= 1.8.11) applications as\nper usual and the filters will automatically be loaded when required.\n\n```\nexport HDF5_PLUGIN_PATH=/path/to/hdf5filters/installation/h5plugin\n```\n\nUsing the HDF5 tools with filters\n---------------------------------\n\nThe HDF5 (\u003e=1.8.12) tool h5repack can be used to decompress a compressed dataset.\n\nFirst check out the '/data' dataset in compressed.h5 and notice the\nFILTER_ID = 320004 is the HDF5 lz4 filter:\n\n```\nh5dump -Hp compressed.h5 \nHDF5 \"compressed.h5\" {\nGROUP \"/\" {\n   DATASET \"data\" {\n      DATATYPE  H5T_STD_I64LE\n      DATASPACE  SIMPLE { ( 1000 ) / ( 1000 ) }\n      STORAGE_LAYOUT {\n         CHUNKED ( 1000 )\n         SIZE 4018\n      }\n      FILTERS {\n         UNKNOWN_FILTER {\n            FILTER_ID 32004\n            \n            COMMENT HDF5 lz4 filter; see http://www.hdfgroup.org/services/contributions.html\n         }\n      }\n      FILLVALUE {\n         FILL_TIME H5D_FILL_TIME_ALLOC\n         VALUE  0\n      }\n      ALLOCATION_TIME {\n         H5D_ALLOC_TIME_INCR\n      }\n   }\n}\n}\n```\n\nTo decompress the '/data' dataset in the file compressed.h5 to bloated.h5:\n\n\n```\nh5repack -f data:NONE compressed.h5 bloated.h5\n```\n\nNow check the result:\n\n```h5dump -Hp bloated.h5 \nHDF5 \"bloated.h5\" {\nGROUP \"/\" {\n   DATASET \"data\" {\n      DATATYPE  H5T_STD_I64LE\n      DATASPACE  SIMPLE { ( 1000 ) / ( 1000 ) }\n      STORAGE_LAYOUT {\n         CHUNKED ( 1000 )\n         SIZE 8000\n      }\n      FILTERS {\n         NONE\n      }\n      FILLVALUE {\n         FILL_TIME H5D_FILL_TIME_ALLOC\n         VALUE  0\n      }\n      ALLOCATION_TIME {\n         H5D_ALLOC_TIME_INCR\n      }\n   }\n}\n}\n```\n\nCompressing with blosc and h5repack\n------------------------------------\n\nFor a test you might want to try to compress a raw, uncompressed data file from an experiment\nin order to get an idea of what level of compression can be achieved:\n\nThe input data is an uncompressed 3D stack of images in a chunked dataset:\n\n```\nh5dump -pH input.h5 \nHDF5 \"input.h5\" {\nGROUP \"/\" {\n   DATASET \"data\" {\n      DATATYPE  H5T_STD_U16LE\n      DATASPACE  SIMPLE { ( 100, 1536, 2048 ) / ( H5S_UNLIMITED, 1536, 2048 ) }\n      STORAGE_LAYOUT {\n         CHUNKED ( 1, 1536, 2048 )\n         SIZE 629145600\n      }\n      FILTERS {\n         NONE\n      }\n      FILLVALUE {\n         FILL_TIME H5D_FILL_TIME_IFSET\n         VALUE  0\n      }\n      ALLOCATION_TIME {\n         H5D_ALLOC_TIME_INCR\n      }\n   }\n}\n}\n```\n\nCompress and check the output file.\n\nNote that 32001 is the Blosc User Defined (UD) filter code:\n\n```\nexport HDF5_PLUGIN_PATH=/dls_sw/prod/tools/RHEL7-x86_64/hdf5filters/0-6-1/prefix/avx2-hdf5_1.10/h5plugin\nh5repack -L -f UD=32001,0,7,0,0,0,0,1,1,1 input.h5 out.h5\nh5dump -pH out.h5 \nHDF5 \"out.h5\" {\nGROUP \"/\" {\n   DATASET \"data\" {\n      DATATYPE  H5T_STD_U16LE\n      DATASPACE  SIMPLE { ( 100, 1536, 2048 ) / ( H5S_UNLIMITED, 1536, 2048 ) }\n      STORAGE_LAYOUT {\n         CHUNKED ( 1, 1536, 2048 )\n         SIZE 8839621 (71.173:1 COMPRESSION)       \u003c== decent compression ratio\n      }\n      FILTERS {\n         USER_DEFINED_FILTER {\n            FILTER_ID 32001\n            COMMENT blosc\n            PARAMS { 2 2 2 6291456 1 1 1 }        \u003c== blosc filter parameters\n         }\n      }\n      FILLVALUE {\n         FILL_TIME H5D_FILL_TIME_IFSET\n         VALUE  0\n      }\n      ALLOCATION_TIME {\n         H5D_ALLOC_TIME_INCR\n      }\n   }\n}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiamondlightsource%2Fhdf5filters","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdiamondlightsource%2Fhdf5filters","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiamondlightsource%2Fhdf5filters/lists"}