{"id":25596843,"url":"https://github.com/nanoporetech/vbz_compression","last_synced_at":"2025-07-12T23:32:21.348Z","repository":{"id":40953939,"uuid":"190373945","full_name":"nanoporetech/vbz_compression","owner":"nanoporetech","description":"VBZ compression plugin for nanopore signal data","archived":false,"fork":false,"pushed_at":"2025-04-07T08:48:16.000Z","size":4059,"stargazers_count":40,"open_issues_count":22,"forks_count":9,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-05-26T04:14:52.272Z","etag":null,"topics":["compression","hdf5","nanopore"],"latest_commit_sha":null,"homepage":"https://nanoporetech.com/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nanoporetech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-06-05T10:20:47.000Z","updated_at":"2025-04-07T08:48:20.000Z","dependencies_parsed_at":"2025-02-28T14:16:26.082Z","dependency_job_id":"f739a5f6-dcaa-482b-b737-528cef03da49","html_url":"https://github.com/nanoporetech/vbz_compression","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/nanoporetech/vbz_compression","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nanoporetech%2Fvbz_compression","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nanoporetech%2Fvbz_compression/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nanoporetech%2Fvbz_compression/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nanoporetech%2Fvbz_compression/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nanoporetech","download_url":"https://codeload.github.com/nanoporetech/vbz_compression/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nanoporetech%2Fvbz_compression/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265067552,"owners_count":23706297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","hdf5","nanopore"],"created_at":"2025-02-21T12:34:44.385Z","updated_at":"2025-07-12T23:32:21.330Z","avatar_url":"https://github.com/nanoporetech.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Oxford Nanopore Technologies logo](images/ONT_logo_590x106.png)\n\nVBZ Compression\n===============\n\nVBZ Compression uses variable byte integer encoding to compress nanopore signal data and is built using the following libraries:\n\n  - https://github.com/lemire/streamvbyte\n  - https://github.com/facebook/zstd\n\nThe performance of VBZ is achieved by taking advantage of the properties of the raw signal and therefore is most effective when applied to the signal dataset. Other datasets you may have in your Fast5 files will not be able to take advantage of the default VBZ settings for compression. VBZ will be used as the default compression scheme in a future release of MinKNOW.\n\nInstallation\n------------\n\nSee the [release](https://github.com/nanoporetech/vbz_compression/releases) section to find the installers for the hdf5 plugin.\n\nPost installation you can then use `HDFView`, `h5repack` or `h5py` as you normally would:\n\n```bash\n# Invoke h5repack to pack input.fast5 into output.fast5\n#\n# The integer values specify how the data is packed:\n#   - 32020: The id of the filter to apply (vbz in this case)\n#   - 5: The number of following arguments\n#   - 0: Filter flag for configuring filter version\n#   - 0: Padding value for configuring filter version\n#   - 2: Packing integers of size 2 bytes\n#   - 1: Use zig zag encoding\n#   - 1: Use zstd compression level 1\n\u003e h5repack -f UD=32020,5,0,0,2,1,1 input.fast5 output.fast5\n\n# To compress 4 byte unsigned integers (no zig zag) with level 3 zstd you could use:\n\u003e h5repack -f UD=32020,5,0,0,4,0,3 input.h5 output.h5\n\n# Invoke h5repack recursively on all reads using 10 processes\n\u003e find . -name \"*.fast5\" | xargs -P 10 -I % h5repack -f UD=32020,5,0,0,2,1,1 % %.vbz\n\n# Invoke h5repack recursively on all reads storing the results inplace using 10 processes\n\u003e find . -name \"*.fast5\" | xargs -P 10 -I % sh -c \"h5repack -f UD=32020,5,0,0,2,1,1 % %.vbz \u0026\u0026 mv %.vbz %\"\n```\n\nBenchmarks\n----------\n\nVBZ outperforms GZIP in both CPU time (\u003e10X compression, \u003e5X decompression) and compression (\u003e30%).\n\n![Compression Ratio](images/vbz_compression_ratio.png)\n![Compression Performance](images/vbz_x86_compression.png)\n![Decompression Performance](images/vbz_x86_decompression.png)\n\n\nDevelopment\n-----------\n\nTo develop the plugin without conan you need the following installed:\n\n- cmake 3.11 (https://cmake.org/)\n\nand the following c++ dependencies\n\n- zstd development libraries available to cmake\n- hdf5 development libraries available to cmake (required for testing)\n\nThe following ubuntu packages provide these libraries:\n  - libhdf5-dev\n  - libzstd-dev\n\nThen configure the project using:\n\n```bash\n\u003e git submodule update --init\n\u003e mkdir build\n\u003e cd build\n\u003e cmake -D CMAKE_BUILD_TYPE=Release -D ENABLE_CONAN=OFF -D ENABLE_PERF_TESTING=OFF -D ENABLE_PYTHON=OFF ..\n\u003e make -j\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnanoporetech%2Fvbz_compression","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnanoporetech%2Fvbz_compression","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnanoporetech%2Fvbz_compression/lists"}