{"id":23268398,"url":"https://github.com/fast-pack/maskedvbyte","last_synced_at":"2026-06-07T23:01:51.669Z","repository":{"id":23502425,"uuid":"26868077","full_name":"fast-pack/MaskedVByte","owner":"fast-pack","description":"Fast decoder for VByte-compressed integers","archived":false,"fork":false,"pushed_at":"2024-05-20T15:26:33.000Z","size":74,"stargazers_count":119,"open_issues_count":0,"forks_count":19,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-12-18T21:05:48.115Z","etag":null,"topics":["compression","integer-compression","simd","simd-instructions","vbyte","vbyte-compressed-integers"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fast-pack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-11-19T15:38:44.000Z","updated_at":"2024-12-18T14:38:38.000Z","dependencies_parsed_at":"2024-10-26T20:28:58.307Z","dependency_job_id":"07bf35eb-079b-4f43-bb64-2616c458cafe","html_url":"https://github.com/fast-pack/MaskedVByte","commit_stats":null,"previous_names":["fast-pack/maskedvbyte"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FMaskedVByte","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FMaskedVByte/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FMaskedVByte/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FMaskedVByte/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fast-pack","download_url":"https://codeload.github.com/fast-pack/MaskedVByte/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230471175,"owners_count":18231193,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","integer-compression","simd","simd-instructions","vbyte","vbyte-compressed-integers"],"created_at":"2024-12-19T17:09:09.243Z","updated_at":"2026-06-07T23:01:51.663Z","avatar_url":"https://github.com/fast-pack.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"MaskedVByte\n===========\n[![Ubuntu](https://github.com/fast-pack/MaskedVByte/actions/workflows/ubuntu.yml/badge.svg)](https://github.com/fast-pack/MaskedVByte/actions/workflows/ubuntu.yml)\n\nFast, vectorized VByte decoding for 32‑bit integers in C, with optional differential (delta) coding.\n\n- Runs on x86-64 with SSE4.1 (available on virtually all modern x64 CPUs) and on 64-bit ARM (AArch64) such as Apple Silicon and AWS Graviton, where it uses NEON via `include/sse_to_neon.h`\n- C99 compatible\n- The build systems select the right SIMD flags automatically: `-msse4.1` on x86-64, no extra flag on AArch64 (NEON is part of the ARMv8 baseline)\n\n\nBuild and test\n--------------\n\n```sh\nmake        # builds the library and the test binary\n./unit      # runs a quick correctness test\n```\n\nCMake build (alternative)\n------------------------\n\n```sh\nmkdir -p build\ncmake -S . -B build -DCMAKE_BUILD_TYPE=Release \\\n      -DMASKEDVBYTE_BUILD_TESTS=ON \\\n      -DMASKEDVBYTE_BUILD_EXAMPLES=ON\ncmake --build build -j\nctest --test-dir build --output-on-failure   # optional\n\n# run the example built by CMake\n./build/example\n```\n\nInstall with CMake (optional):\n\n```sh\ncmake --install build --prefix /usr/local\n```\n\nBuild and run the example\n-------------------------\n\n```sh\nmake example\n./example\n```\n\nYou should see something like:\n\n```\nCompressed 5000 integers down to 5000 bytes.\n```\n\nEmbedded example, explained\n---------------------------\nThe example allocates input/output buffers, encodes a flat array of integers with classic VByte, then decodes it back with the masked (vectorized) decoder and verifies the sizes match.\n\n```c\n#include \u003cstdio.h\u003e\n#include \u003cstdlib.h\u003e\n#include \u003cassert.h\u003e\n\n#include \"varintencode.h\"\n#include \"varintdecode.h\"\n\nint main() {\n            int N = 5000;\n            uint32_t * datain = malloc(N * sizeof(uint32_t));\n            uint8_t * compressedbuffer = malloc(N * sizeof(uint32_t));\n            uint32_t * recovdata = malloc(N * sizeof(uint32_t));\n            for (int k = 0; k \u003c N; ++k)\n                        datain[k] = 120; // constant value fits in one VByte\n            size_t compsize = vbyte_encode(datain, N, compressedbuffer); // encoding\n            // result is stored in 'compressedbuffer' using 'compsize' bytes\n            size_t compsize2 = masked_vbyte_decode(compressedbuffer, recovdata, N); // fast decoding\n            assert(compsize == compsize2); // sanity check\n            free(datain);\n            free(compressedbuffer);\n            free(recovdata);\n            printf(\"Compressed %d integers down to %d bytes.\\n\", N, (int)compsize);\n            return 0;\n}\n```\n\nWhat’s happening:\n- VByte uses a continuation bit; small values like 120 encode to a single byte, so 5000 values compress to 5000 bytes.\n- `masked_vbyte_decode` is a vectorized decoder using SSE4.1 (x86-64) or NEON (AArch64) for speed.\n- Differential coding variants are available when your data is sorted or has small gaps.\n\nAPI at a glance\n---------------\nHeaders are in `include/`.\n\n- Encoding\n      - `size_t vbyte_encode(const uint32_t* in, size_t length, uint8_t* bout);`\n      - `size_t vbyte_encode_delta(const uint32_t* in, size_t length, uint8_t* bout, uint32_t prev);`\n\n- Decoding\n      - `size_t masked_vbyte_decode(const uint8_t* in, uint32_t* out, uint64_t length);`\n      - `size_t masked_vbyte_decode_delta(const uint8_t* in, uint32_t* out, uint64_t length, uint32_t prev);`\n      - `size_t masked_vbyte_decode_fromcompressedsize(const uint8_t* in, uint32_t* out, size_t inputsize);`\n      - `size_t masked_vbyte_decode_fromcompressedsize_delta(const uint8_t* in, uint32_t* out, size_t inputsize, uint32_t prev);`\n      - Random access helpers for delta streams:\n            - `uint32_t masked_vbyte_select_delta(const uint8_t *in, uint64_t length, uint32_t prev, size_t slot);`\n            - `int masked_vbyte_search_delta(const uint8_t *in, uint64_t length, uint32_t prev, uint32_t key, uint32_t *presult);`\n\nTips\n----\n- Prefer delta coding when your sequence is sorted or has small differences; it often reduces the number of bytes per integer.\n- If you know the compressed byte length, use the `*_fromcompressedsize` functions to decode exactly that many bytes.\n\n\nUse from your CMake project\n---------------------------\n\nAfter installation (see above):\n\n```cmake\nfind_package(maskedvbyte CONFIG REQUIRED)\ntarget_link_libraries(your_target PRIVATE maskedvbyte::maskedvbyte)\n```\n\nOr as a subdirectory (vendored):\n\n```cmake\nadd_subdirectory(path/to/MaskedVByte)\ntarget_link_libraries(your_target PRIVATE maskedvbyte::maskedvbyte)\n```\n\n\nInteresting applications \n-----------------------\n\n- [Greg Bowyer has integrated Masked VByte into Lucene, for higher speeds](https://github.com/GregBowyer/lucene-solr/tree/intrinsics).\n- Our fast function is also used by [PISA: Performant Indexes and Search for Academia](https://github.com/pisa-engine/pisa).\n\nReference\n-------------\n\n* Daniel Lemire, Nathan Kurz, Christoph Rupp, Stream VByte: Faster Byte-Oriented Integer Compression, Information Processing Letters 130, February 2018, Pages 1-6 https://arxiv.org/abs/1709.08990\n* Jeff Plaisance, Nathan Kurz, Daniel Lemire, Vectorized VByte Decoding,  International Symposium on Web Algorithms 2015, 2015. http://arxiv.org/abs/1503.07387\n\n\nSee also\n------------\n\n* SIMDCompressionAndIntersection: A C++ library to compress and intersect sorted lists of integers using SIMD instructions https://github.com/lemire/SIMDCompressionAndIntersection\n* The FastPFOR C++ library : Fast integer compression https://github.com/lemire/FastPFor\n* High-performance dictionary coding https://github.com/lemire/dictionary\n* LittleIntPacker: C library to pack and unpack short arrays of integers as fast as possible https://github.com/lemire/LittleIntPacker\n* The SIMDComp library: A simple C library for compressing lists of integers using binary packing https://github.com/lemire/simdcomp\n* StreamVByte: Fast integer compression in C using the StreamVByte codec https://github.com/lemire/streamvbyte\n* CSharpFastPFOR: A C#  integer compression library  https://github.com/Genbox/CSharpFastPFOR\n* JavaFastPFOR: A java integer compression library https://github.com/lemire/JavaFastPFOR\n* Encoding: Integer Compression Libraries for Go https://github.com/zhenjl/encoding\n* FrameOfReference is a C++ library dedicated to frame-of-reference (FOR) compression: https://github.com/lemire/FrameOfReference\n* libvbyte: A fast implementation for varbyte 32bit/64bit integer compression https://github.com/cruppstahl/libvbyte\n* TurboPFor is a C library that offers lots of interesting optimizations. Well worth checking! (GPL license) https://github.com/powturbo/TurboPFor\n* Oroch is a C++ library that offers a usable API (MIT license) https://github.com/ademakov/Oroch\n\n\nLicense\n-------\nSee `LICENSE` for details.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffast-pack%2Fmaskedvbyte","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffast-pack%2Fmaskedvbyte","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffast-pack%2Fmaskedvbyte/lists"}