{"id":23383052,"url":"https://github.com/fast-pack/simdcompressionandintersection","last_synced_at":"2026-06-07T23:03:26.151Z","repository":{"id":12563712,"uuid":"15234128","full_name":"fast-pack/SIMDCompressionAndIntersection","owner":"fast-pack","description":"A C++ library to compress and intersect sorted lists of integers using SIMD instructions ","archived":false,"fork":false,"pushed_at":"2023-07-13T18:31:57.000Z","size":1395,"stargazers_count":431,"open_issues_count":4,"forks_count":59,"subscribers_count":29,"default_branch":"master","last_synced_at":"2025-04-02T23:58:01.268Z","etag":null,"topics":["algorithms","compression","integer-compression","intersection","simd","simd-instructions"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fast-pack.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2013-12-16T18:51:31.000Z","updated_at":"2025-04-02T07:57:43.000Z","dependencies_parsed_at":"2024-01-13T17:29:55.771Z","dependency_job_id":null,"html_url":"https://github.com/fast-pack/SIMDCompressionAndIntersection","commit_stats":null,"previous_names":["fast-pack/simdcompressionandintersection","lemire/simdcompressionandintersection"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FSIMDCompressionAndIntersection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FSIMDCompressionAndIntersection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FSIMDCompressionAndIntersection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FSIMDCompressionAndIntersection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fast-pack","download_url":"https://codeload.github.com/fast-pack/SIMDCompressionAndIntersection/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247299833,"owners_count":20916190,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["algorithms","compression","integer-compression","intersection","simd","simd-instructions"],"created_at":"2024-12-21T22:13:03.841Z","updated_at":"2026-06-07T23:03:26.146Z","avatar_url":"https://github.com/fast-pack.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"SIMDCompressionAndIntersection\n======================\n[![Ubuntu 22.04 CI (GCC 11)](https://github.com/lemire/SIMDCompressionAndIntersection/actions/workflows/ubuntu22.yml/badge.svg)](https://github.com/lemire/SIMDCompressionAndIntersection/actions/workflows/ubuntu22.yml)\n[![VisualStudio](https://github.com/lemire/SIMDCompressionAndIntersection/actions/workflows/visualstudio.yml/badge.svg)](https://github.com/lemire/SIMDCompressionAndIntersection/actions/workflows/visualstudio.yml)\n\n\nAs the name suggests, this is a C/C++ library for fast\ncompression and intersection of lists of sorted integers using\nSIMD instructions. The library focuses on innovative techniques\nand very fast schemes, with particular attention to differential\ncoding. It introduces new SIMD intersections schemes such as\nSIMD Galloping.\n\nThis library can decode at least 4 billions of compressed integers per second on most\ndesktop or laptop processors. That is, it can decompress data at a rate of 15 GB/s.\nThis is significantly faster than generic codecs like gzip, LZO, Snappy or LZ4.\n\nAuthors: Leonid Boystov, Nathan Kurz,  Daniel Lemire,\nOwen Kaser, Andrew Consroe, Shlomi Vaknin, Christoph Rupp, Bradley Grainger, and others.\n\nDocumentation\n-----------------------\n\n* Daniel Lemire, Nathan Kurz, Christoph Rupp, Stream VByte: Faster Byte-Oriented Integer Compression, Information Processing Letters 130, 2018 https://arxiv.org/abs/1709.08990\n* Daniel Lemire, Leonid Boytsov, Nathan Kurz, SIMD Compression and the Intersection of Sorted Integers, Software Practice \u0026 Experience 46 (6), 2016 http://arxiv.org/abs/1401.6399\n* Daniel Lemire and Leonid Boytsov, Decoding billions of integers per second through vectorization, Software Practice \u0026 Experience 45 (1), 2015.  http://arxiv.org/abs/1209.2137 http://onlinelibrary.wiley.com/doi/10.1002/spe.2203/abstract\n* Jeff Plaisance, Nathan Kurz, Daniel Lemire, Vectorized VByte Decoding, International Symposium on Web Algorithms 2015, 2015. http://arxiv.org/abs/1503.07387\n* Wayne Xin Zhao, Xudong Zhang, Daniel Lemire, Dongdong Shan, Jian-Yun Nie, Hongfei Yan, Ji-Rong Wen, A General SIMD-based Approach to Accelerating Compression Algorithms, ACM Transactions on Information Systems 33 (3), 2015. http://arxiv.org/abs/1502.01916\n\nThis work has also inspired other work such as...\n\n* P. Damme, D. Habich, J. Hildebrandt, W. Lehner, Lightweight Data Compression Algorithms: An Experimental Survey (Experiments and Analyses), EDBT 2017 http://openproceedings.org/2017/conf/edbt/paper-146.pdf\n* P. Damme, D. Habich, J. Hildebrandt, W. Lehner, Insights into the Comparative Evaluation of Lightweight Data Compression Algorithms, EDBT 2017 http://openproceedings.org/2017/conf/edbt/paper-414.pdf\n* T. D. Wu, Bitpacking techniques for indexing genomes: I. Hash tables, Algorithms for Molecular Biology 11 (5), 2016. http://almob.biomedcentral.com/articles/10.1186/s13015-016-0069-5\n* Jianguo Wang, Chunbin Lin, Yannis Papakonstantinou, Steven Swanson, An Experimental Study of Bitmap Compression vs. Inverted List Compression, SIGMOD 2017 http://db.ucsd.edu/wp-content/uploads/2017/03/sidm338-wangA.pdf\n\nSimple demo\n------------------------\n\nCheck out example.cpp\n\nYou can run it like so (e.g., under Linux or macOS):\n\n\nUsage (Linux, macOS and similar systems)\n------------------------\n\n```\nmake\n./unit\n```\n\nA static library file is built as `libSIMDCompressionAndIntersection.a` which you can use in your\nown projects along with our header files located in the `include` subdirectory.\n\nYou may also build and run our example:\n\n\n```\nmake example\n./example\n```\n\n\nTo run tests, you can do \n```\n./testcodecs\n```\n\n(follow the instructions)\n\n\nBuilding and installing with CMake\n------------------------\n\nA CMake build is also provided. It works on x86/x64 (SSE/AVX) and on 64-bit ARM\n(the SSE intrinsics are mapped to ARM NEON), and it builds the same library and\ntests as the Makefile.\n\n```\ncmake -S . -B build\ncmake --build build\nctest --test-dir build          # runs the unit tests\n```\n\nTo install the library, headers and a CMake package configuration:\n\n```\ncmake -S . -B build -DCMAKE_INSTALL_PREFIX=/your/prefix -DSIMDCOMP_BUILD_TESTS=OFF\ncmake --build build\ncmake --install build\n```\n\nDownstream CMake projects can then locate it with `find_package` and link the\nimported target:\n\n```cmake\nfind_package(SIMDCompressionAndIntersection REQUIRED)\ntarget_link_libraries(yourapp PRIVATE\n  SIMDCompressionAndIntersection::SIMDCompressionAndIntersection)\n```\n\nThe installed headers live under `\u003cprefix\u003e/include/SIMDCompressionAndIntersection`\nand are added to your include path by the imported target, so `#include\n\u003ccodecfactory.h\u003e` works directly. Useful options: `-DSIMDCOMP_BUILD_TESTS=OFF`\nto skip the tests/benchmarks and `-DSIMDCOMP_INSTALL=OFF` to disable install\nrules (handy when consuming the project via `add_subdirectory`).\n\n\nUsage (Windows users)\n------------------------\n\nWindows users wishing to build using Visual Studio should go into a Developer Powershell, which is\naccessible through the menus in the Visual Studio interface, and run the following from the \ndirectory of the project:\n\n```\nnmake -f .\\makefile.vc\n .\\example.exe\n .\\unit.exe\n ```\n\nUnder Windows, the static library is built as the file `simdcomp_a.lib` which you can use in your\nown projects, along with our header files located in the `include` subdirectory.\n\n\nFor a simple C library\n----------------------\n\nThis library is a C++ research library. For something simpler,\nwritten in C, see:\n\nhttps://github.com/lemire/simdcomp\n\n\nComparison with the FastPFOR C++ library\n-----------------------------------------\n\nThe FastPFOR C++ Library available at https://github.com/lemire/FastPFor\nimplements some of the same compression schemes except that\nit is not optimized for the compression of sorted lists of integers.\n\nOther recommended libraries\n---------------------------------\n\n* libvbyte: A fast implementation for varbyte 32bit/64bit integer compression https://github.com/cruppstahl/libvbyte\n* TurboPFor is a C library that offers lots of interesting optimizations. Well worth checking! (GPL license) https://github.com/powturbo/TurboPFor\n* Oroch is a C++ library that offers a usable API (MIT license) https://github.com/ademakov/Oroch\n* CSharpFastPFOR: A C#  integer compression library  https://github.com/Genbox/CSharpFastPFOR\n* JavaFastPFOR: A java integer compression library https://github.com/lemire/JavaFastPFOR\n* Encoding: Integer Compression Libraries for Go https://github.com/zhenjl/encoding\n* FrameOfReference is a C++ library dedicated to frame-of-reference (FOR) compression: https://github.com/lemire/FrameOfReference\n\nLicensing\n------------------------\n\nApache License, Version 2.0\n\nAs far as the authors know, this work is patent-free.\n\nRequirements\n------------------------\n\nOn x86/x64, a CPU (AMD or Intel) with support for SSE2 (Pentium 4 or better) is\nrequired while a CPU with SSE 4.1* (Penryn  [2007] processors or better) is\nrecommended.\n\nOn 64-bit ARM (AArch64, e.g. Apple Silicon and ARM servers), the SSE intrinsics\nare mapped to ARM NEON via `include/neon_sse.h`, so no x86 hardware is needed.\nNEON is baseline on AArch64, so no special compiler flag is required.\n\nA recent GCC (4.7 or better), Clang, Intel or Visual C++ compiler.\n\nOn x86, a processor supporting AVX (Intel or AMD) is assumed by the default\nmakefile (but AVX is not required, see below).\n\nTested on Linux, MacOS and Windows, on both x64 and ARM64. It should be portable\nto other platforms.\n\n*- On x86, the default makefile might assume AVX support, but AVX is not\nrequired. For GCC compilers, you might need the -msse2 flag, but you will not\nneed the -mavx flag. On ARM64 the makefile automatically drops the x86 `-mavx`\nflag.\n\nFor advanced benchmarking, please see\n\nadvancedbenchmarking/README.md\n\nwhere there is additional information as well as links to real data sets.\n\n\nAcknowledgement\n------------------------\n\nThanks to Kelly Sommers for useful feedback.\n\nThis work was supported by NSERC grant number 26143.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffast-pack%2Fsimdcompressionandintersection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffast-pack%2Fsimdcompressionandintersection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffast-pack%2Fsimdcompressionandintersection/lists"}