{"id":16572809,"url":"https://github.com/fast-pack/FrameOfReference","last_synced_at":"2025-03-21T12:31:14.126Z","repository":{"id":31342345,"uuid":"34905043","full_name":"fast-pack/FrameOfReference","owner":"fast-pack","description":"C++ library to pack and unpack vectors of integers having a small range of values using a technique called Frame of Reference ","archived":false,"fork":false,"pushed_at":"2024-02-19T16:40:51.000Z","size":112,"stargazers_count":52,"open_issues_count":3,"forks_count":7,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-16T02:48:53.164Z","etag":null,"topics":["clang","compression","gcc","integer-compression","visual-studio"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fast-pack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-05-01T13:52:44.000Z","updated_at":"2025-02-24T10:27:27.000Z","dependencies_parsed_at":"2024-10-28T10:24:21.718Z","dependency_job_id":"dfca3050-583e-47cc-8e5f-c4f4d38d24b2","html_url":"https://github.com/fast-pack/FrameOfReference","commit_stats":null,"previous_names":["fast-pack/frameofreference"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FFrameOfReference","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FFrameOfReference/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FFrameOfReference/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FFrameOfReference/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fast-pack","download_url":"https://codeload.github.com/fast-pack/FrameOfReference/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244799317,"owners_count":20512232,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clang","compression","gcc","integer-compression","visual-studio"],"created_at":"2024-10-11T21:28:37.710Z","updated_at":"2025-03-21T12:31:13.737Z","avatar_url":"https://github.com/fast-pack.png","language":"C","readme":"# Frame of Reference (FOR) C++ library\n\n## What is this?\n\nC++ library to pack and unpack vectors of integers having a small\nrange of values using a technique called Frame of Reference (Goldstein et al. 1998).\nIt should run fast even though it is written in simple C++.\n\nCode from this library is part [Apache Arrow](https://github.com/apache/arrow)\nand [Apache Impala](https://github.com/cloudera/Impala).\n\n## Code usage :\n\nGiven an array of 32-bit integers, you can compress it as follows:\n\n```C\n#include \"compression.h\"\n\n...\n\nuint32_t * inputdata = ... // length values\nuint32_t * compresseddata = ... // enough data\nuint32_t *out = compress(inputdata, length, compresseddata);\n// compressed data lies between compresseddata and out\nuint32_t nvalue = 0;\nuint32_t * recoverydata = ... // available buffer with at least length elements\nuncompress(compresseddata, recoverydata, nvalue);\n// nvalue will be equal to length\n```\n\nThere is a similar API with ``turbocompress`` and ``turbouncompress`` with the difference\nthat ``compresseddata`` uses an ``uint8_t`` pointer type.\n\n```C\n#include \"turbocompression.h\"\n\n...\n\nuint32_t * inputdata = ... // length values\nuint8_t * compresseddata = ... // enough data\nuint8_t *out = turbocompress(inputdata, length, compresseddata);\n// compressed data lies between compresseddata and out\nuint32_t nvalue = 0;\nuint32_t * recoverydata = ... // available buffer with at least length elements\nturbouncompress(compresseddata, recoverydata, nvalue);\n// nvalue will be equal to length\n```\n\nWe can also compress 64-bit arrays:\n\n```C\n#include \"turbocompression.h\"\n\n...\n\nuint64_t * inputdata = ... // length values\nuint8_t * compresseddata = ... // enough data\nuint8_t *out = turbocompress64(inputdata, length, compresseddata);\n// compressed data lies between compresseddata and out\nuint32_t nvalue = 0;\nuint64_t * recoverydata = ... // available buffer with at least length elements\nturbouncompress64(compresseddata, recoverydata, nvalue);\n// nvalue will be equal to length\n```\n\n##  Usage (with Makefile)\n\nTo run a simple benchmark, do\n\n     make\n     ./test sampledata.txt\n\nwhere sampledata.txt is a text data file with one integer per line.\n\nFor a parallelized version, type\n\n     make testmp\n     ./testmp sampledata.txt\n\nThis requires OpenMP support however.\n\n\n##  Building (with CMake under macOS and Linux)\n\nYou need to have ``cmake`` installed and available as a command.\n\n     mkdir release\n     cd release\n     cmake ..\n     make\n     make test\n\n# Building (Visual Studio under Windows)\n\nWe are assuming that you have a common Windows PC with at least Visual Studio 2015, and an x64 processor.\n\nTo build with at least Visual Studio 2015 from the command line:\n- Grab the FrameOfReference code from GitHub, e.g., by cloning it using [GitHub Desktop](https://desktop.github.com/).\n- Install [CMake](https://cmake.org/download/). When you install it, make sure to ask that ``cmake`` be made available from the command line.\n- Create a subdirectory within FrameOfReference, such as ``VisualStudio``.\n- Using a shell, go to this newly created directory. For example, within GitHub Desktop, you can right-click on  ``FrameOfReference`` in your GitHub repository list, and select ``Open in Git Shell``, then type ``cd VisualStudio`` in the newly created shell.\n- Type ``cmake -DCMAKE_GENERATOR_PLATFORM=x64 ..`` in the shell while in the ``VisualStudio`` repository.\n- This last command created a Visual Studio solution file in the newly created directory (e.g., ``FrameOfReference.sln``). Open this file in Visual Studio. You should now be able to build the project and run the tests. For example, in the ``Solution Explorer`` window (available from the ``View`` menu), right-click ``ALL_BUILD`` and select ``Build``. To test the code, still in the ``Solution Explorer`` window, select ``RUN_TESTS`` and select ``Build``.\n\nTo build with at least Visual Studio 2017 directly in the IDE:\n- Grab the FrameOfReference code from GitHub, e.g., by cloning it using [GitHub Desktop](https://desktop.github.com/).\n- Select the ``Visual C++ tools for CMake`` optional component when installing the C++ Development Workload within Visual Studio.\n- Within Visual Studio use ``File \u003e Open \u003e Folder...`` to open the FrameOfReference folder.\n- Right click on CMakeLists.txt in the parent directory within ``Solution Explorer`` and select ``Build`` to build the project.\n- For testing, in the Standard toolbar, drop the ``Select Startup Item...`` menu and choose one of the tests. Run the test by pressing the button to the left of the dropdown.\n\n\n## Requirements:\n\nThis was tested with GNU G++ and clang++ After suitable adjustments, it should\nbuild under most C++ compilers.\n\n## Other relevant libraries\n\n* Fast decoder for VByte-compressed integers https://github.com/lemire/MaskedVByte\n* Fast integer compression in C using StreamVByte https://github.com/lemire/streamvbyte\n* FastPFOR is a C++ research library well suited to compress unsorted arrays: https://github.com/lemire/FastPFor\n* SIMDCompressionAndIntersection is a C++ research library well suited for sorted arrays (differential coding)\nand computing intersections: https://github.com/lemire/SIMDCompressionAndIntersection\n\n## References\n\n* Daniel Lemire, Nathan Kurz, Christoph Rupp, Stream VByte: Faster Byte-Oriented Integer Compression, Information Processing Letters (to appear) https://arxiv.org/abs/1709.08990\n* Goldstein J, Ramakrishnan R, Shaft U. Compressing relations and indexes. Proceedings of the Fourteenth International Conference on Data Engineering, ICDE ’98, IEEE Computer Society: Washington, DC, USA, 1998; 370–379.\n* Daniel Lemire and Leonid Boytsov, Decoding billions of integers per second through vectorization, Software Practice \u0026 Experience 45 (1), 2015.  http://arxiv.org/abs/1209.2137 http://onlinelibrary.wiley.com/doi/10.1002/spe.2203/abstract\n* Daniel Lemire, Leonid Boytsov, Nathan Kurz, SIMD Compression and the Intersection of Sorted Integers, Software Practice \u0026 Experience 46 (6), 2016. http://arxiv.org/abs/1401.6399\n* Jeff Plaisance, Nathan Kurz, Daniel Lemire, Vectorized VByte Decoding, International Symposium on Web Algorithms 2015, 2015. http://arxiv.org/abs/1503.07387\n* Wayne Xin Zhao, Xudong Zhang, Daniel Lemire, Dongdong Shan, Jian-Yun Nie, Hongfei Yan, Ji-Rong Wen, A General SIMD-based Approach to Accelerating Compression Algorithms, ACM Transactions on Information Systems 33 (3), 2015. http://arxiv.org/abs/1502.01916\n* Jianguo Wang, Chunbin Lin, Yannis Papakonstantinou, Steven Swanson, An Experimental Study of Bitmap Compression vs. Inverted List Compression, SIGMOD 2017 http://db.ucsd.edu/wp-content/uploads/2017/03/sidm338-wangA.pdf\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffast-pack%2FFrameOfReference","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffast-pack%2FFrameOfReference","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffast-pack%2FFrameOfReference/lists"}