{"id":17091991,"url":"https://github.com/fast-pack/PyFastPFor","last_synced_at":"2026-05-26T13:30:16.309Z","repository":{"id":49114304,"uuid":"121713059","full_name":"searchivarius/PyFastPFor","owner":"searchivarius","description":"Python bindings for the fast integer compression library FastPFor.","archived":false,"fork":false,"pushed_at":"2023-10-23T09:23:13.000Z","size":273,"stargazers_count":58,"open_issues_count":2,"forks_count":9,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-18T04:32:54.729Z","etag":null,"topics":["compression-algorithm","compression-schemes","simd-compression","sorted-lists"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/searchivarius.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-16T03:59:23.000Z","updated_at":"2025-02-13T03:50:41.000Z","dependencies_parsed_at":"2022-09-13T05:52:10.738Z","dependency_job_id":"e457ef25-a308-4ea3-8737-c0ed6a83f2d9","html_url":"https://github.com/searchivarius/PyFastPFor","commit_stats":{"total_commits":40,"total_committers":2,"mean_commits":20.0,"dds":"0.42500000000000004","last_synced_commit":"1fe0fde1c1e1c03c46df1fd03b5b8ca41b28a9a4"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/searchivarius%2FPyFastPFor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/searchivarius%2FPyFastPFor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/searchivarius%2FPyFastPFor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/searchivarius%2FPyFastPFor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/searchivarius","download_url":"https://codeload.github.com/searchivarius/PyFastPFor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239986180,"owners_count":19729584,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression-algorithm","compression-schemes","simd-compression","sorted-lists"],"created_at":"2024-10-14T14:00:00.445Z","updated_at":"2026-05-26T13:30:16.269Z","avatar_url":"https://github.com/searchivarius.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![PyPI version](https://img.shields.io/pypi/v/pyfastpfor.svg)](https://pypi.python.org/pypi/pyfastpfor/)\n[![Downloads](https://pepy.tech/badge/pyfastpfor)](https://pepy.tech/project/pyfastpfor)\n# PyFastPFor\nPython bindings for the fast **light-weight** integer compression library [FastPFor](https://github.com/lemire/FastPFor): A research library with integer compression schemes. FastPFor is broadly applicable to the compression of arrays of 32-bit integers where most integers are small. The library seeks to exploit SIMD instructions (SSE) whenever possible. This library can decode at least 4 billions of compressed integers per second on most desktop or laptop processors. That is, it can decompress data at a rate of 15 GB/s. This is significantly faster than generic codecs like gzip, LZO, Snappy or LZ4.\n\n# Authors\n\nDaniel Lemire, Leonid Boytsov, Owen Kaser, Maxime Caron, Louis Dionne, Michel Lemay, Erik Kruus, Andrea Bedini, Matthias Petri, Robson Braga Araujo, Patrick Damme. Bindings are created by Leonid Boytsov.\n\n# Installation\n\nBindings can be installed locally:\n```\ncd python_bindings\npip install -r requirements.txt\nsudo setup.py build install\n```\nor via pip:\n```\npip install pyfastpfor\n```\nDue to some compilation quirks this currently seem to work with GCC only. I will fix it in some not so distant future. You may also need to install Python dev-files. On Ubuntu, for Python 3 you can do it as follows:\n\n```\nsudo apt-get install python3-dev\n```\n\n\n# Documentation\n\nThe library supports all the codecs implemented in the original [FastPFor](https://github.com/lemire/FastPFor) library by July 2023. To get a list of codecs, use the function ``getCodecList``. \n\nTypical light-weight compression does not take context into account and, consequently, works well only for small integers. When integers are large, data differencing is a common trick to make integers small. In particular, we often deal with sorted lists of integers, which can be represented by differences between neighboring numbers. \n\nThe smallest differences (**fine** deltas) are between adjacent numbers. Respective differencing and difference inverting functions are ``delta1'' and ``prefixSum1''.\n\nHowever, we can do reasonably well, we compute differences between numbers that are four positions apart (**coarse** deltas). Such differences can be computed and inverted more efficiently.  Respective differencing and difference inverting functions are ``delta4'' and ``prefixSum4''.\n\nExamples of three common use scenarios (no differencing, coarse and fine deltas) are outlined in [this Python notebook](python_bindings/examples.ipynb). \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffast-pack%2FPyFastPFor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffast-pack%2FPyFastPFor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffast-pack%2FPyFastPFor/lists"}