{"id":27355634,"url":"https://github.com/fast-pack/pyfastpfor","last_synced_at":"2025-04-12T22:41:14.463Z","repository":{"id":49114304,"uuid":"121713059","full_name":"fast-pack/PyFastPFor","owner":"fast-pack","description":"Python bindings for the fast integer compression library FastPFor.","archived":false,"fork":false,"pushed_at":"2023-10-23T09:23:13.000Z","size":273,"stargazers_count":58,"open_issues_count":1,"forks_count":9,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-04T13:12:21.709Z","etag":null,"topics":["compression-algorithm","compression-schemes","simd-compression","sorted-lists"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fast-pack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-16T03:59:23.000Z","updated_at":"2025-03-23T03:07:51.000Z","dependencies_parsed_at":"2022-09-13T05:52:10.738Z","dependency_job_id":"e457ef25-a308-4ea3-8737-c0ed6a83f2d9","html_url":"https://github.com/fast-pack/PyFastPFor","commit_stats":{"total_commits":40,"total_committers":2,"mean_commits":20.0,"dds":"0.42500000000000004","last_synced_commit":"1fe0fde1c1e1c03c46df1fd03b5b8ca41b28a9a4"},"previous_names":["fast-pack/pyfastpfor"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FPyFastPFor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FPyFastPFor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FPyFastPFor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fast-pack%2FPyFastPFor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fast-pack","download_url":"https://codeload.github.com/fast-pack/PyFastPFor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248643045,"owners_count":21138353,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression-algorithm","compression-schemes","simd-compression","sorted-lists"],"created_at":"2025-04-12T22:41:13.798Z","updated_at":"2025-04-12T22:41:14.457Z","avatar_url":"https://github.com/fast-pack.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![PyPI version](https://img.shields.io/pypi/v/pyfastpfor.svg)](https://pypi.python.org/pypi/pyfastpfor/)\n[![Downloads](https://pepy.tech/badge/pyfastpfor)](https://pepy.tech/project/pyfastpfor)\n# PyFastPFor\nPython bindings for the fast **light-weight** integer compression library [FastPFor](https://github.com/lemire/FastPFor): A research library with integer compression schemes. FastPFor is broadly applicable to the compression of arrays of 32-bit integers where most integers are small. The library seeks to exploit SIMD instructions (SSE) whenever possible. This library can decode at least 4 billions of compressed integers per second on most desktop or laptop processors. That is, it can decompress data at a rate of 15 GB/s. This is significantly faster than generic codecs like gzip, LZO, Snappy or LZ4.\n\n# Authors\n\nDaniel Lemire, Leonid Boytsov, Owen Kaser, Maxime Caron, Louis Dionne, Michel Lemay, Erik Kruus, Andrea Bedini, Matthias Petri, Robson Braga Araujo, Patrick Damme. Bindings are created by Leonid Boytsov.\n\n# Installation\n\nBindings can be installed locally:\n```\ncd python_bindings\npip install -r requirements.txt\nsudo setup.py build install\n```\nor via pip:\n```\npip install pyfastpfor\n```\nDue to some compilation quirks this currently seem to work with GCC only. I will fix it in some not so distant future. You may also need to install Python dev-files. On Ubuntu, for Python 3 you can do it as follows:\n\n```\nsudo apt-get install python3-dev\n```\n\n\n# Documentation\n\nThe library supports all the codecs implemented in the original [FastPFor](https://github.com/lemire/FastPFor) library by July 2023. To get a list of codecs, use the function ``getCodecList``. \n\nTypical light-weight compression does not take context into account and, consequently, works well only for small integers. When integers are large, data differencing is a common trick to make integers small. In particular, we often deal with sorted lists of integers, which can be represented by differences between neighboring numbers. \n\nThe smallest differences (**fine** deltas) are between adjacent numbers. Respective differencing and difference inverting functions are ``delta1'' and ``prefixSum1''.\n\nHowever, we can do reasonably well, we compute differences between numbers that are four positions apart (**coarse** deltas). Such differences can be computed and inverted more efficiently.  Respective differencing and difference inverting functions are ``delta4'' and ``prefixSum4''.\n\nExamples of three common use scenarios (no differencing, coarse and fine deltas) are outlined in [this Python notebook](python_bindings/examples.ipynb). \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffast-pack%2Fpyfastpfor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffast-pack%2Fpyfastpfor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffast-pack%2Fpyfastpfor/lists"}