{"id":32141166,"url":"https://github.com/pgzip/pgzip","last_synced_at":"2026-02-21T17:03:18.954Z","repository":{"id":42128838,"uuid":"405403841","full_name":"pgzip/pgzip","owner":"pgzip","description":"A multi-threading implement of Python gzip module","archived":false,"fork":false,"pushed_at":"2025-12-20T14:44:57.000Z","size":139,"stargazers_count":60,"open_issues_count":8,"forks_count":10,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-12-22T17:44:55.067Z","etag":null,"topics":["compression","gzip","python","zip"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/pgzip/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pgzip.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-11T14:43:03.000Z","updated_at":"2025-08-03T01:39:53.000Z","dependencies_parsed_at":"2024-04-12T21:45:33.100Z","dependency_job_id":null,"html_url":"https://github.com/pgzip/pgzip","commit_stats":{"total_commits":61,"total_committers":10,"mean_commits":6.1,"dds":"0.42622950819672134","last_synced_commit":"a341e13a15e2ba751cc91bb2f5730762f0b61ea1"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/pgzip/pgzip","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgzip%2Fpgzip","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgzip%2Fpgzip/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgzip%2Fpgzip/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgzip%2Fpgzip/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pgzip","download_url":"https://codeload.github.com/pgzip/pgzip/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pgzip%2Fpgzip/sbom","scorecard":{"id":730321,"data":{"date":"2025-08-11","repo":{"name":"github.com/pgzip/pgzip","commit":"a341e13a15e2ba751cc91bb2f5730762f0b61ea1"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.4,"checks":[{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":1,"reason":"Found 5/30 approved changesets -- score normalized to 1","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Info: jobLevel 'actions' permission set to 'read': .github/workflows/codeql-analysis.yml:28","Info: jobLevel 'contents' permission set to 'read': .github/workflows/codeql-analysis.yml:29","Warn: no topLevel permission defined: .github/workflows/codeql-analysis.yml:1","Warn: no topLevel permission defined: .github/workflows/python-publish.yml:1","Warn: no topLevel permission defined: .github/workflows/python-tests.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Pinned-Dependencies","score":1,"reason":"dependency not pinned by hash detected -- score normalized to 1","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/codeql-analysis.yml:42: update your workflow using https://app.stepsecurity.io/secureworkflow/pgzip/pgzip/codeql-analysis.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/codeql-analysis.yml:46: update your workflow using https://app.stepsecurity.io/secureworkflow/pgzip/pgzip/codeql-analysis.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/codeql-analysis.yml:57: update your workflow using https://app.stepsecurity.io/secureworkflow/pgzip/pgzip/codeql-analysis.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/codeql-analysis.yml:71: update your workflow using https://app.stepsecurity.io/secureworkflow/pgzip/pgzip/codeql-analysis.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-publish.yml:20: update your workflow using https://app.stepsecurity.io/secureworkflow/pgzip/pgzip/python-publish.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-publish.yml:22: update your workflow using https://app.stepsecurity.io/secureworkflow/pgzip/pgzip/python-publish.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-tests.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/pgzip/pgzip/python-tests.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-tests.yml:21: update your workflow using https://app.stepsecurity.io/secureworkflow/pgzip/pgzip/python-tests.yml/master?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/python-publish.yml:27","Warn: pipCommand not pinned by hash: .github/workflows/python-publish.yml:28","Warn: pipCommand not pinned by hash: .github/workflows/python-tests.yml:26","Warn: pipCommand not pinned by hash: .github/workflows/python-tests.yml:27","Info:   0 out of   8 GitHub-owned GitHubAction dependencies pinned","Info:   1 out of   1 third-party GitHubAction dependencies pinned","Info:   0 out of   4 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":7,"reason":"SAST tool detected but not run on all commits","details":["Info: SAST configuration detected: CodeQL","Warn: 0 commits out of 9 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-22T14:13:26.118Z","repository_id":42128838,"created_at":"2025-08-22T14:13:26.118Z","updated_at":"2025-08-22T14:13:26.118Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29688216,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-21T15:51:39.154Z","status":"ssl_error","status_checked_at":"2026-02-21T15:49:03.425Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","gzip","python","zip"],"created_at":"2025-10-21T06:11:18.392Z","updated_at":"2026-02-21T17:03:18.949Z","avatar_url":"https://github.com/pgzip.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"👷👷👷 Maintainers Wanted 👷👷👷 See https://github.com/pgzip/pgzip/issues/37\n\n# pgzip\n\n[![Run tests](https://github.com/pgzip/pgzip/actions/workflows/python-tests.yml/badge.svg)](https://github.com/pgzip/pgzip/actions/workflows/python-tests.yml)\n[![CodeQL](https://github.com/pgzip/pgzip/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/pgzip/pgzip/actions/workflows/codeql-analysis.yml)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"pgzip_logo.png\" /\u003e\n\u003c/p\u003e\n\n`pgzip` is a multi-threaded `gzip` implementation for `python` that increases the compression and decompression performance.\n\nCompression and decompression performance gains are made by parallelizing the usage of block indexing within a `gzip` file. Block indexing utilizes gzip's `FEXTRA` feature which records the index of compressed members. `FEXTRA` is defined in the official `gzip` specification starting at version 4.3. Because `FEXTRA` is part of the `gzip` specification, `pgzip` is compatible with regular `gzip` files.\n\n`pgzip` is **~25X** faster for compression and **~7X** faster for decompression when benchmarked on a 24 core machine. Performance is limited only by I/O and the `python` interpreter.\n\nTheoretically, the compression and decompression speed should be linear with the number of cores available. However, I/O and a language's general performance limits the compression and decompression speed in practice.\n\n## Usage and Examples\n\n### CLI\n\n```\n❯ python -m pgzip -h\nusage: __main__.py [-h] [-o OUTPUT] [-f FILENAME] [-d] [-l {0-9}] [-t THREADS] input\n\npositional arguments:\n  input                 Input file or '-' for stdin\n\noptions:\n  -h, --help            show this help message and exit\n  -o OUTPUT, --output OUTPUT\n                        Output file or '-' for stdout (Default: Input file with 'gz' extension or stdout)\n  -f FILENAME, --filename FILENAME\n                        Name for the original file when compressing\n  -d, --decompress      Decompress instead of compress\n  -l {0-9}, --compression-level {0-9}\n                        Compression level; 0 = no compression (Default: 9)\n  -t THREADS, --threads THREADS\n                        Number of threads to use (Default: Determine automatically)\n```\n\n### Programatically\n\nUsing `pgzip` is the same as using the built-in `gzip` module.\n\nCompressing data and writing it to a file:\n\n```python\nimport pgzip\n\ns = \"a big string...\"\n\n# An explanation of parameters:\n# `thread=8` - Use 8 threads to compress. `None` or `0` uses all cores (default)\n# `blocksize=2*10**8` - Use a compression block size of 200MB\nwith pgzip.open(\"test.txt.gz\", \"wt\", thread=8, blocksize=2*10**8) as fw:\n    fw.write(s)\n```\n\nDecompressing data from a file:\n\n```python\nimport pgzip\n\ns = \"a big string...\"\n\nwith pgzip.open(\"test.txt.gz\", \"rt\", thread=8) as fr:\n    assert fr.read(len(s)) == s\n```\n\n## Performance\n\n### Compression Performance\n\n![Compression Performance](CompressionBenchmark.png)\n\n### Decompression Performance\n\n![Decompression Performance](DecompressionBenchmark.png)\n\nDecompression was benchmarked using an 8.0GB `FASTQ` text file with 48 threads across 24 cores on a machine with Xeon(R) E5-2650 v4 @ 2.20GHz CPUs.\n\nThe compressed file used in this benchmark was created with a blocksize of 200MB.\n\n## Warning\n\n`pgzip` only replaces the following methods of `gzip`'s `GzipFile` class:\n\n- `open()`\n- `compress()`\n- `decompress()`\n\nOther class methods and functionality have not been well tested.\n\nContributions or improvements is appreciated for methods such as:\n\n- `seek()`\n- `tell()`\n\n## History\n\nCreated initially by Vincent Li (@vinlyx), this project is a fork of [https://github.com/vinlyx/mgzip](https://github.com/vinlyx/mgzip). We had several bug fixes to implement, but we could not contact them. The `pgzip` team would like to thank Vincent Li (@vinlyx) for their hard work. We hope that they will contact us when they discover this project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpgzip%2Fpgzip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpgzip%2Fpgzip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpgzip%2Fpgzip/lists"}