{"id":17347436,"url":"https://github.com/pr0ps/zipstream-ng","last_synced_at":"2025-04-07T13:05:30.664Z","repository":{"id":40604151,"uuid":"380647295","full_name":"pR0Ps/zipstream-ng","owner":"pR0Ps","description":"A modern and easy to use streamable zip file generator","archived":false,"fork":false,"pushed_at":"2025-01-23T06:35:42.000Z","size":151,"stargazers_count":24,"open_issues_count":5,"forks_count":7,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-31T12:04:19.623Z","etag":null,"topics":["python","streaming","zip"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/zipstream-ng/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pR0Ps.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-27T04:06:40.000Z","updated_at":"2025-03-28T19:09:37.000Z","dependencies_parsed_at":"2025-02-28T08:26:34.579Z","dependency_job_id":null,"html_url":"https://github.com/pR0Ps/zipstream-ng","commit_stats":{"total_commits":39,"total_committers":2,"mean_commits":19.5,"dds":0.05128205128205132,"last_synced_commit":"ddc807ea8050b77baef439c7116a86dbfafeebc8"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pR0Ps%2Fzipstream-ng","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pR0Ps%2Fzipstream-ng/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pR0Ps%2Fzipstream-ng/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pR0Ps%2Fzipstream-ng/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pR0Ps","download_url":"https://codeload.github.com/pR0Ps/zipstream-ng/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247657276,"owners_count":20974344,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","streaming","zip"],"created_at":"2024-10-15T16:48:45.347Z","updated_at":"2025-04-07T13:05:30.641Z","avatar_url":"https://github.com/pR0Ps.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"zipstream-ng\n============\n[![Status](https://github.com/pR0Ps/zipstream-ng/workflows/tests/badge.svg)](https://github.com/pR0Ps/zipstream-ng/actions/workflows/tests.yml)\n[![Version](https://img.shields.io/pypi/v/zipstream-ng.svg)](https://pypi.org/project/zipstream-ng/)\n![Python](https://img.shields.io/pypi/pyversions/zipstream-ng.svg)\n\nA modern and easy to use streamable zip file generator. It can package and stream many files and\nfolders into a zip on the fly without needing temporary files or excessive memory. It can also\ncalculate the final size of the zip file before streaming it.\n\n\n### Features:\n - Generates zip data on the fly as it's requested.\n - Can calculate the total size of the resulting zip file before generation even begins.\n - Low memory usage: Since the zip is generated as it's requested, very little has to be kept in\n   memory (peak usage of less than 20MB is typical, even for TBs of files).\n - Flexible API: Typical use cases are simple, complicated ones are possible.\n - Supports zipping data from files, bytes, strings, and any other iterable objects.\n - Keeps track of the date of the most recently modified file added to the zip file.\n - Threadsafe: Won't mangle data if multiple threads concurrently add data to the same stream.\n - Includes a clone of Python's `http.server` module with zip support added. Try `python -m zipstream.server`.\n - Automatically uses Zip64 extensions, but only if they are required.\n - No external dependencies.\n\n\n### Ideal for web backends:\n - Generating zip data on the fly requires very little memory, no disk usage, and starts producing\n   data with less latency than creating the entire zip up-front. This means faster responses, no\n   temporary files, and very low memory usage.\n - The ability to calculate the total size of the stream before any data is actually generated\n   (provided no compression is used) means web backends can provide a `Content-Length` header in\n   their responses. This allows clients to show a progress bar as the stream is transferred.\n - By keeping track of the date of the most recently modified file added to the zip, web\n   backends can provide a `Last-Modified` header. This allows clients to check if they have the most\n   up-to-date version of the zip with just a HEAD request instead of having to download the entire\n   thing.\n\n\nInstallation\n------------\n```\npip install zipstream-ng\n```\n\n\nExamples\n--------\n\n### Create a local zip file (simple example)\n\nMake an archive named `files.zip` in the current directory that contains all files under\n`/path/to/files`.\n\n```python\nfrom zipstream import ZipStream\n\nzs = ZipStream.from_path(\"/path/to/files/\")\n\nwith open(\"files.zip\", \"wb\") as f:\n    f.writelines(zs)\n```\n\n\n### Create a local zip file (demos more of the API)\n\n```python\nfrom zipstream import ZipStream, ZIP_DEFLATED\n\n# Create a ZipStream that uses the maximum level of Deflate compression.\nzs = ZipStream(compress_type=ZIP_DEFLATED, compress_level=9)\n\n# Set the zip file's comment.\nzs.comment = \"Contains compressed important files\"\n\n# Add all the files under a path.\n# Will add all files under a top-level folder called \"files\" in the zip.\nzs.add_path(\"/path/to/files/\")\n\n# Add another file (will be added as \"data.txt\" in the zip file).\nzs.add_path(\"/path/to/file.txt\", \"data.txt\")\n\n# Add some random data from an iterable.\n# This generator will only be run when the stream is generated.\ndef random_data():\n    import random\n    for _ in range(10):\n        yield random.randbytes(1024)\n\nzs.add(random_data(), \"random.bin\")\n\n# Add a file containing some static text.\n# Will automatically be encoded to bytes before being added (uses utf-8).\nzs.add(\"This is some text\", \"README.txt\")\n\n# Write out the zip file as it's being generated.\n# At this point the data in the files will be read in and the generator\n# will be iterated over.\nwith open(\"files.zip\", \"wb\") as f:\n    f.writelines(zs)\n```\n\n\n### zipserver (included)\n\nA fully-functional and useful example can be found in the included\n[`zipstream.server`](zipstream/server.py) module. It's a clone of Python's built in `http.server`\nwith the added ability to serve multiple files and folders as a single zip file. Try it out by\ninstalling the package and running `zipserver --help` or `python -m zipstream.server --help`.\n\n![zipserver screenshot](zipserver.png)\n\n\n### Integration with a Flask webapp\n\nA very basic [Flask](https://flask.palletsprojects.com/)-based file server that streams all the\nfiles under the requested path to the client as a zip file. It provides the total size of the stream\nin the `Content-Length` header so the client can show a progress bar as the stream is downloaded. It\nalso provides a `Last-Modified` header so the client can check if it already has the most recent\ncopy of the zipped data with a `HEAD` request instead of having to download the file and check.\n\nNote that while this example works, it's not a good idea to deploy it as-is due to the lack of input\nvalidation and other checks.\n\n```python\nimport os.path\nfrom flask import Flask, Response\nfrom zipstream import ZipStream\n\napp = Flask(__name__)\n\n@app.route(\"/\", defaults={\"path\": \".\"})\n@app.route(\"/\u003cpath:path\u003e\")\ndef stream_zip(path):\n    name = os.path.basename(os.path.abspath(path))\n    zs = ZipStream.from_path(path)\n    return Response(\n        zs,\n        mimetype=\"application/zip\",\n        headers={\n            \"Content-Disposition\": f\"attachment; filename={name}.zip\",\n            \"Content-Length\": len(zs),\n            \"Last-Modified\": zs.last_modified,\n        }\n    )\n\nif __name__ == \"__main__\":\n    app.run(host=\"0.0.0.0\", port=5000)\n```\n\n\n### Partial generation and last-minute file additions\n\nIt's possible to generate the zip stream, but stop before finalizing it. This enables adding\nsomething like a file manifest or compression log after all the files have been added.\n\n`ZipStream` provides a `info_list` method that returns information on all the files added to the\nstream. In this example, all that information will be added to the zip in a file named\n\"manifest.json\" before finalizing it.\n\n```python\nfrom zipstream import ZipStream\nimport json\n\ndef gen_zipfile():\n    zs = ZipStream.from_path(\"/path/to/files\")\n    yield from zs.all_files()\n    zs.add(\n        json.dumps(\n            zs.info_list(),\n            indent=2\n        ),\n        \"manifest.json\"\n    )\n    yield from zs.finalize()\n```\n\n\nComparison to stdlib\n--------------------\nSince Python 3.6 it has actually been possible to generate zip files as a stream using just the\nstandard library, it just hasn't been very ergonomic or efficient. Consider the typical use case of\nzipping up a directory of files while streaming it over a network connection:\n\n(note that the size of the stream is not pre-calculated in this case as this would make the stdlib\nexample way too long).\n\nUsing ZipStream:\n```python\nfrom zipstream import ZipStream\n\nsend_stream(\n    ZipStream.from_path(\"/path/to/files/\")\n)\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eThe same(ish) functionality using just the stdlib:\u003c/summary\u003e\n\n```python\nimport os\nimport io\nfrom zipfile import ZipFile, ZipInfo\n\nclass Stream(io.RawIOBase):\n    \"\"\"An unseekable stream for the ZipFile to write to\"\"\"\n\n    def __init__(self):\n        self._buffer = bytearray()\n        self._closed = False\n\n    def close(self):\n        self._closed = True\n\n    def write(self, b):\n        if self._closed:\n            raise ValueError(\"Can't write to a closed stream\")\n        self._buffer += b\n        return len(b)\n\n    def readall(self):\n        chunk = bytes(self._buffer)\n        self._buffer.clear()\n        return chunk\n\ndef iter_files(path):\n    for dirpath, _, files in os.walk(path, followlinks=True):\n        if not files:\n            yield dirpath  # Preserve empty directories\n        for f in files:\n            yield os.path.join(dirpath, f)\n\ndef read_file(path):\n    with open(path, \"rb\") as fp:\n        while True:\n            buf = fp.read(1024 * 64)\n            if not buf:\n                break\n            yield buf\n\ndef generate_zipstream(path):\n    stream = Stream()\n    with ZipFile(stream, mode=\"w\") as zf:\n        toplevel = os.path.basename(os.path.normpath(path))\n        for f in iter_files(path):\n            # Use the basename of the path to set the arcname\n            arcname = os.path.join(toplevel, os.path.relpath(f, path))\n            zinfo = ZipInfo.from_file(f, arcname)\n\n            # Write data to the zip file then yield the stream content\n            with zf.open(zinfo, mode=\"w\") as fp:\n                if zinfo.is_dir():\n                    continue\n                for buf in read_file(f):\n                    fp.write(buf)\n                    yield stream.readall()\n    yield stream.readall()\n\nsend_stream(\n    generate_zipstream(\"/path/to/files/\")\n)\n```\n\u003c/details\u003e\n\n\nTests\n-----\nThis package contains extensive tests. To run them, install `pytest` (`pip install pytest`) and run\n`py.test` in the project directory.\n\n\nLicense\n-------\nLicensed under the [GNU LGPLv3](https://www.gnu.org/licenses/lgpl-3.0.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpr0ps%2Fzipstream-ng","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpr0ps%2Fzipstream-ng","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpr0ps%2Fzipstream-ng/lists"}