{"id":24291637,"url":"https://github.com/scosman/zipstreamer","last_synced_at":"2025-04-05T15:09:25.884Z","repository":{"id":26992935,"uuid":"112101619","full_name":"scosman/zipstreamer","owner":"scosman","description":"Zip File Streaming Microservice - stream zip files on the fly","archived":false,"fork":false,"pushed_at":"2025-01-28T04:18:44.000Z","size":153,"stargazers_count":121,"open_issues_count":4,"forks_count":18,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-29T14:12:41.106Z","etag":null,"topics":["golang","microservice","on-the-fly","serverless","streaming","streaming-server","zip","zip-file"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scosman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-26T17:16:45.000Z","updated_at":"2025-01-30T18:54:26.000Z","dependencies_parsed_at":"2024-06-18T22:55:06.529Z","dependency_job_id":"0f5ce933-bc4b-4059-9fd0-a6cab2a9f3e4","html_url":"https://github.com/scosman/zipstreamer","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scosman%2Fzipstreamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scosman%2Fzipstreamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scosman%2Fzipstreamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scosman%2Fzipstreamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scosman","download_url":"https://codeload.github.com/scosman/zipstreamer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247353749,"owners_count":20925329,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","microservice","on-the-fly","serverless","streaming","streaming-server","zip","zip-file"],"created_at":"2025-01-16T14:56:18.743Z","updated_at":"2025-04-05T15:09:25.853Z","avatar_url":"https://github.com/scosman.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\u003cimg alt=\"Zipstreamer Logo\" src=\"https://user-images.githubusercontent.com/848343/200652990-70e5f59d-c699-42ed-9a4f-a92cdfddda0d.png\" width=\"300\"\u003e\u003c/h1\u003e\n\n[![Build and Test](https://github.com/scosman/zipstreamer/actions/workflows/test.yml/badge.svg)](https://github.com/scosman/zipstreamer/actions/workflows/test.yml)\n[![Format and Vet](https://github.com/scosman/zipstreamer/actions/workflows/format_check.yml/badge.svg)](https://github.com/scosman/zipstreamer/actions/workflows/format_check.yml)\n[![Docker Generation](https://github.com/scosman/zipstreamer/actions/workflows/publish.yml/badge.svg)](https://github.com/scosman/zipstreamer/pkgs/container/packages%2Fzipstreamer)\n[![Go Report Card](https://goreportcard.com/badge/github.com/scosman/zipstreamer)](https://goreportcard.com/report/github.com/scosman/zipstreamer)\n[![Go Reference](https://pkg.go.dev/badge/github.com/scosman/zipstreamer.svg)](https://pkg.go.dev/github.com/scosman/zipstreamer)\n\n**ZipStreamer** is a golang microservice for streaming zip files from a series of web links, on the fly. For example, if you have 200 files on S3, and you want to download a zip file of them, you can do so in 1 request to this server.\n\nHighlights include:\n\n - Low memory: the files are streamed out to the client immediately\n - Low CPU: the default server doesn't compress files, only packages them into a zip, so there's minimal CPU load ([configurable](#customize-a))\n - High concurrency: the two properties above allow a single small server to stream hundreds of large zips simultaneous\n - [Easy to host](#deploy-a): several deployment options, including Docker images and two one-click deployers\n - It includes a HTTP server, but can be used as a library (see `zip_streamer.go`)\n\n## Content\n\n - [JSON Zip File Descriptor](#json-descriptor-a)\n - [HTTP Endpoints](#http-endpoints-a)\n   - [POST /download](#post-download-a)\n   - [GET /download](#get-download-a)\n   - [POST /create_download_link](#post-create-a)\n   - [GET /download_link/{link_id}](#get-link-a)\n - [Deploy](#deploy-a)\n   - [Heroku - One Click Deploy](#deploy-heroku-a)\n   - [Google Cloud Run - One Click Deploy, Serverless](#deploy-google-a)\n   - [Docker](#deploy-docker-a)\n - [Configuration Options](#customize-a)\n\n\u003ca name=\"json-descriptor-a\"\u003e\u003c/a\u003e\n## JSON Zip File Descriptor \n\nEach HTTP endpoint requires a JSON description of the desired zip file. It includes a root object with the following structure:\n\n - `suggestedFilename` [Optional, string]: The filename to suggest in the \"Save As\" UI in browsers. Defaults to `archive.zip` if not provided or invalid. [Limited to US-ASCII.](https://www.rfc-editor.org/rfc/rfc2183#section-2.3)\n - `files` [Required, array]: an array descibing the files to include in the zip file. Each array entry required 2 properties:\n    - `url` [Required, string]: the public URL of the file to include in the zip. Zipstreamer will fetch this via a GET request. The file must be publically accessible via this URL; if you're files are private, most file hosts provide query string authentication options which work well with Zipstreamer (example [AWS S3 Docs](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html)).\n    - `zipPath`  [Required, string]: the path and filename where this entry should appear in the resulting zip file. This is a relative path to the root of the zip file.\n\nExample JSON description with 2 files:\n\n```\n{\n  \"suggestedFilename\": \"tps_reports.zip\",\n  \"files\": [\n    {\n      \"url\":\"https://server.com/image1.jpg\",\n      \"zipPath\":\"image1.jpg\"\n    },\n    {\n      \"url\":\"https://server.com/image2.jpg\",\n      \"zipPath\":\"in-a-sub-folder/image2.jpg\"\n    }\n  ]\n}\n```\n\n\u003ca name=\"http-endpoints-a\"\u003e\u003c/a\u003e\n## HTTP Endpoints\n\n\u003ca name=\"post-download-a\"\u003e\u003c/a\u003e\n### `POST /download`\n\nThis endpoint takes a http POST body containing the [JSON zip file descriptor](#json-descriptor-a), and returns a zip file.\n\n\u003cdetails\u003e\n  \u003csummary\u003eExample usage with curl\u003c/summary\u003e\n\n#### Example curl usage of `POST /download` endpoint\n\n```\n# download a sample json descriptor\ncurl https://gist.githubusercontent.com/scosman/f57a3561fed98caab2d0ae285a0d7251/raw/4a9630951373e50f467f41d8c7b9d440c13a14d2/zipJsonDescriptor.json \u003e zipJsonDescriptor.json\n# call POST /download endpoint, passing json descriptor in body\ncurl --data-binary \"@./zipJsonDescriptor.json\" http://localhost:4008/download \u003e archive.zip\n```\n\u003c/details\u003e\n\n\u003ca name=\"get-download-a\"\u003e\u003c/a\u003e\n### `GET /download`\n\nThis endpoint fetches a [JSON zip file descriptor](#json-descriptor-a) hosted on another server, and returns a zip file. This is useful over the `POST /download` endpoint for a few use cases:\n\n - You want to hide from the client where the original files are hosted (see `zsid` parameter)\n - Use cases where POST requests aren't easy to adopt (traditional static webpages)\n - You want to trigger a browsers' \"Save File\" UI, which isn't shown for POST requests. See `POST /create_download_link` for a client side alternitive to achieve this.\n\nThis endpoint requires one of two query parameters describing where to find the JSON zip file descriptor:\n\n - `zsurl`: the full URL to the JSON file describing the zip. Example: `/download?zsurl=https://yourserver.com/path_to_descriptors/82a1b54cd20ab44a916bd76a5`\n - `zsid`: must be used with the `ZS_LISTFILE_URL_PREFIX` environment variable. The JSON file will be fetched from `ZS_LISTFILE_URL_PREFIX + zsid`. This allows you to hide the full URL path from clients, revealing only the end of the URL. Example: `ZS_LISTFILE_URL_PREFIX = \"https://yoursever.com/path_to_descriptors/\"` and `/download?zsid=82a1b54cd20ab44a916bd76a5`\n\n\u003cdetails\u003e\n  \u003csummary\u003eExample usage with curl\u003c/summary\u003e\n\n#### Example curl usage of `GET /download` endpoint with zsurl parameter\n\n```\ncurl -X GET \"http://localhost:4008/download?zsurl=https://gist.githubusercontent.com/scosman/f57a3561fed98caab2d0ae285a0d7251/raw/4a9630951373e50f467f41d8c7b9d440c13a14d2/zipJsonDescriptor.json\" \u003e archive.zip\n```\n\n#### Example curl usage of `GET /download` endpoint with zsid parameter\n\n```\n# start server with ZS_LISTFILE_URL_PREFIX\nZS_LISTFILE_URL_PREFIX=\"https://gist.githubusercontent.com/scosman/\" ./zipstreamer\n```\n\n```\n# call `GET /download` endpoint with zsid\ncurl -X GET \"http://localhost:4008/download?zsid=f57a3561fed98caab2d0ae285a0d7251/raw/4a9630951373e50f467f41d8c7b9d440c13a14d2/zipJsonDescriptor.json\" \u003e archive.zip\n```\n\n\u003c/details\u003e\n\n\u003ca name=\"post-create-a\"\u003e\u003c/a\u003e\n### `POST /create_download_link`\n\nThis endpoint takes a http POST body containing the [JSON zip file descriptor](#json-descriptor-a), stores it in a local cache, and returns a link ID which allows the caller to fetch the zip file via an additional call to `GET /download_link/{link_id}`.\n\nThis is useful for if you want to trigger a browser \"Save File\" UI, which isn't shown for POST requests. See `GET /download` for a server side alternative to achieve this.\n\n*Important*:\n\n - These links only live for 60 seconds. They are expected to be used immediately.\n - This stores the link in an in-memory cache, so it's not suitable for deploying to a multi-server cluster without extra configuration. If you are hosting on a multi-server cluster, see the deployment section for configuration advice.\n\nHere is an example response body containing the link ID. See docs for `GET /download_link/{link_id}` below for how to fetch this zip file:\n\n```\n{\n  \"status\":\"ok\",\n  \"link_id\":\"b4ecfdb7-e0fa-4aca-ad87-cb2e4245c8dd\"\n}\n```\n\nExample usage: see `GET /download_link/{link_id}` documentation below.\n\n\u003ca name=\"get-link-a\"\u003e\u003c/a\u003e\n### `GET /download_link/{link_id}`\n\nCall this endpoint with a `link_id` generated with `/create_download_link` to download that zip file.\n\n\u003cdetails\u003e\n  \u003csummary\u003eExample usage with curl\u003c/summary\u003e\n\n#### Example curl usage of `POST /create_download_link` and `GET /download_link/{link_id}` endpoints working together\n\n```\n# download a sample json descriptor\ncurl https://gist.githubusercontent.com/scosman/f57a3561fed98caab2d0ae285a0d7251/raw/4a9630951373e50f467f41d8c7b9d440c13a14d2/zipJsonDescriptor.json \u003e zipJsonDescriptor.json\n# call POST endpoint to create link\ncurl --data-binary \"@./zipJsonDescriptor.json\" http://localhost:4008/create_download_link\n# Call GET endpoint to download zip. Note: must copy UUID from output of above POST command into this URL\ncurl -X GET \"http://localhost:4008/download_link/UUID_FROM_ABOVE\" \u003e archive.zip\n```\n\u003c/details\u003e\n\n\u003ca name=\"deploy-a\"\u003e\u003c/a\u003e\n## Deploy\n\n\u003ca name=\"deploy-heroku-a\"\u003e\u003c/a\u003e\n### Heroku - One Click Deploy\n\n[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy?template=https://github.com/scosman/zipstreamer/tree/master)\n\nBe sure to enable [session affinity](https://devcenter.heroku.com/articles/session-affinity) if you're using multiple servers and using `/create_download_link`.\n\n\u003ca name=\"deploy-google-a\"\u003e\u003c/a\u003e\n### Google Cloud Run - One Click Deploy, Serverless\n\n[\u003cimg src=\"https://deploy.cloud.run/button.svg\" width=180 alt=\"Run on Google Cloud\"\u003e](https://deploy.cloud.run?git_repo=https%3A%2F%2Fgithub.com%2Fscosman%2Fzipstreamer)\n\nCloud Run is ideal serverless environment for ZipStreamer, as it routes many requests to a single container instance. ZipStreamer is designed to handle many concurrent requests, and will be cheaper to run on this serverless architecture than a instance-per-request architecture like AWS Lamba or Google Cloud Functions.\n\n**Important** \n - The one-click deploy button has [a bug](https://github.com/GoogleCloudPlatform/cloud-run-button/issues/232) and may force you to set the optional environment variables. If the server isn't working, check `ZS_URL_PREFIX` is blank in the Cloud Run console.\n - Be sure to enable [session affinity](https://cloud.google.com/run/docs/configuring/session-affinity) if you're using using `/create_download_link`. Cloud Run may scale up to multiple containers automatically.\n\n\u003ca name=\"deploy-docker-a\"\u003e\u003c/a\u003e\n### Docker \n\nThis repo contains an dockerfile, and an image is published [on Github Packages](https://github.com/scosman/zipstreamer/pkgs/container/packages%2Fzipstreamer).\n\n#### Build Your Own Image\n\nTo build your own image, clone the repo and run: \n\n```\ndocker build --tag docker-zipstreamer .\n# Start on port 8080\ndocker run --env PORT=8080 -p 8080:8080 docker-zipstreamer\n```\n\n#### Run Official Package from Github Packages\n\nOfficial packages are published on Github packages. To pull latest stable release:\n\n```\ndocker pull ghcr.io/scosman/packages/zipstreamer:stable\n# Start on port 8080\ndocker run --env PORT=8080 -p 8080:8080 ghcr.io/scosman/packages/zipstreamer:stable\n```\n\nNote: `stable` pulls the latest github release. Use `ghcr.io/scosman/packages/zipstreamer:latest` for top of tree.\n\n\u003ca name=\"customize-a\"\u003e\u003c/a\u003e\n## Configuration Options\n\nThese environment variables can be used to configure the server:\n\n - `PORT` - Defaults to 4008. Sets which port the HTTP server binds to.\n - `ZS_URL_PREFIX` - If set, the server will verify the `url` property of the files in the JSON zip file descriptors start with this prefix. Useful to preventing others from using your server to serve their files.\n - `ZS_COMPRESSION` - Defaults to no compression. It's not universally known, but zip files can be uncompressed, and used only to combining many files into one file. Set to `DEFLATE` to use zip deflate compression. **WARNING - enabling compression uses CPU, and will reduce throughput of server**. Note: for files with internal compression (JPEGs, MP4s, etc), zip DEFLATE compression will often increase the total zip file size.\n  - `ZS_LISTFILE_URL_PREFIX` - See documentation for `GET /download`\n\n## Why\n\nI was mentoring at a \"Teens Learning Code\" class, but we had too many mentors, so I had some downtime.\n\n## Logo\n\nZipper portion of logo by Kokota from Noun Project (Creative Commons CCBY)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscosman%2Fzipstreamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscosman%2Fzipstreamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscosman%2Fzipstreamer/lists"}