{"id":28637490,"url":"https://github.com/philipaconrad/gzipstreamwriter","last_synced_at":"2025-06-12T18:11:07.624Z","repository":{"id":289914140,"uuid":"869934196","full_name":"philipaconrad/gzipstreamwriter","owner":"philipaconrad","description":"Ever wanted to merge gzipped []byte blobs together without decompressing first? Now you can.","archived":false,"fork":false,"pushed_at":"2025-04-25T17:40:20.000Z","size":70,"stargazers_count":1,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-25T18:42:01.100Z","etag":null,"topics":["compression","go","golang","gzip","gzip-compression"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/philipaconrad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-09T06:45:07.000Z","updated_at":"2025-04-16T21:29:40.000Z","dependencies_parsed_at":"2025-04-25T18:52:04.363Z","dependency_job_id":null,"html_url":"https://github.com/philipaconrad/gzipstreamwriter","commit_stats":null,"previous_names":["philipaconrad/gzipstreamwriter"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/philipaconrad/gzipstreamwriter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipaconrad%2Fgzipstreamwriter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipaconrad%2Fgzipstreamwriter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipaconrad%2Fgzipstreamwriter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipaconrad%2Fgzipstreamwriter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/philipaconrad","download_url":"https://codeload.github.com/philipaconrad/gzipstreamwriter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipaconrad%2Fgzipstreamwriter/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259518828,"owners_count":22870304,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","go","golang","gzip","gzip-compression"],"created_at":"2025-06-12T18:11:07.006Z","updated_at":"2025-06-12T18:11:07.609Z","avatar_url":"https://github.com/philipaconrad.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GzipStreamWriter\n\n*Ever wanted to merge gzipped `[]byte` blobs together without decompressing first? Now you can.*\n\nThis project exists to solve a very specific problem: efficiently concatenating gzip blobs together, as if they'd all been written as a single stream. This is trickier than it sounds, because we *don't* want to decompress the gzipped blobs while writing them!\n\n## Design Goals:\n - Allow writing either raw bytes or compressed gzip blobs to a destination, resulting in a valid, concatenated gzip blob at the destination, as if the blob had been written \"all at once\" as a single gzip byte stream.\n - Avoid decompressing the compressed blobs.\n - Avoid excessive memory and CPU burn if possible.\n - Avoid exposing the awful complex parts to happy-path users. Still provide some support scaffolding for hard-mode folks.\n\n## Design Anti-Goals:\n - Best possible compression performance: We know that some usage patterns can result in a less-than-optimal overall compression ratio.\n\n## Algorithm\n\nFor writing compressed blobs:\n - Drop the header.\n - Extract CRC32 and uncompressed length fields from the trailer, and drop the trailer.\n - Write the blob to the stream.\n - Update the running CRC32 by the XOR trick from zlib.\n - Update the length field using the trailer length field.\n\nFor uncompressed `[]byte` writes:\n - Update the length field using `len(slice)`.\n - Compress into a gzip blob.\n - Extract the CRC32 from the trailer.\n - Drop header and trailer.\n - Write the blob to the stream.\n\nThis gives us a powerful abstraction that \"does the right thing\" behind the scenes, while being ridiculously cheaper to compute than decompressing and recompressing compressed gzip data.\n\n## Go Version Support\n\nI'm currently supporting the latest Go major version.\nPreviously, I'd aimed to support the current major version - 2, but there's a world of performance and features availble on Go 1.24+, and I'd like to have access to those things.\nThe larger open source project that this library was originally developed for has caught up to at least Go 1.23+, so this library should be usable for them soon.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipaconrad%2Fgzipstreamwriter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphilipaconrad%2Fgzipstreamwriter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipaconrad%2Fgzipstreamwriter/lists"}