{"id":13878885,"url":"https://github.com/WeTransfer/zip_tricks","last_synced_at":"2025-07-16T14:33:14.527Z","repository":{"id":8604307,"uuid":"59018625","full_name":"WeTransfer/zip_tricks","owner":"WeTransfer","description":"[DEPRECATED] Compact ZIP file writing/reading for Ruby, for streaming applications ","archived":true,"fork":false,"pushed_at":"2024-03-28T11:01:51.000Z","size":903,"stargazers_count":348,"open_issues_count":2,"forks_count":32,"subscribers_count":21,"default_branch":"main","last_synced_at":"2024-11-14T12:56:15.857Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WeTransfer.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2016-05-17T12:02:02.000Z","updated_at":"2024-10-05T18:57:51.000Z","dependencies_parsed_at":"2024-01-13T20:51:52.559Z","dependency_job_id":"ba0d8981-cce2-4664-a230-d1bb982d673c","html_url":"https://github.com/WeTransfer/zip_tricks","commit_stats":{"total_commits":468,"total_committers":21,"mean_commits":"22.285714285714285","dds":0.1581196581196581,"last_synced_commit":"4a36824cabac1c1a66b6ed5b96b59e4b3c91e007"},"previous_names":[],"tags_count":56,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WeTransfer%2Fzip_tricks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WeTransfer%2Fzip_tricks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WeTransfer%2Fzip_tricks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WeTransfer%2Fzip_tricks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WeTransfer","download_url":"https://codeload.github.com/WeTransfer/zip_tricks/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226138849,"owners_count":17579496,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-06T08:02:03.283Z","updated_at":"2024-11-24T07:31:23.257Z","avatar_url":"https://github.com/WeTransfer.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# zip_tricks\n\n## ⚠️ Deprecation notice\n\nzip_tricks will not receive further updates or support, and will no longer be maintained. The story of zip_tricks continues\nin [zip_kit](https://github.com/julik/zip_kit) which is going to be receiving regular updates and supports all\nof the zip_tricks functionality (and more). Thank you for being part of the zip_tricks community!\n\n[![Gem Version](https://badge.fury.io/rb/zip_tricks.svg)](https://badge.fury.io/rb/zip_tricks)\n\n--------------\n\nAllows streaming, non-rewinding ZIP file output from Ruby.\n\nInitially written and as a spiritual successor to [zipline](https://github.com/fringd/zipline)\nand now proudly powering it under the hood.\n\nAllows you to write a ZIP archive out to a File, Socket, String or Array without having to rewind it at any\npoint. Usable for creating very large ZIP archives for immediate sending out to clients, or for writing\nlarge ZIP archives without memory inflation.\n\nzip_tricks currently handles all our zipping needs (millions of ZIP files generated per day), so we are\npretty confident it is widely compatible with a large number of unarchiving end-user applications.\n\n## Requirements\n\nRuby 2.1+ syntax support (keyword arguments with defaults) and a working zlib (all available to jRuby as well).\njRuby might experience problems when using the reader methods due to the argument of `IO#seek` being limited\nto [32 bit sizes.](https://github.com/jruby/jruby/issues/3817)\n\n\n## Diving in: send some large CSV reports from Rails\n\nThe easiest is to include the `ZipTricks::RailsStreaming` module into your\ncontroller.\n\n```ruby\nclass ZipsController \u003c ActionController::Base\n  include ZipTricks::RailsStreaming\n\n  def download\n    zip_tricks_stream do |zip|\n      zip.write_deflated_file('report1.csv') do |sink|\n        CSV(sink) do |csv_write|\n          csv_write \u003c\u003c Person.column_names\n          Person.all.find_each do |person|\n            csv_write \u003c\u003c person.attributes.values\n          end\n        end\n      end\n      zip.write_deflated_file('report2.csv') do |sink|\n        ...\n      end\n    end\n  end\nend\n```\n\nIf you want some more conveniences you can also use [zipline](https://github.com/fringd/zipline) which\nwill automatically process and stream attachments (Carrierwave, Shrine, ActiveStorage) and remote objects\nvia HTTP.\n\n## Create a ZIP file without size estimation, compress on-the-fly during writes\n\nBasic use case is compressing on the fly. Some data will be buffered by the Zlib deflater, but\nmemory inflation is going to be very constrained. Data will be written to destination at fairly regular\nintervals. Deflate compression will work best for things like text files.\n\n```ruby\nout = my_tempfile # can also be a socket\nZipTricks::Streamer.open(out) do |zip|\n  zip.write_stored_file('mov.mp4.txt') do |sink|\n    File.open('mov.mp4', 'rb'){|source| IO.copy_stream(source, sink) }\n  end\n  zip.write_deflated_file('long-novel.txt') do |sink|\n    File.open('novel.txt', 'rb'){|source| IO.copy_stream(source, sink) }\n  end\nend\n```\nUnfortunately with this approach it is impossible to compute the size of the ZIP file being output,\nsince you do not know how large the compressed data segments are going to be.\n\n## Send a ZIP from a Rack response\n\nTo \"pull\" data from ZipTricks you can create an `OutputEnumerator` object which will yield the binary chunks piece\nby piece, and apply some amount of buffering as well. Since this `OutputEnumerator` responds to `#each` and yields\nStrings it also can (and should!) be used as a Rack response body. Return it to your webserver and you will\nhave your ZIP streamed. The block that you give to the `OutputEnumerator` will only start executing once your\nresponse body starts getting iterated over - when actually sending the response to the client\n(unless you are using a buffering Rack webserver, such as Webrick).\n\n```ruby\nbody = ZipTricks::Streamer.output_enum do | zip |\n  zip.write_stored_file('mov.mp4') do |sink| # Those MPEG4 files do not compress that well\n    File.open('mov.mp4', 'rb'){|source| IO.copy_stream(source, sink) }\n  end\n  zip.write_deflated_file('long-novel.txt') do |sink|\n    File.open('novel.txt', 'rb'){|source| IO.copy_stream(source, sink) }\n  end\nend\n[200, {}, body]\n```\n\n## Send a ZIP file of known size, with correct headers\n\nUse the `SizeEstimator` to compute the correct size of the resulting archive.\n\n```ruby\n# Precompute the Content-Length ahead of time\nbytesize = ZipTricks::SizeEstimator.estimate do |z|\n z.add_stored_entry(filename: 'myfile1.bin', size: 9090821)\n z.add_stored_entry(filename: 'myfile2.bin', size: 458678)\nend\n\n# Prepare the response body. The block will only be called when the response starts to be written.\nzip_body = ZipTricks::RackBody.new do | zip |\n  zip.add_stored_entry(filename: \"myfile1.bin\", size: 9090821, crc32: 12485)\n  zip \u003c\u003c read_file('myfile1.bin')\n  zip.add_stored_entry(filename: \"myfile2.bin\", size: 458678, crc32: 89568)\n  zip \u003c\u003c read_file('myfile2.bin')\nend\n\n[200, {'Content-Length' =\u003e bytesize.to_s}, zip_body]\n```\n\n## Writing ZIP files using the Streamer bypass\n\nYou do not have to \"feed\" all the contents of the files you put in the archive through the Streamer object.\nIf the write destination for your use case is a `Socket` (say, you are writing using Rack hijack) and you know\nthe metadata of the file upfront (the CRC32 of the uncompressed file and the sizes), you can write directly\nto that socket using some accelerated writing technique, and only use the Streamer to write out the ZIP metadata.\n\n```ruby\n# io has to be an object that supports #\u003c\u003c\nZipTricks::Streamer.open(io) do | zip |\n  # raw_file is written \"as is\" (STORED mode).\n  # Write the local file header first..\n  zip.add_stored_entry(filename: \"first-file.bin\", size: raw_file.size, crc32: raw_file_crc32)\n\n  # Adjust the ZIP offsets within the Streamer\n  zip.simulate_write(my_temp_file.size)\n\n  # ...and then send the actual file contents bypassing the Streamer interface\n  io.sendfile(my_temp_file)\n\nend\n```\n\n## Other usage examples\n\nCheck out the `examples/` directory at the root of the project. This will give you a good idea\nof various use cases the library supports.\n\n## Computing the CRC32 value of a large file\n\n`BlockCRC32` computes the CRC32 checksum of an IO in a streaming fashion.\nIt is slightly more convenient for the purpose than using the raw Zlib library functions.\n\n```ruby\ncrc = ZipTricks::StreamCRC32.new\ncrc \u003c\u003c next_chunk_of_data\n...\n\ncrc.to_i # Returns the actual CRC32 value computed so far\n...\n# Append a known CRC32 value that has been computed previosuly\ncrc.append(precomputed_crc32, size_of_the_blob_computed_from)\n```\n\nYou can also compute the CRC32 for an entire IO object if it responds to `#eof?`:\n\n```ruby\ncrc = ZipTricks::StreamCRC32.from_io(file) # Returns an Integer\n```\n\n## Reading ZIP files\n\nThe library contains a reader module, play with it to see what is possible. It is not a complete ZIP reader\nbut it was designed for a specific purpose (highly-parallel unpacking of remotely stored ZIP files), and\nas such it performs it's function quite well. Please beware of the security implications of using ZIP readers\nthat have not been formally verified (ours hasn't been).\n\n## Contributing to zip_tricks\n\n* Check out the latest `main` to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.\n* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.\n* Fork the project.\n* Start a feature/bugfix branch.\n* Commit and push until you are happy with your contribution.\n* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.\n* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.\n\n## Copyright and license\n\nCopyright (c) 2020 WeTransfer.\n\n`zip_tricks` is distributed under the conditions of the [Hippocratic License](https://firstdonoharm.dev/version/1/2/license.html)\nSee LICENSE.txt for further details. If this license is not acceptable for your use case we still maintain the 4.x version tree\nwhich remains under the MIT license, see https://rubygems.org/gems/zip_tricks/versions for more information.\nNote that we only backport some performance optimizations and crucial bugfixes but not the new features to that tree.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FWeTransfer%2Fzip_tricks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FWeTransfer%2Fzip_tricks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FWeTransfer%2Fzip_tricks/lists"}