{"id":13879519,"url":"https://github.com/lautis/piperator","last_synced_at":"2025-04-04T12:08:10.784Z","repository":{"id":21296131,"uuid":"88634406","full_name":"lautis/piperator","owner":"lautis","description":"Composable pipelines for Enumerators.","archived":false,"fork":false,"pushed_at":"2023-08-01T09:01:41.000Z","size":61,"stargazers_count":205,"open_issues_count":3,"forks_count":8,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-28T11:08:35.174Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lautis.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-18T14:26:38.000Z","updated_at":"2025-01-27T09:00:23.000Z","dependencies_parsed_at":"2024-10-30T13:11:19.650Z","dependency_job_id":"694037aa-c928-44ce-809e-da15f0b3cbe7","html_url":"https://github.com/lautis/piperator","commit_stats":{"total_commits":57,"total_committers":6,"mean_commits":9.5,"dds":0.1228070175438597,"last_synced_commit":"d809575e3ae5444614f23f07b5acb34a17501834"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lautis%2Fpiperator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lautis%2Fpiperator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lautis%2Fpiperator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lautis%2Fpiperator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lautis","download_url":"https://codeload.github.com/lautis/piperator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247174419,"owners_count":20896078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-06T08:02:23.652Z","updated_at":"2025-04-04T12:08:10.768Z","avatar_url":"https://github.com/lautis.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# Piperator\n\nPipelines for streaming large collections. The pipeline enables composition of streaming pipelines with lazy enumerables.\n\nThe library is heavily inspired by [Elixir pipe operator](https://elixirschool.com/en/lessons/basics/pipe-operator/) and [Node.js Stream](https://nodejs.org/api/stream.html).\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n**Table of Contents**\n\n- [Installation](#installation)\n- [Usage](#usage)\n  - [Pipelines](#pipelines)\n  - [Enumerators as IO objects](#enumerators-as-io-objects)\n- [Development](#development)\n- [Related Projects](#related-projects)\n- [Contributing](#contributing)\n- [License](#license)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n## Installation\n\nPiperator is distributed as a ruby gem and can be installed with\n\n```\n$ gem install piperator\n```\n\n## Usage\n\nStart by requiring the gem\n\n```ruby\nrequire 'piperator'\n```\n\n### Pipelines\n\nAs an appetizer, here's a pipeline that triples all input values and then sums the values.\n\n```ruby\nPiperator.\n  pipe(-\u003e(values) { values.lazy.map { |i| i * 3 } }).\n  pipe(-\u003e(values) { values.sum }).\n  call([1, 2, 3])\n# =\u003e 18\n```\n\nThe same could also be achieved using DSL instead of method chaining:\n\n```ruby\nPiperator.build do\n  pipe(-\u003e(values) { values.lazy.map { |i| i * 3 } })\n  pipe(-\u003e(values) { values.sum })\nend.call([1, 2, 3])\n```\n\nIf desired, the input enumerable can also be given as the first element of the pipeline using `Piperator.wrap`.\n\n```ruby\nPiperator.\n  wrap([1, 2, 3]).\n  pipe(-\u003e(values) { values.lazy.map { |i| i * 3 } }).\n  pipe(-\u003e(values) { values.sum }).\n  call\n# =\u003e 18\n```\n\nHave reasons to defer constructing a pipe? Evaluate it lazily:\n\n```ruby\nsumming = -\u003e(values) { values.sum }\nPiperator.build\n  pipe(-\u003e(values) { values.lazy.map { |i| i * 3 } })\n  lazy do\n    summing\n  end\nend.call([1, 2, 3])\n```\n\nThere is, of course, a much more idiomatic alternative in Ruby:\n\n```ruby\n[1, 2, 3].map { |i| i * 3 }.sum\n```\n\nSo why bother?\n\nTo run code before the stream processing start and after processing has ended. Let's use the same pattern to calculate the decompressed length of a GZip file fetched over HTTP with streaming.\n\n```ruby\nrequire 'piperator'\nrequire 'uri'\nrequire 'em-http-request'\nrequire 'net/http'\n\nmodule HTTPFetch\n  def self.call(url)\n    uri = URI(url)\n    Enumerator.new do |yielder|\n      Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') do |http|\n        request = Net::HTTP::Get.new(uri.request_uri)\n        http.request request do |response|\n          response.read_body { |chunk| yielder \u003c\u003c chunk }\n        end\n      end\n    end\n  end\nend\n\nmodule GZipDecoder\n  def self.call(enumerable)\n    Enumerator.new do |yielder|\n      decoder = EventMachine::HttpDecoders::GZip.new do |chunk|\n        yielder \u003c\u003c chunk\n      end\n\n      enumerable.each { |chunk| decoder \u003c\u003c chunk }\n      yielder \u003c\u003c decoder.finalize.to_s\n    end\n  end\nend\n\nlength = proc do |enumerable|\n  enumerable.lazy.reduce(0) { |aggregate, chunk| aggregate + chunk.length }\nend\n\nPiperator.\n  pipe(HTTPFetch).\n  pipe(GZipDecoder).\n  pipe(length).\n  call('http://ftp.gnu.org/gnu/gzip/gzip-1.2.4.tar.gz')\n```\n\nAt no point is it necessary to keep the full response or decompressed content in memory. This is a huge win when the file sizes grow beyond the 780kB seen in the example.\n\nPipelines themselves respond to `#call`. This enables using pipelines as pipes in other pipelines.\n\n```ruby\nappend_end = proc do |enumerator|\n  Enumerator.new do |yielder|\n    enumerator.each { |item| yielder \u003c\u003c item }\n    yielder \u003c\u003c 'end'\n  end\nend\n\nprepend_start = proc do |enumerator|\n  Enumerator.new do |yielder|\n    yielder \u003c\u003c 'start'\n    enumerator.each { |item| yielder \u003c\u003c item }\n  end\nend\n\ndouble = -\u003e(enumerator) { enumerator.lazy.map { |i| i * 2 } }\n\nprepend_append = Piperator.pipe(prepend_start).pipe(append_end)\nPiperator.pipe(double).pipe(prepend_append).call([1, 2, 3]).to_a\n# =\u003e ['start', 2, 4, 6, 'end']\n```\n\n### Enumerators as IO objects\n\nPiperator also provides a helper class that allows `Enumerator`s to be used as\nIO objects. This is useful to provide integration with libraries that work only\nwith IO objects such as [Nokogiri](http://www.nokogiri.org) or\n[Oj](https://github.com/ohler55/oj).\n\nAn example pipe that would yield all XML node in a document read in streams:\n\n```ruby\n\nrequire 'nokogiri'\nstreaming_xml = lambda do |enumerable|\n  Enumerator.new do |yielder|\n    io = Piperator::IO.new(enumerable.each)\n    reader = Nokogiri::XML::Reader(io)\n    reader.each { |node| yielder \u003c\u003c node }\n  end\nend\n```\n\nIn real-world scenarios, the pipe would need to filter the nodes. Passing every\nsingle XML node forward is not that useful.\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n## Related Projects\n\n* [D★Stream](https://github.com/ddfreyne/d-stream) - Set of extensions for writing stream-processing code.\n* [ddbuffer](https://github.com/ddfreyne/ddbuffer) - Buffer enumerables/enumerators.\n* [Down::ChunkedIO](https://github.com/janko-m/down/blob/master/lib/down/chunked_io.rb) - A similar IO class as Piperator::IO.\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/lautis/piperator.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flautis%2Fpiperator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flautis%2Fpiperator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flautis%2Fpiperator/lists"}