{"id":20842824,"url":"https://github.com/mudge/readable","last_synced_at":"2026-04-19T16:02:52.442Z","repository":{"id":66366863,"uuid":"121862737","full_name":"mudge/readable","owner":"mudge","description":"A generic way to create IO-like objects from any source","archived":false,"fork":false,"pushed_at":"2018-02-18T17:29:16.000Z","size":7,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-04-19T12:37:56.735Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mudge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-02-17T14:40:07.000Z","updated_at":"2019-02-28T17:14:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"f8a9afa7-7d17-47a1-b905-72ee48a4a25e","html_url":"https://github.com/mudge/readable","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mudge/readable","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mudge%2Freadable","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mudge%2Freadable/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mudge%2Freadable/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mudge%2Freadable/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mudge","download_url":"https://codeload.github.com/mudge/readable/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mudge%2Freadable/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32012787,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T20:23:30.271Z","status":"online","status_checked_at":"2026-04-19T02:00:07.110Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T01:25:38.325Z","updated_at":"2026-04-19T16:02:52.437Z","avatar_url":"https://github.com/mudge.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Readable\n\nInspired by [`Enumerator`](http://ruby-doc.org/core/Enumerator.html), an attempt to provide a generic way of creating an IO-like object from any source.\n\n```ruby\nrequire 'csv'\nrequire 'zlib'\n\nreadable = Readable.new do |yielder|\n  yielder \u003c\u003c \"\\u001F\\x8B\\b\\u0000[\\u0017\\x88Z\\u0000\"\n  yielder \u003c\u003c \"\\u0003\\xF3H\\xCD\\xC9\\xC9\\xD7)\\xCF/\\xCAI\"\n  yielder \u003c\u003c \"\\u0001\\u0000)^\\u0014\\xFC\\v\\u0000\\u0000\\u0000\"\nend\ngz_reader = Zlib::GzipReader.new(readable)\n\nCSV(gz_reader).gets\n#=\u003e ['Hello', 'world']\n```\n\n## Why?\n\nThis came from a problem at [work](https://www.altmetric.com) where we needed to do the following:\n\n1. Download a series of large, compressed JSON files from S3\n2. Uncompress each file\n3. Parse JSON out of them, saving each parsed object to a database\n\nI wished we could do each of these things lazily so we only download as much as we needed to uncompress and only uncompress enough to parse and only parse as much as we need to save to the database.\n\nTantalisingly, each step of this process offered _some_ form of streaming interface, e.g. the S3 client allows you to read objects in chunks:\n\n```ruby\ns3_client.get_object do |chunk|\n  # do something with chunk\nend\n```\n\nAnd Ruby's standard library has a [`GzipReader`](http://ruby-doc.org/stdlib/libdoc/zlib/rdoc/Zlib/GzipReader.html) that supports uncompressing a compressed [`IO`](http://ruby-doc.org/core/IO.html) object a line at a time.\n\n```ruby\nreader = Zlib::GzipReader.new(io)\nreader.each_line do |line|\n  # do something with line\nend\n```\n\nMany JSON libraries support passing an `IO` object as an input source and some support yielding objects as they are parsed.\n\nAs a lot of libraries report supporting an \"IO-like\" object, the missing piece is being able to turn something like the S3 client interface into an IO. I was hoping there'd be an interface like Ruby's [`Enumerable`](https://ruby-doc.org/core/Enumerable.html) (where you need only implement `each`) but for creating your own `IO`-compatible class. Sadly, this doesn't seem to exist and the IO interface is pretty large.\n\nInspired by [`Enumerator`](https://ruby-doc.org/core/Enumerator.html), I wanted to provide the easiest possible way to convert any streaming input source into an `IO` and tried to reverse engineer exactly which methods on `IO` classes like `CSV` and `Zlib::GzipReader` actually use.\n\nWhile I had [some success](https://github.com/mudge/readable/blob/master/spec/readable_spec.rb#L7-L28), usage of `IO` methods is pretty inconsistent. Yajl has [its own wrapper for `GzipReader`](https://github.com/brianmario/yajl-ruby/blob/master/lib/yajl/gzip/stream_reader.rb) because its `read` implementation does not match `IO`'s. More damningly, you can't plug together a `Zlib::GzipReader` and the default `JSON` parser as `Zlib::GzipReader#to_io` returns the inner, compressed source and not an `IO`-compatible object as intended.\n\nIf there was a smaller, well-defined interface for `IO` (ala `Enumerable`) then it might be more ergonomic to model everything as a stream that you can glue together but for now this is a bit of a failed experiment.\n\n## License\n\nCopyright © 2018 Paul Mucur\n\nDistributed under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmudge%2Freadable","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmudge%2Freadable","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmudge%2Freadable/lists"}