{"id":13878380,"url":"https://github.com/jhollinger/json-emitter","last_synced_at":"2025-07-16T14:31:59.455Z","repository":{"id":56879534,"uuid":"168261811","full_name":"jhollinger/json-emitter","owner":"jhollinger","description":"Efficiently generate tons of JSON in Ruby!","archived":false,"fork":false,"pushed_at":"2023-09-22T15:30:33.000Z","size":52,"stargazers_count":22,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-10-31T14:46:24.558Z","etag":null,"topics":["json","ruby"],"latest_commit_sha":null,"homepage":"https://jhollinger.github.io/json-emitter/","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jhollinger.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-30T01:57:56.000Z","updated_at":"2024-05-11T15:09:07.000Z","dependencies_parsed_at":"2023-02-19T10:45:55.952Z","dependency_job_id":null,"html_url":"https://github.com/jhollinger/json-emitter","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhollinger%2Fjson-emitter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhollinger%2Fjson-emitter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhollinger%2Fjson-emitter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhollinger%2Fjson-emitter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jhollinger","download_url":"https://codeload.github.com/jhollinger/json-emitter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226138849,"owners_count":17579496,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","ruby"],"created_at":"2024-08-06T08:01:47.866Z","updated_at":"2025-07-16T14:31:59.446Z","avatar_url":"https://github.com/jhollinger.png","language":"Ruby","readme":"# JsonEmitter\n\n*NOTE this project may appear inactive, but that's just because it's **done** (I think). Enjoy!*\n\nJsonEmitter is a library for efficiently generating very large bits of JSON in Ruby. Need to generate a JSON array of 10,000 database records without eating up all your RAM? No problem! Objects? Nested structures? JsonEmitter has you covered.\n\nUse JsonEmitter in your Rack/Rails/Sinatra/Grape API to stream large JSON responses without worrying (much) about RAM or HTTP timeouts. Use it to write large JSON objects to your filesystem, S3, or ~~a 3D printer~~ anywhere else!\n\n# HTTP Chunked Responses\n\nThese examples will use the Order enumerator to generate chunks of JSON and send them to the client as more chunks are generated. No more than 500 orders will be in memory at a time, regardless of how many orders there are. And only small portions of the JSON will be in memory at once, no matter how much we're generating. (NOTE There **are** limits to how long something will stream - your app server, nginx/apache, your browser, etc. may cut it off eventually.)\n\n```ruby\nenumerator = Order.\n  where(\"created_at \u003e= ?\", 1.year.ago).\n  find_each(batch_size: 500)\n```\n\n**Rails**\n\n```ruby\nclass OrdersController \u003c ApplicationController\n  def index\n    headers[\"Content-Type\"] = \"application/json\"\n    headers[\"Last-Modified\"] = Time.now.ctime.to_s\n    self.response_body = JsonEmitter.array(enumerator, rack: env) { |order|\n      order.to_h\n    }\n  end\nend\n```\n\n**Sinatra**\n\n```ruby\nget \"/orders\" do\n  content_type :json\n  JsonEmitter.array(enumerator, rack: env) { |order|\n    order.to_h\n  }\nend\n```\n\n**Grape**\n\n```ruby\nget :orders do\n  stream JsonEmitter.array(enumerator, rack: env) { |order|\n    ApiV1::Entities::Order.new(order)\n  }\nend\n```\n\n**Rack**\n\n```ruby\nmap \"/orders\" do\n  run -\u003e(env) {\n    stream = JsonEmitter.array(enumerator, rack: env) { |order|\n      order.to_h\n    }\n    [200, {\"Content-Type\" =\u003e \"application/json\"}, stream]\n  }\nend\n```\n\n## Sending objects\n\nYou may also stream Hashes as JSON objects. Keys must be Strings or Symbols, but values may be anything: literals, Enumerators, Arrays, other Hashes, or Procs that return any of those.\n\n```ruby\nJsonEmitter.object({\n  orders: Order.find_each.lazy.map { |order|\n    {id: order.id, desc: order.description}\n  },\n  big_text: -\u003e() { load_tons_of_text },\n}, rack: env)\n```\n\n## Rack middleware won't work!\n\n**IMPORTANT** Your Rack middleware will be *finished* by the time your JSON is built! So if you're depending on middleware to set `Time.zone`, report exceptions, etc. it won't work here. Fortunately, you can use `JsonEmitter.wrap` and `JsonEmitter.error` as replacements.\n\nPut these somewhere like `config/initializers/json_emitter.rb`.\n\n### JsonEmitter.wrap\n\n```ruby\n# Ensure that ActiveRecord connections are returned to the connection pool\nJsonEmitter.wrap do\n  -\u003e(app) { ActiveRecord::Base.with_connection(\u0026app.call) }\nend\n\nJsonEmitter.wrap do\n  # Get TZ at the call site\n  current_tz = Time.zone\n  \n  # Return a Proc that restores the call site's TZ before building the JSON\n  -\u003e(app) {\n    default_tz = Time.zone\n    Time.zone = current_tz\n    res = app.call\n    Time.zone = default_tz\n    res\n  }\nend\n```\n\n### JsonEmitter.error\n\n```ruby\nJsonEmitter.error do |ex, context|\n  Airbrake.notify(ex, {\n    request_path: context.request\u0026.path,\n    query_string: context.request\u0026.query_string,\n  })\nend\n```\n\n## Returning errors\n\nWhen streaming an HTTP response, you can't change the response code once you start sending data. So if you hit an error after you start, you need another way to communicate errors to the client.\n\nOne way is to always steam an object that includes an `errors` field. Any errors will be collected while the `Enumerator` is running. After it's finished, they'll be added to the JSON object.\n\n```ruby\n  def get_data\n    errors = []\n    enum = Enumerator.new { |y|\n      finished = false\n      until finished\n        data, errs, finished = get_data_chunk\n        if errs\n          errors += errs\n          finished = true\n          next\n        end\n        data.each { x| y \u003c\u003c x }\n      end\n    }\n    return enum, -\u003e { errors }\n  end\n  \n  items_enum, errors_proc = get_data\n  JsonEmitter.object({\n    items: items_enum,\n    errors: errors_proc,\n  })\n```\n\n# Non-HTTP uses\n\n`JsonEmitter.array` takes an `Enumerable` and returns a stream that generates chunks of JSON.\n\n```ruby\nJsonEmitter.array(enumerator).each { |json_chunk|\n  # write json_chunk somewhere\n}\n```\n\nStreams have a `#write` method for writing directly to a `File` or `IO` object.\n\n```ruby\nFile.open(\"~/out.json\", \"w+\") { |f|\n  JsonEmitter.array(enumerator).write f\n}\n```\n\n# License\n\nMIT License. See LICENSE for details.\n\n# Copyright\n\nCopyright (c) 2019 Jordan Hollinger.\n","funding_links":[],"categories":["Ruby"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjhollinger%2Fjson-emitter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjhollinger%2Fjson-emitter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjhollinger%2Fjson-emitter/lists"}