{"id":13484294,"url":"https://github.com/brianmario/yajl-ruby","last_synced_at":"2025-05-14T22:05:08.273Z","repository":{"id":550271,"uuid":"180613","full_name":"brianmario/yajl-ruby","owner":"brianmario","description":"A streaming JSON parsing and encoding library for Ruby (C bindings to yajl)","archived":false,"fork":false,"pushed_at":"2024-12-17T14:30:06.000Z","size":3605,"stargazers_count":1488,"open_issues_count":77,"forks_count":173,"subscribers_count":30,"default_branch":"master","last_synced_at":"2025-05-07T21:13:44.911Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://rdoc.info/projects/brianmario/yajl-ruby","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brianmario.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2009-04-20T07:49:09.000Z","updated_at":"2025-04-15T23:48:59.000Z","dependencies_parsed_at":"2024-05-01T10:35:41.380Z","dependency_job_id":"97744e6b-941e-4f6a-96c1-e30d08a02f7a","html_url":"https://github.com/brianmario/yajl-ruby","commit_stats":{"total_commits":731,"total_committers":53,"mean_commits":13.79245283018868,"dds":0.2913816689466484,"last_synced_commit":"63760720e58d8cb818d59ae6c4f3d96760cd7854"},"previous_names":[],"tags_count":65,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brianmario%2Fyajl-ruby","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brianmario%2Fyajl-ruby/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brianmario%2Fyajl-ruby/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brianmario%2Fyajl-ruby/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brianmario","download_url":"https://codeload.github.com/brianmario/yajl-ruby/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253415848,"owners_count":21904939,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T17:01:22.011Z","updated_at":"2025-05-14T22:05:08.152Z","avatar_url":"https://github.com/brianmario.png","language":"C","readme":"# YAJL C Bindings for Ruby\n\nThis gem is a C binding to the excellent YAJL JSON parsing and generation library.\n\nYou can read more info at the project's website http://lloyd.github.com/yajl or check out its code at http://github.com/lloyd/yajl.\n\n## Features\n\n* JSON parsing and encoding directly to and from an IO stream (file, socket, etc) or String. Compressed stream parsing and encoding supported for Bzip2, Gzip and Deflate.\n* Parse and encode *multiple* JSON objects to and from streams or strings continuously.\n* JSON gem compatibility API - allows yajl-ruby to be used as a drop-in replacement for the JSON gem\n* Basic HTTP client (only GET requests supported for now) which parses JSON directly off the response body *as it's being received*\n* ~3.5x faster than JSON.generate\n* ~1.9x faster than JSON.parse\n* ~4.5x faster than YAML.load\n* ~377.5x faster than YAML.dump\n* ~1.5x faster than Marshal.load\n* ~2x faster than Marshal.dump\n\n## How to install\n\nGo ahead and install it as usual:\n\n```\ngem install yajl-ruby\n```\n\nOr use your Gemfile:\n\n``` ruby\ngem 'yajl-ruby', require: 'yajl'\n```\n\n## Example of use\n\nNOTE: I'm building up a collection of small examples in the examples (http://github.com/brianmario/yajl-ruby/tree/master/examples) folder.\n\nFirst, you're probably gonna want to require it:\n\n``` ruby\nrequire 'yajl'\n```\n\n### Parsing\n\nThen maybe parse some JSON from:\n\na File IO\n\n``` ruby\njson = File.new('test.json', 'r')\nparser = Yajl::Parser.new\nhash = parser.parse(json)\n```\n\nor maybe a StringIO\n\n``` ruby\njson = StringIO.new(\"...some JSON...\")\nparser = Yajl::Parser.new\nhash = parser.parse(json)\n```\n\nor maybe STDIN\n\n```\ncat someJsonFile.json | ruby -ryajl -e \"puts Yajl::Parser.parse(STDIN).inspect\"\n```\n\nOr lets say you didn't have access to the IO object that contained JSON data, but instead\nonly had access to chunks of it at a time. No problem!\n\n(Assume we're in an EventMachine::Connection instance)\n\n``` ruby\ndef post_init\n  @parser = Yajl::Parser.new(:symbolize_keys =\u003e true)\nend\n\ndef object_parsed(obj)\n  puts \"Sometimes one pays most for the things one gets for nothing. - Albert Einstein\"\n  puts obj.inspect\nend\n\ndef connection_completed\n  # once a full JSON object has been parsed from the stream\n  # object_parsed will be called, and passed the constructed object\n  @parser.on_parse_complete = method(:object_parsed)\nend\n\ndef receive_data(data)\n  # continue passing chunks\n  @parser \u003c\u003c data\nend\n```\n\nOr if you don't need to stream it, it'll just return the built object from the parse when it's done.\nNOTE: if there are going to be multiple JSON strings in the input, you *must* specify a block or callback as this\nis how yajl-ruby will hand you (the caller) each object as it's parsed off the input.\n\n``` ruby\nobj = Yajl::Parser.parse(str_or_io)\n```\n\nOr how about a JSON API HTTP request?\nThis actually makes a request using a raw TCPSocket, then parses the JSON body right off the socket. While it's being received over the wire!\n\n``` ruby\nrequire 'uri'\nrequire 'yajl/http_stream'\n\nurl = URI.parse(\"http://search.twitter.com/search.json?q=engineyard\")\nresults = Yajl::HttpStream.get(url)\n```\n\nOr do the same request, with Gzip and Deflate output compression support (also supports Bzip2, if loaded):\n(this does the same raw socket Request, but transparently parses the compressed response body)\n\n``` ruby\nrequire 'uri'\nrequire 'yajl/gzip'\nrequire 'yajl/deflate'\nrequire 'yajl/http_stream'\n\nurl = URI.parse(\"http://search.twitter.com/search.json?q=engineyard\")\nresults = Yajl::HttpStream.get(url)\n```\n\nSince yajl-ruby parses JSON as a stream, supporting APIs like Twitter's Streaming API are a piece-of-cake.\nYou can simply supply a block to `Yajl::HttpStream.get`, which is used as the callback for when a JSON object has been\nunserialized off the stream. For the case of this Twitter Streaming API call, the callback gets fired a few times a second (depending on your connection speed).\nThe code below is all that's needed to make the request and stream unserialized Ruby hashes off the response, continuously.\nYou'll note that I've enabled the :symbolize_keys parser option as well. Doing so is much more efficient for parsing JSON streams with\nlots of repetitive keys - for things like result sets or multiple API requests - than the same parse with string keys.\nThis is because Ruby will reuse (and never GC) its symbol table. Be that as it may, if you want to parse JSON strings with random key names\nit's much better to leave string keys enabled (the default), so they can get GC'd later.\n\n``` ruby\nrequire 'uri'\nrequire 'yajl/http_stream'\n\nuri = URI.parse(\"http://#{username}:#{password}@stream.twitter.com/spritzer.json\")\nYajl::HttpStream.get(uri, :symbolize_keys =\u003e true) do |hash|\n  puts hash.inspect\nend\n```\n\nOr how about parsing directly from a compressed file?\n\n``` ruby\nrequire 'yajl/bzip2'\n\nfile = File.new('some.json.bz2', 'r')\nresult = Yajl::Bzip2::StreamReader.parse(file)\n```\n\n### Encoding\n\nSince yajl-ruby does everything using streams, you simply need to pass the object to encode, and the IO to write the stream to (this happens in chunks).\n\nThis allows you to encode JSON as a stream, writing directly to a socket\n\n``` ruby\nsocket = TCPSocket.new('192.168.1.101', 9000)\nhash = {:foo =\u003e 12425125, :bar =\u003e \"some string\", ... }\nYajl::Encoder.encode(hash, socket)\n```\n\nOr what if you wanted to compress the stream over the wire?\n\n``` ruby\nrequire 'yajl/gzip'\nsocket = TCPSocket.new('192.168.1.101', 9000)\nhash = {:foo =\u003e 12425125, :bar =\u003e \"some string\", ... }\nYajl::Gzip::StreamWriter.encode(hash, socket)\n```\n\nOr what about encoding multiple objects to JSON over the same stream?\nThis example will encode and send 50 JSON objects over the same stream, continuously.\n\n``` ruby\nsocket = TCPSocket.new('192.168.1.101', 9000)\nencoder = Yajl::Encoder.new\n50.times do\n  hash = {:current_time =\u003e Time.now.to_f, :foo =\u003e 12425125}\n  encoder.encode(hash, socket)\nend\n```\n\nUsing `EventMachine` and you want to encode and send in chunks?\n(Assume we're in an `EventMachine::Connection` instance)\n\n``` ruby\ndef post_init\n   # Passing a :terminator character will let us determine when the encoder\n   # is done encoding the current object\n   @encoder = Yajl::Encoder.new\n   motd_contents = File.read(\"/path/to/motd.txt\")\n   status = File.read(\"/path/to/huge/status_file.txt\")\n   @motd = {:motd =\u003e motd_contents, :system_status =\u003e status}\nend\n\ndef connection_completed\n  # The encoder will do its best to hand you data in chunks that\n  # are around 8kb (but you may see some that are larger)\n  #\n  # It should be noted that you could have also assigned the _on_progress_ callback\n  # much like you can assign the _on_parse_complete_ callback with the parser class.\n  # Passing a block (like below) essentially tells the encoder to use that block\n  # as the callback normally assigned to _on_progress_.\n  #\n  # Send our MOTD and status\n  @encoder.encode(@motd) do |chunk|\n    if chunk.nil? # got our terminator, encoding is done\n      close_connection_after_writing\n    else\n      send_data(chunk)\n    end\n  end\nend\n```\n\nBut to make things simple, you might just want to let yajl-ruby do all the hard work for you and just hand back\na string when it's finished. In that case, just don't provide and IO or block (or assign the on_progress callback).\n\n``` ruby\nstr = Yajl::Encoder.encode(obj)\n```\n\nYou can also use `Yajl::Bzip2::StreamWriter` and `Yajl::Deflate::StreamWriter`. So you can pick whichever fits your CPU/bandwidth sweet-spot.\n\n### HTML Safety\n\nIf you plan on embedding the output from the encoder in the DOM, you'll want to make sure you use the html_safe option on the encoder. This will escape all '/' characters to ensure no closing tags can be injected, preventing XSS.\n\nMeaning the following should be perfectly safe:\n\n``` html\n\u003cscript type=\"text/javascript\"\u003e\n  var escaped_str = \u003c%= Yajl::Encoder.encode(\"\u003c/script\u003e\u003cscript\u003ealert('hi!');\u003c/script\u003e\", :html_safe =\u003e true) %\u003e;\n\u003c/script\u003e\n```\n\n## JSON gem Compatibility API\n\nThe JSON gem compatibility API isn't enabled by default. You have to explicitly require it like so:\n\n``` ruby\nrequire 'yajl/json_gem'\n```\n\nThat's right, you can just replace `\"require 'json'\"` with the line above and you're done!\n\nThis will require yajl-ruby itself, as well as enable its JSON gem compatibility API.\n\nThis includes the following API:\n\nJSON.parse, JSON.generate, JSON.pretty_generate, JSON.load, JSON.dump\nand all of the #to_json instance method overrides for Ruby's primitive objects\n\n\nOnce the compatibility API is enabled, your existing or new project should work as if the JSON gem itself were being used. Only you'll be using Yajl ;)\n\nThere are a lot more possibilities that I'd love to see other gems/plugins for someday.\n\nSome ideas:\n\n* parsing logs in JSON format\n* a Rails plugin - DONE! (http://github.com/technoweenie/yajl-rails)\n* official support in Rails 3 - DONE (http://github.com/rails/rails/commit/a96bf4ab5e73fccdafb78b99e8a122cc2172b505)\n * and is the default (if installed) - http://github.com/rails/rails/commit/63bb955a99eb46e257655c93dd64e86ebbf05651\n* Rack middleware (ideally the JSON body could be handed to the parser while it's still being received, this is apparently possible with Unicorn)\n* JSON API clients (http://github.com/brianmario/freckle-api)\n\n## Benchmarks\n\nAfter I finished implementation - this library performs close to the same as the current JSON.parse (C gem) does on small/medium files.\n\nBut on larger files, and higher amounts of iteration, this library was around 2x faster than JSON.parse.\n\nThe main benefit of this library is in its memory usage.\nSince it's able to parse the stream in chunks, its memory requirements are very, very low.\n\nHere's what parsing a 2.43MB JSON file off the filesystem 20 times looks like:\n\n### Memory Usage\n\n#### Average\n\n* Yajl::Parser#parse: 32MB\n* JSON.parse: 54MB\n* ActiveSupport::JSON.decode: 63MB\n\n#### Peak\n\n* Yajl::Parser#parse: 32MB\n* JSON.parse: 57MB\n* ActiveSupport::JSON.decode: 67MB\n\n### Parse Time\n\n* Yajl::Parser#parse: 4.54s\n* JSON.parse: 5.47s\n* ActiveSupport::JSON.decode: 64.42s\n\n### Encode Time\n\n* Yajl::Encoder#encode: 3.59s\n* JSON#to_json: 6.2s\n* ActiveSupport::JSON.encode: 45.58s\n\n### Compared to YAML\n\nNOTE: I converted the 2.4MB JSON file to YAML for this test.\n\n#### Parse Time (from their respective formats)\n\n* Yajl::Parser#parse: 4.33s\n* JSON.parse: 5.37s\n* YAML.load: 19.47s\n\n#### Encode Time (to their respective formats)\n\n* Yajl::Encoder#encode: 3.47s\n* JSON#to_json: 6.6s\n* YAML.dump(obj, io): 1309.93s\n\n### Compared to Marshal.load/Marshal.dump\n\nNOTE: I converted the 2.4MB JSON file to a Hash and a dump file from Marshal.dump for this test.\n\n#### Parse Time (from their respective formats)\n\n* Yajl::Parser#parse: 4.54s\n* JSON.parse: 7.40s\n* Marshal.load: 7s\n\n#### Encode Time (to their respective formats)\n\n* Yajl::Encoder#encode: 2.39s\n* JSON#to_json: 8.37s\n* Marshal.dump: 4.66s\n\n## Third Party Sources Bundled\n\nThis project includes code from the BSD licensed yajl project, copyright 2007-2009 Lloyd Hilaiel\n\n## Special Thanks \u0026 Contributors\n\nFor those of you using yajl-ruby out in the wild, please hit me up on Twitter (brianmario) or send me a message here on the Githubs describing the site and how you're using it. I'd love to get a list going!\n\nI've had a lot of inspiration, and a lot of help. Thanks to everyone who's been a part of this and those to come!\n\n* Lloyd Hilaiel - http://github.com/lloyd - for writing Yajl!!\n* Josh Ferguson - http://github.com/besquared - for peer-pressuring me into getting back into C; it worked ;) Also tons of support over IM\n* Jonathan Novak - http://github.com/cypriss - pointer-hacking help\n* Tom Smith - http://github.com/rtomsmith - pointer-hacking help\n* Rick Olson - http://github.com/technoweenie - for making an ActiveSupport patch with support for this library and teasing me that it might go into Rails 3. You sure lit a fire under my ass and I got a ton of work done because of it! :)\n* The entire Github Crew - http://github.com/ - my inspiration, time spent writing this, finding Yajl, So many-MANY other things wouldn't have been possible without this awesome service. I owe you guys some whiskey at Kilowatt.\n* Ben Burkert - http://github.com/benburkert\n* Aman Gupta - http://github.com/tmm1 - tons of suggestions and inspiration for the most recent features, and hopefully more to come ;)\n* Filipe Giusti\n* Jonathan George\n* Luke Redpath\n* Neil Berkman\n* Pavel Valodzka\n* Rob Sharp\n","funding_links":[],"categories":["C","High Performance Libraries","others","Optimizations","Gems"],"sub_categories":["Application Insights","Serialization"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrianmario%2Fyajl-ruby","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrianmario%2Fyajl-ruby","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrianmario%2Fyajl-ruby/lists"}