{"id":16548038,"url":"https://github.com/dgraham/yajl-ffi","last_synced_at":"2025-04-21T15:30:45.198Z","repository":{"id":18203402,"uuid":"21335687","full_name":"dgraham/yajl-ffi","owner":"dgraham","description":"Ruby FFI bindings to the native YAJL streaming JSON parser.","archived":false,"fork":false,"pushed_at":"2024-04-23T02:42:53.000Z","size":57,"stargazers_count":20,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-17T01:40:17.018Z","etag":null,"topics":["ffi","json","yajl"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dgraham.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-06-30T00:53:01.000Z","updated_at":"2025-01-19T16:29:18.000Z","dependencies_parsed_at":"2022-08-25T14:51:00.386Z","dependency_job_id":null,"html_url":"https://github.com/dgraham/yajl-ffi","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dgraham%2Fyajl-ffi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dgraham%2Fyajl-ffi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dgraham%2Fyajl-ffi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dgraham%2Fyajl-ffi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dgraham","download_url":"https://codeload.github.com/dgraham/yajl-ffi/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250080493,"owners_count":21371516,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ffi","json","yajl"],"created_at":"2024-10-11T19:24:35.331Z","updated_at":"2025-04-21T15:30:44.866Z","avatar_url":"https://github.com/dgraham.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Yajl::FFI\n\nYajl::FFI is a [JSON](http://json.org) parser, based on\n[FFI](https://github.com/ffi/ffi) bindings into the native\n[YAJL](https://github.com/lloyd/yajl) library, that generates\nevents for each state change. This allows streaming both the JSON document into\nmemory and the parsed object graph out of memory to some other process.\n\nThis is similar to an XML SAX parser that generates events during parsing. There\nis no requirement for the document, or the object graph, to be fully buffered in\nmemory. Yajl::FFI is best suited for huge JSON documents that won't fit in memory.\n\n## Usage\n\nThe simplest way to parse is to read the full JSON document into memory\nand then parse it into a full object graph. This is fine for small documents\nbecause we have room for both the text and parsed object in memory.\n\n```ruby\nrequire 'yajl/ffi'\njson = File.read('/tmp/test.json')\nobj = Yajl::FFI::Parser.parse(json)\n```\n\nWhile it's possible to do this with Yajl::FFI, we should really use the\nstandard library's [json](https://github.com/flori/json) gem for documents\nlike this. It's faster because it doesn't need to generate events and notify\nobservers each time the parser changes state. It parses and builds the Ruby\nobject entirely in native code and hands it back to us, fully formed.\n\nFor larger documents, we can use an IO object to stream it into the parser.\nWe still need room for the parsed object, but the document itself is never\nfully read into memory.\n\n```ruby\nrequire 'yajl/ffi'\nstream = File.open('/tmp/test.json')\nobj = Yajl::FFI::Parser.parse(stream)\n```\n\nHowever, when streaming small documents from disk, or over the network, the\n[yajl-ruby](https://github.com/brianmario/yajl-ruby) gem will give us the best\nperformance.\n\nHuge documents arriving over the network in small chunks to an\n[EventMachine](https://github.com/eventmachine/eventmachine)\n`receive_data` loop is where Yajl::FFI is uniquely suited. Inside an\n`EventMachine::Connection` subclass we might have:\n\n```ruby\ndef post_init\n  @parser = Yajl::FFI::Parser.new\n  @parser.start_document { puts \"start document\" }\n  @parser.end_document   { puts \"end document\" }\n  @parser.start_object   { puts \"start object\" }\n  @parser.end_object     { puts \"end object\" }\n  @parser.start_array    { puts \"start array\" }\n  @parser.end_array      { puts \"end array\" }\n  @parser.key            { |k| puts \"key: #{k}\" }\n  @parser.value          { |v| puts \"value: #{v}\" }\nend\n\ndef receive_data(data)\n  begin\n    @parser \u003c\u003c data\n  rescue Yajl::FFI::ParserError =\u003e e\n    close_connection\n  end\nend\n```\n\nThe parser accepts chunks of the JSON document and parses up to the end of the\navailable buffer. Passing in more data resumes the parse from the prior state.\nWhen an interesting state change happens, the parser notifies all registered\ncallback procs of the event.\n\nThe event callback is where we can do interesting data filtering and passing\nto other processes. The above example simply prints state changes, but the\ncallbacks might look for an array named `rows` and process sets of these row\nobjects in small batches. Millions of rows, streaming over the network, can be\nprocessed in constant memory space this way.\n\n## Dependencies\n\n* [libyajl2](https://github.com/lloyd/yajl)\n\n## Library loading\n\nFFI uses the the `dlopen` system call to dynamically load the libyajl library\ninto memory at runtime. It searches the usual directories for the library file,\nlike `/usr/lib` and `/usr/local/lib`, and raises an error if it's not found.\nIf libyajl is installed in an unusual directory, we can tell `dlopen` where to\nlook by setting the `LD_LIBRARY_PATH` environment variable.\n\n```sh\n# test normal library load\n$ ruby -r 'yajl/ffi' -e 'puts Yajl::FFI::VERSION'\n\n# if it fails, specify the search path\n$ LD_LIBRARY_PATH=/somewhere/yajl/lib \\\n  ruby -r 'yajl/ffi' -e 'puts Yajl::FFI::VERSION'\n```\n\n## Installation\n\nThe libyajl library needs to be installed before this gem can bind to it.\n\n### OS X\n\nUse [Homebrew](http://brew.sh) or compile from source below.\n\n```\n$ brew install yajl\n```\n\n### Fedora\n\nFedora 20 provides libyajl2 in a package. Older versions might need to compile\nthe latest yajl version from source.\n\n```\n$ sudo yum install yajl\n```\n\n### Ubuntu\n\nUbuntu 14.04 provides a libyajl2 package. Older versions might also need to\ncompile yajl from source.\n\n```\n$ sudo apt-get install libyajl2\n```\n\n### Source\n\nBy default, this compiles and installs to `/usr/local`. Use\n`./configure -p /tmp/somewhere` to install to a different directory.\nSetting `LD_LIBRARY_PATH` will be required in that case.\n\n```\n$ git clone https://github.com/lloyd/yajl\n$ cd yajl\n$ ./configure\n$ make \u0026\u0026 make install\n```\n\n## Performance\n\nThis gem provides a benchmark script to test the relative performance of\nseveral parsers. Here's a sample run.\n\n```\n$ bin/rake benchmark\n                  user     system      total        real\njson          0.037963   0.002951   0.040914 (  0.041196)\nyajl-ruby     0.043128   0.001845   0.044973 (  0.045292)\nyajl-ffi      0.181198   0.004324   0.185522 (  0.186301)\njson-stream   2.169778   0.010984   2.180762 (  2.196817)\n```\n\nYajl::FFI is about 4x slower than the pure native parsers. JSON::Stream is a\npure Ruby parser, and it performs accordingly. But it's useful in cases where\nyou're unable to use native bindings or when the limiting factor is the\nnetwork, rather than processor speed.\n\nSo if you need to parse many small JSON documents, the json and yajl-ruby gems\nare the best options. If you need to stream, and incrementally parse, pieces of a\nlarge document in constant memory space, yajl-ffi and json-stream are good\nchoices.\n\n## Alternatives\n\n* [json](https://github.com/flori/json)\n* [yajl-ruby](https://github.com/brianmario/yajl-ruby)\n* [json-stream](https://github.com/dgraham/json-stream)\n* [application/json-seq](http://www.rfc-editor.org/rfc/rfc7464.txt)\n\n## Development\n\n```\n$ bin/setup\n$ bin/rake test\n```\n\n## License\n\nYajl::FFI is released under the MIT license. Check the LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdgraham%2Fyajl-ffi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdgraham%2Fyajl-ffi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdgraham%2Fyajl-ffi/lists"}