{"id":16118759,"url":"https://github.com/michel-kraemer/actson-rs","last_synced_at":"2025-04-05T17:07:07.347Z","repository":{"id":176009395,"uuid":"654219161","full_name":"michel-kraemer/actson-rs","owner":"michel-kraemer","description":"🎬 A reactive (or non-blocking, or asynchronous) JSON parser","archived":false,"fork":false,"pushed_at":"2025-03-14T04:21:09.000Z","size":1599,"stargazers_count":38,"open_issues_count":5,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-29T16:08:48.488Z","etag":null,"topics":["asynchronous","big-data","json","non-blocking","non-blocking-io","parser","reactive","streaming"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michel-kraemer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-15T16:28:49.000Z","updated_at":"2025-03-03T21:09:19.000Z","dependencies_parsed_at":"2023-12-10T18:29:17.710Z","dependency_job_id":"3c95d69e-3850-4793-856b-764d205ced5f","html_url":"https://github.com/michel-kraemer/actson-rs","commit_stats":{"total_commits":105,"total_committers":4,"mean_commits":26.25,"dds":0.02857142857142858,"last_synced_commit":"30ca6ea22cb0131a47dd612955bbf59465da0fea"},"previous_names":["michel-kraemer/actson-rs"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michel-kraemer%2Factson-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michel-kraemer%2Factson-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michel-kraemer%2Factson-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michel-kraemer%2Factson-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michel-kraemer","download_url":"https://codeload.github.com/michel-kraemer/actson-rs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247369952,"owners_count":20927928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asynchronous","big-data","json","non-blocking","non-blocking-io","parser","reactive","streaming"],"created_at":"2024-10-09T20:50:33.036Z","updated_at":"2025-04-05T17:07:07.327Z","avatar_url":"https://github.com/michel-kraemer.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Actson [![Actions Status](https://github.com/michel-kraemer/actson-rs/workflows/Rust/badge.svg)](https://github.com/michel-kraemer/actson-rs/actions) [![MIT license](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) [![Latest Version](https://img.shields.io/crates/v/actson.svg)](https://crates.io/crates/actson) [![Documentation](https://img.shields.io/docsrs/actson/latest)](https://docs.rs/actson/latest/actson/)\n\nActson is a low-level JSON parser for reactive applications and non-blocking I/O. It is event-based and can be used in asynchronous code (for\nexample in combination with [Tokio](https://tokio.rs/)).\n\n\u003cdiv align=\"center\"\u003e\n  \u003cbr\u003e\n  \u003cbr\u003e\n  \u003cimg width=\"550\" src=\"assets/teaser.webp\" alt=\"Teaser Image\"\u003e\n  \u003cbr\u003e\n  \u003cbr\u003e\n  \u003cbr\u003e\n\u003c/div\u003e\n\n## Why another JSON parser?\n\n* **Non-blocking.** Reactive applications should use non-blocking I/O so that no thread needs to wait indefinitely for a shared resource to become available (see the\n  [Reactive Manifesto](http://www.reactivemanifesto.org/)). Actson supports this pattern.\n* **Big Data.** Actson can handle arbitrarily large JSON text without having to completely load it into memory. It is very fast and achieves constant parsing throughput (see the [Performance](#performance) section below).\n* **Event-based.** Actson produces events during parsing and can be used for streaming. For example, if you write an HTTP server, you can receive a file and parse it at the same time.\n\nActson was primarily developed for the [GeoJSON](http://geojson.org/) support in [GeoRocket](http://georocket.io), a high-performance reactive data store for geospatial files. For this application, we needed a way to parse very large JSON files with varying contents. The files are received through an HTTP server, parsed into JSON events while they are being read from the socket, and indexed into a database at the same time. The whole process runs asynchronously.\n\nIf this use case sounds familiar, then Actson might be a good solution for you. Read more about its [performance](#performance) and how it [compares to Serde JSON](#should-i-use-actson-or-serde-json) below.\n\n## Usage\n\n### Push-based parsing\n\nPush-based parsing is the most flexible way of using Actson. Push new bytes\ninto a `PushJsonFeeder` and then let the parser consume them until it returns\n`Some(JsonEvent::NeedMoreInput)`. Repeat this process until you receive\n`None`, which means the end of the JSON text has been reached. The parser\nreturns `Err` if the JSON text is invalid or some other error has occurred.\n\nThis approach is very low-level but gives you the freedom to provide new bytes\nto the parser whenever they are available and to generate JSON events whenever\nyou need them.\n\n```rust\nuse actson::{JsonParser, JsonEvent};\nuse actson::feeder::{PushJsonFeeder, JsonFeeder};\n\nlet json = r#\"{\"name\": \"Elvis\"}\"#.as_bytes();\n\nlet feeder = PushJsonFeeder::new();\nlet mut parser = JsonParser::new(feeder);\nlet mut i = 0;\nwhile let Some(event) = parser.next_event().unwrap() {\n    match event {\n        JsonEvent::NeedMoreInput =\u003e {\n            // feed as many bytes as possible to the parser\n            i += parser.feeder.push_bytes(\u0026json[i..]);\n            if i == json.len() {\n                parser.feeder.done();\n            }\n        }\n\n        JsonEvent::FieldName =\u003e assert!(matches!(parser.current_str(), Ok(\"name\"))),\n        JsonEvent::ValueString =\u003e assert!(matches!(parser.current_str(), Ok(\"Elvis\"))),\n\n        _ =\u003e {} // there are many other event types you may process here\n    }\n}\n```\n\n### Asynchronous parsing with Tokio\n\nActson can be used with Tokio to parse JSON asynchronously.\n\nThe main idea here is to call `JsonParser::next_event()` in a loop to\nparse the JSON document and to produce events. Whenever you get\n`JsonEvent::NeedMoreInput`, call `AsyncBufReaderJsonFeeder::fill_buf()`\nto asynchronously read more bytes from the input and to provide them to\nthe parser.\n\n\u003e [!NOTE]\n\u003e The `tokio` feature has to be enabled for this. It is disabled\nby default.\n\n```rust\nuse tokio::fs::File;\nuse tokio::io::{self, AsyncReadExt, BufReader};\n\nuse actson::{JsonParser, JsonEvent};\nuse actson::tokio::AsyncBufReaderJsonFeeder;\n\n#[tokio::main]\nasync fn main() {\n    let file = File::open(\"tests/fixtures/pass1.txt\").await.unwrap();\n    let reader = BufReader::new(file);\n\n    let feeder = AsyncBufReaderJsonFeeder::new(reader);\n    let mut parser = JsonParser::new(feeder);\n    while let Some(event) = parser.next_event().unwrap() {\n        match event {\n            JsonEvent::NeedMoreInput =\u003e parser.feeder.fill_buf().await.unwrap(),\n            _ =\u003e {} // do something useful with the event\n        }\n    }\n}\n```\n\n### Parsing from a `BufReader`\n\n`BufReaderJsonFeeder` allows you to feed the parser from a `std::io::BufReader`.\n\n\u003e [!NOTE]\n\u003e By following this synchronous and blocking approach, you are missing\n\u003e out on Actson's reactive properties. We recommend using Actson together\n\u003e with Tokio instead to parse JSON asynchronously (see above).\n\n```rust\nuse actson::{JsonParser, JsonEvent};\nuse actson::feeder::BufReaderJsonFeeder;\n\nuse std::fs::File;\nuse std::io::BufReader;\n\nlet file = File::open(\"tests/fixtures/pass1.txt\").unwrap();\nlet reader = BufReader::new(file);\n\nlet feeder = BufReaderJsonFeeder::new(reader);\nlet mut parser = JsonParser::new(feeder);\nwhile let Some(event) = parser.next_event().unwrap() {\n    match event {\n        JsonEvent::NeedMoreInput =\u003e parser.feeder.fill_buf().unwrap(),\n        _ =\u003e {} // do something useful with the event\n    }\n}\n```\n\n### Parsing a slice of bytes\n\nFor convenience, `SliceJsonFeeder` allows you to feed the parser from a slice\nof bytes.\n\n```rust\nuse actson::{JsonParser, JsonEvent};\nuse actson::feeder::SliceJsonFeeder;\n\nlet json = r#\"{\"name\": \"Elvis\"}\"#.as_bytes();\n\nlet feeder = SliceJsonFeeder::new(json);\nlet mut parser = JsonParser::new(feeder);\nwhile let Some(event) = parser.next_event().unwrap() {\n    match event {\n        JsonEvent::FieldName =\u003e assert!(matches!(parser.current_str(), Ok(\"name\"))),\n        JsonEvent::ValueString =\u003e assert!(matches!(parser.current_str(), Ok(\"Elvis\"))),\n        _ =\u003e {}\n    }\n}\n```\n\n### Parsing into a Serde JSON Value\n\nFor testing and compatibility reasons, Actson is able to parse a byte slice\ninto a [Serde JSON](https://github.com/serde-rs/json) Value.\n\n\u003e [!NOTE]\n\u003e You need to enable the `serde_json` feature for this.\n\n```rust\nuse actson::serde_json::from_slice;\n\nlet json = r#\"{\"name\": \"Elvis\"}\"#.as_bytes();\nlet value = from_slice(json).unwrap();\n\nassert!(value.is_object());\nassert_eq!(value[\"name\"], \"Elvis\");\n```\n\nHowever, if you find yourself doing this, you probably don't need the reactive\nfeatures of Actson and your data seems to completely fit into memory. In this\ncase, you're most likely better off using Serde JSON directly (see the [comparison](#should-i-use-actson-or-serde-json) below)\n\n### Parsing in streaming mode (multiple top-level JSON values)\n\nIf you want to parse a stream of multiple top-level JSON values, you can enable\nstreaming mode. Values must be clearly separable. They must either be\nself-delineating values (i.e. arrays, objects, strings) or keywords (i.e.\n`true`, `false`, `null`), or they must be separated either by white space, at\nleast one self-delineating value, or at least one keyword.\n\n#### Example streams\n\n`1 2 3 true 4 5`\n\n`[1,2,3][4,5,6]{\"key\": \"value\"} 7 8 9`\n\n`\"a\"\"b\"[1, 2, 3] {\"key\": \"value\"}`\n\n#### Example\n\n```rust\nuse actson::feeder::SliceJsonFeeder;\nuse actson::options::JsonParserOptionsBuilder;\nuse actson::{JsonEvent, JsonParser};\n\nlet json = r#\"1 2\"\"{\"key\":\"value\"}\n[\"a\",\"b\"]4true\"#.as_bytes();\n\nlet feeder = SliceJsonFeeder::new(json);\nlet mut parser = JsonParser::new_with_options(\n    feeder,\n    JsonParserOptionsBuilder::default()\n        .with_streaming(true)\n        .build(),\n);\n\nlet mut events = Vec::new();\nwhile let Some(e) = parser.next_event().unwrap() {\n    events.push(e);\n}\n\nassert_eq!(events, vec![\n    JsonEvent::ValueInt,\n    JsonEvent::ValueInt,\n    JsonEvent::ValueString,\n    JsonEvent::StartObject,\n    JsonEvent::FieldName,\n    JsonEvent::ValueString,\n    JsonEvent::EndObject,\n    JsonEvent::StartArray,\n    JsonEvent::ValueString,\n    JsonEvent::ValueString,\n    JsonEvent::EndArray,\n    JsonEvent::ValueInt,\n    JsonEvent::ValueTrue,\n]);\n```\n\n## Performance\n\nActson has been optimized to perform best with large files. It scales linearly, which means it exhibits constant parsing speed and memory consumption regardless of the size of the input JSON text.\n\nThe figures below show the parser's [throughput](#throughput-higher-is-better) and [runtime](#runtime-lower-is-better) for different [GeoJSON](https://geojson.org/) input files and in comparison to [Serde JSON](https://github.com/serde-rs/json).\n\nActson with a `BufReader` performs best on every file tested (`actson-bufreader` benchmark). Its throughput stays constant and its runtime only grows linearly with the input size.\n\nThe same applies to the other Actson benchmarks using Tokio (`actson-tokio` and `actson-tokio-twotasks`). Asynchronous code has a slight overhead, which is mostly compensated for by using two concurrently running Tokio tasks (`actson-tokio-twotasks`).\n\nThe `serde-value` benchmark shows that the parser's throughput collapses the larger the file becomes. This is because it has to load its entire contents into memory (into a Serde JSON `Value`). The `serde-struct` benchmark deserializes the file into a struct that replicates the [GeoJSON](https://geojson.org/) format. It suffers from the same issue as the `serde-value` benchmark, namely that the whole file has to be loaded into memory. In this case, the impact on the throughput is not visible in the figure since the custom struct is smaller than Serde JSON's `Value` and the test system had 36 GB of RAM.\n\nThe `serde-custom-deser` benchmark is the only Serde benchmark whose performance is on par with the slowest asynchronous Actson benchmark `actson-tokio` (which runs with only one Tokio task). This is because `serde-custom-deser` uses a custom deserializer, which avoids having to load the whole file into memory (see [example on the Serde website](https://serde.rs/stream-array.html)). This very specific implementation only works because the structure of the input files is known and the used GeoJSON files are not deeply nested. The solution is not generalizable.\n\nRead more about the individual benchmarks and the test files [here](geojson_benchmarks/README.md).\n\n### Throughput (higher is better)\n\n\u003cimg width=\"750\" src=\"assets/results-throughput.svg\" alt=\"Throughput\"\u003e\n\n*Tested on a MacBook Pro 16\" 2023 with an M3 Pro Chip and 36 GB of RAM.*\n\n### Runtime (lower is better)\n\n\u003cimg width=\"750\" src=\"assets/results-runtime.svg\" alt=\"Runtime\"\u003e\n\n*Tested on a MacBook Pro 16\" 2023 with an M3 Pro Chip and 36 GB of RAM.*\n\n## Should I use Actson or Serde JSON?\n\nAs can be seen from the benchmarks above, Actson performs best with large files. However, if your JSON input files are small (a few KB or maybe 1 or 2 MB), you should probably stick to [Serde JSON](https://github.com/serde-rs/json), which is a rock-solid, battle-tested parser and which will perform extremely fast in this case.\n\nOn the other hand, if you require scalability and your input files can be of arbitrary size, or if you want to parse JSON asynchronously, use Actson.\n\nThe aim of this section is not to make one parser appear better than the other. Actson and Serde JSON are two very distinct libraries that each have advantages and disadvantages. The following table may help you decide whether you require Actson or if you should prefer Serde JSON:\n\n| Actson | Serde JSON |\n|--------|------------|\n| The input files can be of arbitrary size (several GB) | The input files are just a few KB or MB in size |\n| The JSON text is streamed, e.g. through a web server | The JSON text is stored on the file system or in memory |\n| You want to concurrently read and parse the JSON text | Sequential parsing is sufficient |\n| Parsing should not block other tasks in your application ([reactive](http://www.reactivemanifesto.org/) [programming](https://en.wikipedia.org/wiki/Reactive_programming)) | The JSON text is so small that parsing is quick enough, or your application is not reactive and does not run multiple tasks in parallel |\n| You want to process individual JSON events | You prefer convenience and do not care about events |\n| The structure of the JSON text can vary or is not known at all | The structure is very well known |\n| You don't require deserialization (mapping the JSON text to a struct), or deserialization is impossible due to the varying or unknown structure of the JSON text | You want and can deserialize the JSON text into a struct |\n\n## Compliance\n\nWe test Actson thoroughly to make sure it is compliant with [RFC 8259](https://datatracker.ietf.org/doc/html/rfc8259),\ncan parse valid JSON documents, and rejects invalid ones.\n\nBesides own unit tests, Actson passes the tests from\n[JSON_checker.c](http://www.json.org/JSON_checker/) and all 283 accept and\nreject tests from the very comprehensive\n[JSON Parsing Test Suite](https://github.com/nst/JSONTestSuite/).\n\n## Other languages\n\nBesides this implementation in Rust here, there is a\n[Java implementation](https://github.com/michel-kraemer/actson).\n\n## Acknowledgments\n\nThe event-based parser code and the JSON files used for testing are largely\nbased on the file [JSON_checker.c](http://www.json.org/JSON_checker/) and\nthe JSON test suite from [JSON.org](http://www.json.org/) originally released\nunder [this license](LICENSE_JSON_checker) (basically MIT license).\n\nThe directory `tests/json_test_suite` is a Git submodule pointing to the\n[JSON Parsing Test Suite](https://github.com/nst/JSONTestSuite/) curated by\nNicolas Seriot and released under the MIT license.\n\n## License\n\nActson is released under the **MIT license**. See the\n[LICENSE](LICENSE) file for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichel-kraemer%2Factson-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichel-kraemer%2Factson-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichel-kraemer%2Factson-rs/lists"}