{"id":19329657,"url":"https://github.com/akanoa/nom-stream-parser","last_synced_at":"2026-06-08T23:31:17.206Z","repository":{"id":224715223,"uuid":"764019413","full_name":"Akanoa/nom-stream-parser","owner":"Akanoa","description":"An Rust lib which allow to downstream result data from an upstream","archived":false,"fork":false,"pushed_at":"2024-04-05T13:45:44.000Z","size":130,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-12T05:40:02.503Z","etag":null,"topics":["nom","parsing","performance","rust","stream"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Akanoa.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-27T10:36:21.000Z","updated_at":"2024-02-27T10:46:10.000Z","dependencies_parsed_at":"2024-11-10T02:29:28.898Z","dependency_job_id":"77d21daa-b44e-47d3-aa2b-719412152474","html_url":"https://github.com/Akanoa/nom-stream-parser","commit_stats":null,"previous_names":["akanoa/nom-stream-parser"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Akanoa/nom-stream-parser","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Akanoa%2Fnom-stream-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Akanoa%2Fnom-stream-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Akanoa%2Fnom-stream-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Akanoa%2Fnom-stream-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Akanoa","download_url":"https://codeload.github.com/Akanoa/nom-stream-parser/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Akanoa%2Fnom-stream-parser/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34085321,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-08T02:00:07.615Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nom","parsing","performance","rust","stream"],"created_at":"2024-11-10T02:29:17.808Z","updated_at":"2026-06-08T23:31:17.185Z","avatar_url":"https://github.com/Akanoa.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Nom Stream Parser\n\n## Usage\n\n## Streaming mode\n\nThe [nom](https://docs.rs/nom/latest/nom/) crate is a parser combinator which allow to parse\nvarious kind of data.\n\nOne of its ability apart its zero-copy promise, is to be able\nto parse a stream of data.\n\nLet's take an example.\n\nYou want to parse something like this\n\n`\n(1,2,3,45)\n`\n\nA comma separated list of numbers delimited by parentheses.\n\nWith `nom`, you'll write a parse like this.\n\n```ignore\nuse nom;\nfn parser(input: \u0026[u8]) -\u003e IResult\u003c\u0026[u8], Vec\u003cu8\u003e\u003e {\n    delimited(\n        tag(\"(\"),\n        separated_list1(\n            bytes::complete::tag(\",\"),\n            map_parser(character::complete::digit1, character::complete::u8),\n        ),\n        tag(\")\"),\n    )(input)\n}\n```\n\nAs you can see I use the `complete` version of `tag` and `digit1` parsers.\n\nThis means, that this parser has two return value possible, either success or fail.\n\nBut if the input data is something like this ?\n\n`\n(1,2,\n`\n\nThis kind of data can happen when input is chunked, like when reading a socket.\n\nIs this an error or something incomplete ?\n\nThat why **nom** provides the streaming mode\n\nHere the same parser but using the streaming version.\n\n```ignore\nuse nom;\nfn parser(input: \u0026[u8]) -\u003e IResult\u003c\u0026[u8], Vec\u003cu8\u003e\u003e {\n    delimited(\n        tag(\"(\"),\n        separated_list1(\n            bytes::streaming::tag(\",\"),\n            map_parser(character::streaming::digit1, character::complete::u8),\n        ),\n        tag(\")\"),\n    )(input)\n}\n```\n\nNow there is 3 alternatives:\n\n- A success\n- An error\n- An incomplete data, the parser can't take a decision, he needs more data\n\nAs the data are `(1,2,`, the parser return an Incomplete state.\n\nSo to get a final decision we need to concatenate `(1,2,` and `3,45)`.\n\nFinally, we get the parsing decision.\n\n## Noising\n\nWhile this example can seem simple, this action of accumulating data can\nbecome tedious when data are chunked in more than two parts.\n\nAnd becomes even harder when some data in the buffering process aren't relevant\ndata but noise.\n\n```ignore\n(1,2,3,45)noise(1,2)\n```\n\nThis data may be chuked as\n\n`(1,2,`   `3,45)noi`  `se(1,2)`\n\nThis noise must be evinced.\n\n## Buffering\n\nOne of the others problematics is how to handle the buffer growing.\n\nWhat is the right size of the buffer, is it allows to grow or must\nbe preallocated and fix sized during the parse.\n\nThat's a lot of question when we need to parse incomplete data.\n\n## Downstream parsing results\n\nThe stream of data can be infinite, by definition, with no end\n\n*At which moment do we can and must release parsing results ?*\n\nWe don't know, and moreover, we need to accumulate all these results.\n\nThe solution is to downstream results as the data yields by upstream\ndata stream are sufficient for yielding a parse result.\n\nThis way we don't need to accumulate all results, just the current one.\n\n```ignore\n[data stream] ----{data chunk}----\u003e   Parser    ---\u003e [result stream]\n   upstream                          buffering         downstream\n```\n\n## API\n\nSo we want something which takes an [Iterator] or a [std::io::Read] and return another [Iterator].","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakanoa%2Fnom-stream-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakanoa%2Fnom-stream-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakanoa%2Fnom-stream-parser/lists"}