{"id":16116734,"url":"https://github.com/dpc/pariter","last_synced_at":"2025-08-15T22:42:56.416Z","repository":{"id":57621675,"uuid":"430988098","full_name":"dpc/pariter","owner":"dpc","description":"Parallel iterator processing library for Rust","archived":false,"fork":false,"pushed_at":"2023-08-16T22:46:46.000Z","size":3487,"stargazers_count":104,"open_issues_count":3,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-06T17:11:17.090Z","etag":null,"topics":["iterator","parallel","rust"],"latest_commit_sha":null,"homepage":"https://docs.rs/dpc-pariter","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-11-23T06:33:59.000Z","updated_at":"2025-04-05T05:01:50.000Z","dependencies_parsed_at":"2023-09-24T18:10:27.663Z","dependency_job_id":null,"html_url":"https://github.com/dpc/pariter","commit_stats":{"total_commits":42,"total_committers":3,"mean_commits":14.0,"dds":0.04761904761904767,"last_synced_commit":"ea97cc651fd600fa5df6fc02319ad66fbd55ef0a"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Fpariter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Fpariter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Fpariter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Fpariter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpc","download_url":"https://codeload.github.com/dpc/pariter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247517913,"owners_count":20951719,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["iterator","parallel","rust"],"created_at":"2024-10-09T20:25:31.710Z","updated_at":"2025-04-07T08:17:36.077Z","avatar_url":"https://github.com/dpc.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Parallel iterator processing library for Rust\n\nSee [`IteratorExt`] ([latest `IteratorExt` on docs.rs](https://docs.rs/pariter/latest/pariter/trait.IteratorExt.html))\nfor supported operations.\n\n## Notable features\n\n* drop-in replacement for standard iterators(*)\n  * preserves order\n  * lazy, somewhat like single-threaded iterators\n  * panic propagation\n* support for iterating over borrowed values using scoped threads\n* backpressure\n* profiling methods (useful for analyzing pipelined processing bottlenecks)\n\n## When to use and alternatives\n\nThis library is a good general purpose solution to adding multi-threaded\nprocessing to an existing iterator-based code. When you have\na chain of iterator steps, and would like to process one or some\nof them in parallel to speed things up, this library goes a long way\nto make it as close to a drop-in replacement as possible in all aspects.\n\nThe implementation is based on spawning thread-pools of worker threads\nand sending them work using channels, then receiving and sorting the\nresults to turn them into a normal iterator again.\n\nSending iterator items through channels is fast, but not free.\nMake sure to parallelize operations that are heavy enough to justify\noverhead of sending data through channels. E.g.\noperations involving IO or some CPU-heavy computation.\n\nYou can use `cargo bench` or view the `/docs/bench-report/report/index.html`\nlocally for criterion.rs benchmark report, but as a rule of thumb, each call to function\nbeing parallized should take more than 200ns for the parallelization\nto outweight the overheads.\n\nWhen you have a lot items **already stored in a collection**,\nthat you want to \"roll over and perform some simple computation\"\nyou probably want to use `rayon` instead. It's a library optimized\nfor parallelizing processing of whole chunks of larger set of data,\nwhich minimizes any per-item overheads.\nA downside of that is that [converting `rayon`'s iterators back to ordered\nsequencial iterator is non-trivial](https://github.com/rayon-rs/rayon/issues/210).\n\n## Usage\n\n Adding new ones based\non the existing code should be relatively easy, so PRs are welcome.\n\nIn short, if you have:\n\n```rust\n# fn step_a(x: usize) -\u003e usize {\n#   x * 7\n# }\n# \n# fn filter_b(x: \u0026usize) -\u003e bool {\n#   x % 2 == 0\n# }\n# \n# fn step_c(x: usize) -\u003e usize {\n#   x + 1\n# }\nassert_eq!(\n  (0..10)\n    .map(step_a)\n    .filter(filter_b)\n    .map(step_c).collect::\u003cVec\u003c_\u003e\u003e(),\n    vec![1, 15, 29, 43, 57]\n);\n```\n\nYou can change it to:\n\n```rust\nuse pariter::IteratorExt as _;\n# fn step_a(x: usize) -\u003e usize {\n#   x * 7\n# }\n# \n# fn filter_b(x: \u0026usize) -\u003e bool {\n#   x % 2 == 0\n# }\n# \n# fn step_c(x: usize) -\u003e usize {\n#   x + 1\n# }\nassert_eq!(\n  (0..10)\n    .map(step_a)\n    .filter(filter_b)\n    .parallel_map(step_c).collect::\u003cVec\u003c_\u003e\u003e(),\n    vec![1, 15, 29, 43, 57]\n);\n```\n\nand it will run faster (conditions apply), because\n`step_c` will run in parallel on multiple-threads.\n\n## Iterating over borrowed values\n\nHitting a `borrowed value does not live long enough` error? Looks like you\nare iterating over values containing borrowed references. Sending them over\nto different threads for processing could lead to memory unsafety issues.\nBut no problem, we got you covered.\n\nFirst, if the values you are iterating over can be cheaply cloned, just try\nadding `.cloned()` and turning them into owned values.\n\nIf you can't, you can use scoped-threads API from [`crossbeam`] crate:\n\n\n```rust\nuse pariter::{IteratorExt as _, scope};\n# fn step_a(x: \u0026usize) -\u003e usize {\n#   *x * 7\n# }\n#\n# fn filter_b(x: \u0026usize) -\u003e bool {\n#   x % 2 == 0\n# }\n#\n# fn step_c(x: usize) -\u003e usize {\n#   x + 1\n# }\nlet v : Vec\u003c_\u003e = (0..10).collect();\n\nscope(|scope| {\n  assert_eq!(\n    v\n      .iter() // iterating over `\u0026usize` now, `parallel_map` will not work\n      .parallel_map_scoped(scope, step_a)\n      .filter(filter_b)\n      .map(step_c).collect::\u003cVec\u003c_\u003e\u003e(),\n      vec![1, 15, 29, 43, 57]\n  );\n});\n\n// or:\n\nassert_eq!(\n  scope(|scope| {\n  v\n    .iter()\n    .parallel_map_scoped(scope, step_a)\n    .filter(filter_b)\n    .map(step_c).collect::\u003cVec\u003c_\u003e\u003e()}).expect(\"handle errors properly in production code\"),\n    vec![1, 15, 29, 43, 57]\n);\n```\n\nThe additional `scope` argument comes from [`crossbeam::thread::scope`] and is\nthere to enforce memory-safety. Just wrap your iterator chain in a `scope`\nwrapper that does not outlive the borrowed value, and everything will work smoothly.\n\n## Customizing settings\n\nIf you need to change settings like buffer sizes and number of threads:\n\n```rust\n# use pariter::IteratorExt as _;\nassert_eq!(\n  (0..10)\n    .map(|x| x + 1)\n    .parallel_filter_custom(|o| o.threads(16), |x| *x == 5)\n    .map(|x| x /2).collect::\u003cVec\u003c_\u003e\u003e(),\n    vec![2]\n);\n```\n## Status \u0026 plans\n\nI keep needing this exact functionality, so I've cleaned up my\nad-hoc code, put it in a proper library. I'm usually very busy,\nso if you want something added, please submit a PR.\n\nI'm open to share/transfer ownership \u0026 maintenance into reputable hands.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpc%2Fpariter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpc%2Fpariter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpc%2Fpariter/lists"}