{"id":51514713,"url":"https://github.com/jamesgober/pipe-io","last_synced_at":"2026-07-08T10:01:42.298Z","repository":{"id":358805581,"uuid":"1242971785","full_name":"jamesgober/pipe-io","owner":"jamesgober","description":"Data pipeline and stream processing primitives. Typed source-transform-sink pipelines with backpressure, batching, windowing, and error isolation per stage. Runtime-agnostic. The missing middle ground between raw iterators and full distributed stream processing.","archived":false,"fork":false,"pushed_at":"2026-05-19T07:49:47.000Z","size":95,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-19T07:51:45.693Z","etag":null,"topics":["backpressure","batching","pipeline","reps","rust","streaming"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jamesgober.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-18T23:54:40.000Z","updated_at":"2026-05-19T07:47:09.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jamesgober/pipe-io","commit_stats":null,"previous_names":["jamesgober/pipe-io"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/jamesgober/pipe-io","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesgober%2Fpipe-io","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesgober%2Fpipe-io/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesgober%2Fpipe-io/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesgober%2Fpipe-io/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jamesgober","download_url":"https://codeload.github.com/jamesgober/pipe-io/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesgober%2Fpipe-io/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35260671,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-08T02:00:06.796Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backpressure","batching","pipeline","reps","rust","streaming"],"created_at":"2026-07-08T10:01:41.558Z","updated_at":"2026-07-08T10:01:42.289Z","avatar_url":"https://github.com/jamesgober.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n    \u003cimg width=\"99\" alt=\"Rust logo\" src=\"https://raw.githubusercontent.com/jamesgober/rust-collection/72baabd71f00e14aa9184efcb16fa3deddda3a0a/assets/rust-logo.svg\"\u003e\n    \u003cbr\u003e\n    \u003cstrong\u003epipe-io\u003c/strong\u003e\n    \u003cbr\u003e\n    \u003csup\u003e\u003csub\u003eTYPED DATA PIPELINE PRIMITIVES FOR RUST\u003c/sub\u003e\u003c/sup\u003e\n\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://crates.io/crates/pipe-io\"\u003e\u003cimg alt=\"crates.io\" src=\"https://img.shields.io/crates/v/pipe-io.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://crates.io/crates/pipe-io\"\u003e\u003cimg alt=\"downloads\" src=\"https://img.shields.io/crates/d/pipe-io.svg?color=0099ff\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://docs.rs/pipe-io\"\u003e\u003cimg alt=\"docs.rs\" src=\"https://docs.rs/pipe-io/badge.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/rust-lang/rfcs/blob/master/text/2495-min-rust-version.md\" title=\"MSRV\"\u003e\u003cimg alt=\"MSRV\" src=\"https://img.shields.io/badge/MSRV-1.75%2B-blue\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/jamesgober/pipe-io/actions/workflows/ci.yml\"\u003e\u003cimg alt=\"CI\" src=\"https://github.com/jamesgober/pipe-io/actions/workflows/ci.yml/badge.svg\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003cstrong\u003eSource -\u0026gt; transform -\u0026gt; sink, with backpressure, batching, windowing, and isolated error lanes per stage.\u003c/strong\u003e\n    \u003cbr\u003e\n    \u003cem\u003eThe missing middle ground between raw iterators and full distributed stream processing.\u003c/em\u003e\n\u003c/p\u003e\n\n\u003cp\u003e\n    \u003cb\u003epipe-io\u003c/b\u003e is an in-process stream processor for Rust. It gives you a typed builder for source / transform / sink pipelines, bounded buffers for backpressure, count / byte / age-triggered batching, tumbling / sliding / session windows behind a pluggable \u003ccode\u003eClock\u003c/code\u003e, per-stage \u003ccode\u003eFailFast\u003c/code\u003e / \u003ccode\u003eContinue\u003c/code\u003e / \u003ccode\u003eDeadLetter\u003c/code\u003e error policies, and a synchronous and threaded driver - all on \u003cstrong\u003ezero runtime dependencies\u003c/strong\u003e with \u003ccode\u003e#![forbid(unsafe_code)]\u003c/code\u003e.\n\u003c/p\u003e\n\n\u003cp\u003e\n    Use it when raw \u003ccode\u003eIterator\u003c/code\u003e is too thin (no backpressure, no batching, no per-stage error isolation) and a full distributed processor is too heavy. \u003cstrong\u003eSingle host, single process, in memory.\u003c/strong\u003e\n\u003c/p\u003e\n\n\u003cp\u003e\n    Custom executors are first-class: the \u003ccode\u003eDriver\u003c/code\u003e trait is open, so a future \u003ccode\u003epipe-io-tokio\u003c/code\u003e adapter (or your own rayon-backed driver) plugs in without coordinating with this crate.\n\u003c/p\u003e\n\n\u003chr\u003e\n\n## Features\n\n### Pipeline composition\n\n- **Typed builder** - the carrier type is tracked at compile time across every stage transition. Wrong types are a build error, not a runtime panic.\n- **Closure adapters** - `map`, `filter`, `filter_map`, `flat_map`, `inspect`, `try_map` all live on the builder. No `.next()` ceremony.\n- **Custom stages** - implement `Stage` directly when state spreads across multiple fields or you need 1:N emission.\n- **Custom sources and sinks** - the `Source` and `Sink` traits are two methods each (with `flush` / `close` defaults).\n\n### Backpressure, batching, windowing\n\n- **Bounded buffers** - inter-stage edges are bounded; full downstream slows the upstream.\n- **Count / byte / age batching** - `BatchPolicy::new().max_items(N).max_bytes(M).max_age(D)`. Triggers OR together.\n- **Tumbling / sliding / session windows** - behind a pluggable `Clock` trait. Custom clocks for deterministic tests and embedded time sources.\n- **`ByteSize` trait** - opt-in for byte-aware batching; blanket impls for `\u0026str`, `String`, `Vec\u003cu8\u003e`, `\u0026[u8]`.\n\n### Error isolation\n\n- **`ErrorPolicy::FailFast`** (default) - first error in a stage bubbles out of `run`.\n- **`ErrorPolicy::Continue`** - drop the failing item, keep going. Useful for noisy enrichment stages.\n- **`ErrorPolicy::DeadLetter`** - route the failure to a dead-letter `Sink\u003cItem = StageFailure\u003e` installed via `.dead_letter(sink)`. Sink install order does not matter.\n- **`StageId`** - every error carries the `\u0026'static str` name of the stage that produced it.\n\n### Runtime and execution\n\n- **`SyncDriver`** - drives the pipeline on the calling thread; no_std-compatible.\n- **`ThreadedDriver`** - spawns a worker thread; the calling thread blocks on `join`.\n- **`Driver` trait** - open for external executors. Implement for tokio, rayon, glommio, or custom thread farms.\n- **`Pipeline::run`**, **`run_threaded`**, **`run_with(driver)`** - three terminal forms.\n\n### Reliability and safety\n\n- **`#![forbid(unsafe_code)]`** at the crate root.\n- **Zero runtime dependencies.** Built on `core` + `alloc` (always) plus `std` (default feature). `proptest` is a dev-only dependency.\n- **84 tests** pass under `--all-features` (32 unit + 42 integration + 10 doctest), plus **11 property tests** and **4 fuzz harnesses**.\n- **`cargo-semver-checks`** runs in CI from `0.9.0` onward and is a hard gate from `1.0.0`.\n- **MSRV 1.75**, locked. Bumps require a minor version increment.\n\n\u003e Benchmark numbers under `--release` on a developer laptop: source -\u003e null sink ~500 M items/s; source -\u003e map -\u003e sink ~260 M items/s; full three-stage chain ~170 M items/s. See [`docs/BENCH.md`](docs/BENCH.md).\n\n---\n\n## Installation\n\nAdd to `Cargo.toml`:\n\n```toml\n[dependencies]\npipe-io = \"1\"\n```\n\nFor `no_std` builds (data primitives and `SyncDriver` only):\n\n```toml\n[dependencies]\npipe-io = { version = \"1\", default-features = false }\n```\n\n### Feature flags\n\n| Flag    | Default | Effect                                                                                                |\n|---------|---------|-------------------------------------------------------------------------------------------------------|\n| `std`   | yes     | `ThreadedDriver`, `Pipeline::run_threaded`, `ChannelSource`, `ReaderSource`, `ChannelSink`, `WriterSink`, `VecSink`, `Window` / `Clock` / `SystemClock`, `BatchPolicy::max_age`, dead-letter routing. |\n\nThe `no_std` build still ships the full trait surface, closure adapters, count + byte batching, the synchronous driver, and the error model. The `std`-only items above are the ones that need `std::sync::Mutex` or `std::thread::spawn`.\n\n---\n\n## Quick start\n\n```rust\nuse pipe_io::{Pipeline, sink::VecSink};\n\nlet sink = VecSink::\u003ci64\u003e::new();\nlet handle = sink.handle();\n\nPipeline::from_iter(1..=5)\n    .map(|n: i32| i64::from(n) * 10)\n    .filter(|n: \u0026i64| *n \u003e 20)\n    .sink(sink)\n    .run()\n    .expect(\"pipeline run\");\n\nassert_eq!(handle.take(), vec![30, 40, 50]);\n```\n\n### Batching\n\n```rust\nuse pipe_io::{Pipeline, Batch, BatchPolicy, sink::VecSink};\n\nlet sink = VecSink::\u003cVec\u003cu32\u003e\u003e::new();\nlet handle = sink.handle();\n\nPipeline::from_iter(1u32..=11)\n    .batch(BatchPolicy::new().max_items(4))\n    .map(|b: Batch\u003cu32\u003e| b.into_inner())\n    .sink(sink)\n    .run()\n    .unwrap();\n\nassert_eq!(handle.take().len(), 3);\n```\n\n### Windowing\n\n```rust\nuse core::time::Duration;\nuse pipe_io::{Pipeline, Window, WindowPolicy, sink::VecSink};\n\nlet sink = VecSink::\u003cu64\u003e::new();\nPipeline::from_iter(metric_stream)\n    .window(WindowPolicy::Tumbling { size: Duration::from_secs(60) })\n    .map(|w: Window\u003cu64\u003e| w.into_inner().iter().sum::\u003cu64\u003e())\n    .sink(sink)\n    .run()?;\n```\n\n### Dead-letter routing\n\n```rust\nuse pipe_io::{ErrorPolicy, Pipeline, StageFailure, sink::VecSink};\n\nlet dlq = VecSink::\u003cStageFailure\u003e::new();\nlet dlq_handle = dlq.handle();\n\nPipeline::from_iter(records)\n    .stage_id(\"parse\")\n    .on_error(ErrorPolicy::DeadLetter)\n    .try_map(parse_row)\n    .dead_letter(dlq)\n    .sink(main_sink)\n    .run()?;\n```\n\n### Threaded driver\n\n```rust\nPipeline::from_iter(records)\n    .map(transform)\n    .sink(writer)\n    .run_threaded()?;\n```\n\n### Custom driver\n\n```rust\nuse pipe_io::driver::{Driver, RunStats, SyncDriver};\n\nstruct MyDriver;\n\nimpl Driver for MyDriver {\n    fn run\u003cS\u003e(self, pipeline: pipe_io::Pipeline\u003cS\u003e) -\u003e pipe_io::Result\u003cRunStats\u003e\n    where\n        S: pipe_io::Source + Send + 'static,\n        S::Item: Send + 'static,\n        S::Error: Send + 'static,\n    {\n        SyncDriver::new().run(pipeline)\n    }\n}\n```\n\nMore patterns are in [`docs/GUIDE.md`](docs/GUIDE.md) and the [`examples/`](examples/) directory.\n\n---\n\n## Examples\n\nEvery example lives under [`examples/`](examples/) and runs with `cargo run --example \u003cname\u003e`:\n\n| Example         | Demonstrates                                                  |\n|-----------------|---------------------------------------------------------------|\n| `basic`         | Minimal map / filter pipeline collecting into `VecSink`.      |\n| `batching`      | `BatchPolicy::new().max_items(N)` count-triggered batching.   |\n| `windowing`     | Tumbling window with a deterministic scripted clock.          |\n| `dead_letter`   | `ErrorPolicy::DeadLetter` + `.dead_letter(sink)` routing.     |\n| `threaded`      | `Pipeline::run_threaded()` on a spawned worker thread.        |\n| `custom_driver` | Implementing the `Driver` trait with timing instrumentation.  |\n| `custom_source` | Implementing `Source` for a stateful Fibonacci producer.      |\n| `etl`           | Multi-stage CSV ETL with `Continue`, enrichment, batching.    |\n\n---\n\n## Status\n\n**`1.0.0` - Stable.** The public API is frozen. Backwards-incompatible changes after `1.0.0` require a major version bump per Semantic Versioning. See [`REPS.md`](REPS.md) section 8 for the binding policy and [`docs/API.md`](docs/API.md) for the complete public-symbol reference.\n\n**Stability rules (from `1.0.0` forward):**\n\n- **Patch (`1.0.x`)** - bug fixes, doc improvements, internal performance work, test additions. No new public items.\n- **Minor (`1.x.0`)** - pure additions to the public surface, new opt-in features, new variants on enums reserved for growth, MSRV bumps.\n- **Major (`2.0.0`)** - removes, renames, or signature changes of public symbols, or non-opt-in runtime dependency additions.\n\nThe `cargo-semver-checks` CI job enforces this for the public API surface from `1.0.0` onward.\n\n---\n\n## Documentation\n\n- [User guide (`docs/GUIDE.md`)](docs/GUIDE.md) - 11-section walkthrough of every public surface.\n- [API reference (`docs/API.md`)](docs/API.md) - offline mirror of the rustdoc on docs.rs.\n- [Project specification (`REPS.md`)](REPS.md) - the binding public surface and stability policy.\n- [Benchmarks (`docs/BENCH.md`)](docs/BENCH.md) - methodology and measured numbers.\n- [Migration guide (`docs/MIGRATION.md`)](docs/MIGRATION.md) - per-version upgrade notes.\n- [Release notes (`docs/release/`)](docs/release/) - one note per tagged release.\n- [Examples (`examples/`)](examples/) - 8 runnable example files.\n\nExternal:\n\n- **docs.rs** - \u003chttps://docs.rs/pipe-io\u003e\n- **crates.io** - \u003chttps://crates.io/crates/pipe-io\u003e\n\n---\n\n## Version compatibility\n\n| Crate version | MSRV    | Status                |\n|---------------|---------|-----------------------|\n| `1.0.x`       | `1.75`  | Stable (this release) |\n| `0.9.x`       | `1.75`  | Pre-1.0 stabilization |\n| `0.x` earlier | `1.75`  | Feature releases      |\n\nMSRV is locked at `1.75`. A bump requires a minor version increment and a CHANGELOG entry under `### Changed` with rationale, per `REPS.md` section 6.\n\n---\n\n## Contributing\n\nThe public surface is locked at `1.0.0`. Issues, bug reports, and pull requests for bug fixes, documentation improvements, and internal performance work are welcome. Additions to the public API need a minor version bump and a documented motivation in `.dev/` planning material.\n\nSee [`.dev/DIRECTIVES.md`](.dev/DIRECTIVES.md) (not published) for the project's development directives: code style, build matrix, banned-word policy, and commit conventions.\n\n---\n\n\u003cdiv id=\"license\"\u003e\n\n## License\n\nLicensed under the Apache License, Version 2.0. See [`LICENSE`](LICENSE) for the full text.\n\n### Contribution\n\nUnless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you shall be licensed as above, without any additional terms or conditions.\n\n\u003c/div\u003e\n\n\u003c!--\n:: COPYRIGHT\n=============================================== --\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003cbr\u003e\n  \u003ch2\u003e\u003c/h2\u003e\n  \u003csup\u003eCOPYRIGHT \u003csmall\u003e\u0026copy;\u003c/small\u003e 2026 \u003cstrong\u003eJAMES GOBER.\u003c/strong\u003e\u003c/sup\u003e\n\u003c/div\u003e\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesgober%2Fpipe-io","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjamesgober%2Fpipe-io","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesgober%2Fpipe-io/lists"}