{"id":51469539,"url":"https://github.com/openlab-x/pipestage","last_synced_at":"2026-07-06T15:30:50.588Z","repository":{"id":367427293,"uuid":"1280517405","full_name":"openlab-x/pipestage","owner":"openlab-x","description":"A Python library for building async data pipelines with safe staged processing and bounded concurrency.","archived":false,"fork":false,"pushed_at":"2026-06-26T00:36:14.000Z","size":2072,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-26T01:11:25.412Z","etag":null,"topics":["async","asyncio","concurrency","data-pipeline","openlabx","pipeline","pipestage","python","python-library","stream"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openlab-x.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-25T17:00:31.000Z","updated_at":"2026-06-26T00:36:18.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/openlab-x/pipestage","commit_stats":null,"previous_names":["openlab-x/pipestage"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/openlab-x/pipestage","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openlab-x%2Fpipestage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openlab-x%2Fpipestage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openlab-x%2Fpipestage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openlab-x%2Fpipestage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openlab-x","download_url":"https://codeload.github.com/openlab-x/pipestage/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openlab-x%2Fpipestage/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35197534,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-06T02:00:07.184Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","asyncio","concurrency","data-pipeline","openlabx","pipeline","pipestage","python","python-library","stream"],"created_at":"2026-07-06T15:30:49.985Z","updated_at":"2026-07-06T15:30:50.569Z","avatar_url":"https://github.com/openlab-x.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pipestage\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/openlab-x/pipestage/main/assets/logo.png\" width=\"400\"/\u003e\n\n![Python](https://img.shields.io/badge/Python-3.11%2B-blue)\n![License](https://img.shields.io/badge/License-MIT-green)\n![Version](https://img.shields.io/badge/Version-0.1.1-gray)\n![Dependencies](https://img.shields.io/badge/Dependencies-Zero-brightgreen)\n\n\u003c/div\u003e\n\n```python\nfrom pipestage import stream\n\nawait (\n    stream(documents)\n    .flat_map(chunk,  concurrency=8)\n    .map(embed,       concurrency=16)\n    .batch(100)\n    .for_each(upsert, concurrency=4)\n)\n```\n\n## About\n\npipestage is an open-source Python library for building async data pipelines with safe staged processing and bounded concurrency. It is developed by OpenLabX and built entirely on the Python standard library with zero runtime dependencies.\n\nThe key difference from raw `asyncio.gather()`: gather is a barrier - stage 2 cannot start until every item from stage 1 is done. pipestage stages are lazy async generators that overlap in time, so stage 2 starts consuming stage 1's output as soon as the first item is ready.\n\npipestage is aimed at crawlers, ingestion jobs, API fan-out, file processing, and LLM data pipelines.\n\n## Table of Contents\n\n- [About](#about)\n- [Features](#features)\n- [Install](#install)\n- [Quick Start](#quick-start)\n- [API](#api)\n  - [stream()](#stream)\n  - [.map()](#map)\n  - [.filter()](#filter)\n  - [.flat_map()](#flat_map)\n  - [.batch()](#batch)\n  - [.collect()](#collect)\n  - [.for_each()](#for_each)\n  - [Async iteration](#async-iteration)\n- [Error Handling](#error-handling)\n- [Examples](#examples)\n- [Project Structure](#project-structure)\n  - [Architecture](#architecture)\n  - [Concurrency Model](#concurrency-model)\n- [Dependencies](#dependencies)\n- [Python Versions Tested](#python-versions-tested)\n- [Source Code Version 0.1.1](#source-code-version-011)\n- [Known Issues at v0.1.1](#known-issues-at-v011)\n- [Contributing](#contributing)\n- [License](#license)\n- [Contact](#contact)\n\n## Features\n\n- **Stage Overlap**: All stages run simultaneously. No gather barriers between steps.\n- **Bounded Concurrency**: Per-stage semaphore. Set concurrency once, the rest is handled.\n- **Ordered or Unordered**: Preserve input order, or emit results as ready. One flag.\n- **Sync and Async**: Pass sync or async functions anywhere. No wrapping required.\n- **Async Generator Source**: `stream()` accepts any async iterable directly.\n- **Fail-Fast Error Handling**: First exception cancels in-flight tasks and propagates unchanged.\n- **Zero Dependencies**: Pure Python standard library. Nothing to install except pipestage itself.\n\n## Install\n\n**From PyPI:**\n```bash\npip install pipestage\n```\n\n**From source:**\n```bash\ngit clone https://github.com/openlab-x/pipestage.git\ncd pipestage\npip install -e .\n```\n\nRequires Python 3.11 or later. No runtime dependencies.\n\n## Quick Start\n\n**Concurrent fetch and parse:**\n```python\nfrom pipestage import stream\n\nresults = await (\n    stream(urls)\n    .map(fetch, concurrency=20)\n    .filter(lambda r: r[\"status\"] == 200)\n    .map(parse, concurrency=8)\n    .collect()\n)\n```\n\n**Batch DB inserts:**\n```python\nawait (\n    stream(records)\n    .map(transform, concurrency=16)\n    .batch(100)\n    .for_each(insert_batch, concurrency=4)\n)\n```\n\n**LLM fan-out - emit as ready:**\n```python\nresults = await (\n    stream(prompts)\n    .map(call_llm, concurrency=8, ordered=False)\n    .collect()\n)\n```\n\n**Async generator source:**\n```python\nresults = await (\n    stream(scan_directory())\n    .map(read_file, concurrency=10)\n    .filter(lambda f: f[\"word_count\"] \u003e= 500)\n    .collect()\n)\n```\n\n## API\n\n### stream()\n\nCreate a pipeline from any sync or async iterable.\n\n```python\nstream([1, 2, 3])\nstream(range(1000))\nstream(async_generator())\n```\n\n### .map()\n\n```python\n.map(fn, *, concurrency=1, ordered=True)\n```\n\nApply `fn` to every item. `fn` may be sync or async. `ordered=False` emits results as they finish instead of preserving input order.\n\n### .filter()\n\n```python\n.filter(pred, *, concurrency=1, ordered=True)\n```\n\nKeep only items for which `pred` returns truthy.\n\n### .flat_map()\n\n```python\n.flat_map(fn, *, concurrency=1, ordered=True)\n```\n\nMap `fn` over each item then flatten one level. `fn` should return a list, generator, or async iterable.\n\n### .batch()\n\n```python\n.batch(size)\n```\n\nGroup items into lists of at most `size` elements. The final batch may be smaller.\n\n### .collect()\n\nTerminal. Consume the pipeline and return all results as a list.\n\n```python\nresults = await stream(items).map(fn).collect()\n```\n\n### .for_each()\n\n```python\n.for_each(fn, *, concurrency=1, ordered=True)\n```\n\nTerminal. Consume the pipeline calling `fn` on each item for side effects. Return values are discarded.\n\n### Async iteration\n\n`Stream` is an async iterable. Use it directly without `collect()`.\n\n```python\nasync for item in stream(records).map(transform, concurrency=8):\n    print(item)\n```\n\n## Error Handling\n\nThe pipeline fails fast by default. The first exception stops the pipeline, cancels in-flight tasks, and propagates the original exception unchanged to the caller.\n\n```python\ntry:\n    await stream(urls).map(fetch, concurrency=10).collect()\nexcept RuntimeError as e:\n    print(e)\n```\n\nTo continue past individual failures, handle exceptions inside `fn`:\n\n```python\nasync def safe_fetch(url):\n    try:\n        return await fetch(url)\n    except Exception:\n        return None\n\nresults = await (\n    stream(urls)\n    .map(safe_fetch, concurrency=10)\n    .filter(lambda r: r is not None)\n    .collect()\n)\n```\n\n## Examples\n\nEach feature in `examples/` has three files: a raw asyncio implementation, a pipestage implementation, and a compare script that runs both and prints timing and line-count metrics.\n\n```bash\npython examples/compare_embed.py   # single comparison\npython examples/run_all.py         # all eight\n```\n\n| # | Example | Features | Key result |\n|---|---------|----------|------------|\n| 1 | Fetch and Parse | map, filter | Stage overlap: ~1.7x speedup |\n| 2 | Batch Inserts | map, batch, for_each | ~1.6x speedup |\n| 3 | Fan-out | ordered=False | 6x lower time-to-first-result |\n| 4 | Paginated Search | flat_map | Replaces gather + nested loop |\n| 5 | Resilient Calls | error handling | Error logic stays in fn |\n| 6 | Notifications | for_each | No Lock needed for shared state |\n| 7 | File Processing | async generator source | Streams without materializing |\n| 8 | RAG Pipeline | flat_map, map, batch, for_each | All stages overlap simultaneously |\n\n## Project Structure\n\n```\nsrc/pipestage/\n    __init__.py     - public entry point: stream()\n    _stream.py      - Stream class, fluent API\n    _ops.py         - async generator stages\n    _utils.py       - internal helpers\ntests/\n    test_basic.py\n    test_concurrency.py\n    test_errors.py\nexamples/\n    raw_X.py        - plain asyncio implementations\n    ps_X.py         - pipestage implementations\n    compare_X.py    - timing comparisons\n```\n\n### Architecture\n\nEvery transformation returns a new `Stream` wrapping an async generator. Nothing executes until `collect()` or `for_each()` is awaited. Each stage pulls from the previous one - all stages run simultaneously.\n\n### Concurrency Model\n\n| concurrency | ordered | Behavior |\n|---|---|---|\n| 1 | any | Serial. No tasks created. |\n| \u003e 1 | True | Semaphore limits active tasks. Results in input order. |\n| \u003e 1 | False | Results emitted via Queue as tasks complete. |\n\n## Dependencies\n\n**Runtime:** none.\n\n**Development:**\n```bash\npip install -e \".[dev]\"\n# installs: pytest, pytest-asyncio, ruff, mypy\n```\n\n## Python Versions Tested\n\n- [x] **Python 3.11**\n- [x] **Python 3.12**\n- [x] **Python 3.13**\n- [x] **Python 3.14**\n\n## Source Code Version 0.1.1\n\n- **Core pipeline**: `stream()`, `map`, `filter`, `flat_map`, `batch`, `collect`, `for_each`\n- **Concurrent execution**: ordered and unordered modes with `asyncio.Semaphore`\n- **Async iteration**: `Stream` usable directly in `async for` loops\n- **Full test suite**: 44 tests across correctness, concurrency, and error propagation\n- **Zero runtime dependencies**: Python 3.11+ standard library only\n\n## Known Issues at v0.1.1\n\n- All source items are consumed upfront before any results are yielded. For very large sources this can accumulate many Task objects in memory.\n- No per-item timeout. A hung `fn` call blocks its concurrency slot indefinitely.\n- If a consumer breaks out of `async for` early, in-flight tasks keep running until the event loop cleans them up.\n\n## Contributing\n\nWe welcome contributions.\n\n1. Give the project a star.\n2. Follow us on GitHub.\n3. Fork the repository.\n4. Create a new branch for your feature or fix.\n5. Make your changes and add tests.\n6. Submit a pull request.\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n\n## Contact\n\nIn pursuit of innovation,  \n**OpenLabX Team**\n\n- **Website**: [https://openlabx.com](https://openlabx.com)\n- **Email**: contact@openlabx.com\n\n**Follow Us:**\n\n\u003cdiv align=\"center\"\u003e\n\n| \u003ca href=\"https://www.instagram.com/openlabx_official/\" target=\"_blank\"\u003e\u003cstrong\u003eInstagram\u003c/strong\u003e\u003c/a\u003e | \u003ca href=\"https://x.com/openlabx\" target=\"_blank\"\u003e\u003cstrong\u003eX (formerly Twitter)\u003c/strong\u003e\u003c/a\u003e | \u003ca href=\"https://www.facebook.com/openlabx/\" target=\"_blank\"\u003e\u003cstrong\u003eFacebook\u003c/strong\u003e\u003c/a\u003e | \u003ca href=\"https://www.youtube.com/@OpenLabX\" target=\"_blank\"\u003e\u003cstrong\u003eYouTube\u003c/strong\u003e\u003c/a\u003e | \u003ca href=\"https://github.com/openlab-x\" target=\"_blank\"\u003e\u003cstrong\u003eGitHub\u003c/strong\u003e\u003c/a\u003e |\n|---|---|---|---|---|\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenlab-x%2Fpipestage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenlab-x%2Fpipestage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenlab-x%2Fpipestage/lists"}