https://github.com/shivkun/corstream
A coroutine composition framework for declarative async pipelines in Python
https://github.com/shivkun/corstream
async composition coroutine declarative framework pipeline python
Last synced: 11 months ago
JSON representation
A coroutine composition framework for declarative async pipelines in Python
- Host: GitHub
- URL: https://github.com/shivkun/corstream
- Owner: shivkun
- License: mit
- Created: 2025-05-06T14:18:56.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2025-05-06T19:57:27.000Z (11 months ago)
- Last Synced: 2025-05-08T02:52:40.683Z (11 months ago)
- Topics: async, composition, coroutine, declarative, framework, pipeline, python
- Language: Python
- Homepage:
- Size: 35.2 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
CorStream
[](https://www.python.org/)
[]()
[]()
[]()
[](/LICENSE)
---
A coroutine composition framework for Python that enables declarative, streaming-style async pipelines.
## 📝 Table of Contents
- [About](#about)
- [Getting Started](#getting_started)
- [Deployment](#deployment)
- [Usage](#usage)
- [Built Using](#built_using)
- [TODO](../TODO.md)
- [Contributing](../CONTRIBUTING.md)
- [Authors](#authors)
- [Acknowledgments](#acknowledgement)
**CorStream** is a Python library that lets you build elegant, composable pipelines using asynchronous iterables and coroutine-based operators. Think of it like a blend of `asyncio`, `RxPy`, and Unix pipes — but designed for readability, type safety, and modern Python development.
With CorStream, you can easily:
- Transform and filter async data flows
- Batch, throttle, or log stream data
- Apply async functions with concurrency control
- Collect or reduce outputs with simple syntax
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.
### Prerequisites
You’ll need Python 3.9 or higher and [Poetry](https://python-poetry.org/) installed:
```
python3 --version
# Should be 3.9+
curl -sSL https://install.python-poetry.org | python3 -
```
### Installing
Clone the repo and install dependencies with Poetry:
```
git clone https://github.com/shivkun/corstream.git
cd corstream
poetry install
```
CorStream is also published on PyPI, you can install it directly with:
```
pip install corstream
# or use Poetry
poetry add corstream
```
You can now run tests, examples, or start building pipelines!
To run all tests:
```
poetry run pytest
```
### Break down into end-to-end tests
Each operator and sink has its own test file under `tests/`.
Example:
```
tests/test_map.py # Tests for .map()
tests/test_batch.py # Tests for .batch()
tests/test_to_list.py # Tests for .to_list()
```
### And coding style tests
CorStream follows strict linting and formatting with `black`, `mypy`, and `ruff`:
```
poetry run black corstream/
poetry run ruff check corstream/
poetry run mypy corstream/
```
Here's a simple pipeline:
```python
from corstream import Stream
async def get_email(user_id: int) -> str:
return f"user{user_id}@example.com"
async def send_batch(batch: list[str]) -> None:
print("Sending:", batch)
await (
Stream
.from_iterable(range(1, 11))
.filter(lambda x: x % 2 == 0)
.map_async(get_email, max_concurrent=3)
.batch(5)
.log(label="batch")
.for_each(send_batch)
)
```
- [Python 3.9+](https://www.python.org/)
- [Poetry](https://python-poetry.org/)
- [AsyncIO](https://docs.python.org/3/library/asyncio.html)
- [Pytest](https://docs.pytest.org/)
- [@shivkun](https://github.com/shivkun) — Design & Implementation
See also the list of [contributors](https://github.com/shivkun/corstream/contributors).
- Inspiration from functional programming and reactive streams
- Thanks to the maintainers of `asyncio`, `RxPy`, and `toolz`