https://github.com/johnazariah/python-pipelines
Strongly Typed Pipelines in Python
https://github.com/johnazariah/python-pipelines
Last synced: 2 months ago
JSON representation
Strongly Typed Pipelines in Python
- Host: GitHub
- URL: https://github.com/johnazariah/python-pipelines
- Owner: johnazariah
- License: mit
- Created: 2024-11-26T03:21:32.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-04T13:34:28.000Z (11 months ago)
- Last Synced: 2025-02-04T14:25:41.664Z (11 months ago)
- Language: Python
- Homepage:
- Size: 58.6 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Repository README
## Overview
This repository contains two Python packages designed to streamline data processing workflows and simplify common pipeline operations. Below is an overview of each package, their use cases, and examples to help you quickly integrate them into your project.
---
## Packages
### 1. **`generics` Package**
#### Description
The `generics` package provides advanced utilities for working with generics and type annotations in Python. It extends the capabilities of Python's `Generic` and `ABC` classes, allowing developers to define, validate, and utilize parameterized types effectively.
#### Features
- Define generic classes with multiple type parameters.
- Validate type annotations and ensure correct usage of type arguments.
- Seamlessly integrate generics with abstract base classes.
#### Installation
To install the `generics` package, use the following command:
```bash
pip install generics
```
#### Quick Start
Here's an example of how to use the `generics` package:
```python
from generics.generics import TypeAnnotatedMeta
from typing import Generic, TypeVar
T = TypeVar("T")
U = TypeVar("U")
class MyGenericClass(Generic[T, U], metaclass=TypeAnnotatedMeta):
pass
# Validate type parameters
specialized_class = MyGenericClass[int, str]
print(specialized_class.T, specialized_class.U) # Output:
```
---
### 2. **`pipeline` Package**
#### Description
The `pipeline` package focuses on building and executing modular data pipelines. It provides tools for defining pipeline stages, managing input-output transformations, and handling execution with contextual data.
#### Features
- Define pipelines as a sequence of customizable stages.
- Support for contextual pipelines that use a shared dictionary to manage stage inputs and outputs.
- Special stages for producing and consuming data at specific points.
#### Installation
To install the `pipeline` package, use the following command:
```bash
pip install pipeline
```
#### Quick Start
Here's an example of how to use the `pipeline` package:
```python
from pipeline.pipeline.pipeline import Pipeline, PipelineStage
class InitialStage(PipelineStage[int, int]):
def process(self, item):
return item * 2
class FinalStage(PipelineStage[int, str]):
def process(self, item):
return str(item)
pipeline = Pipeline([InitialStage(), FinalStage()])
result = pipeline.run([5, 10])
print(result) # Output: ['10', '20']
```
---
## Acknowledgments
This project is made possible thanks to the contributions of:
- Christian Smith ([GitHub](https://github.com/smith1511))
- Harshdeep Singh
- Yiwen Zhu
- Mathieu Demarne
## Contributing
We welcome contributions! Please read our [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to get started.
## License
This project is licensed under the terms of the Creative Commons Zero v1.0 Universal. See the [LICENSE](LICENSE) file for details.
## Support
If you encounter any issues or have questions, please [open an issue](https://github.com/your-repo/issues) or contact us at john.azariah@gmail.com.