https://github.com/izo0x90/pipeline-factory
Pipeline Factory is a Python server framework to define, manage, schedule and execute custom defined code pipelines and series of pipeline across multiple worker processes.
https://github.com/izo0x90/pipeline-factory
automation pipeline pipelines python python-framework server-framework task-runner task-scheduler tasks worker-pool workload
Last synced: 2 months ago
JSON representation
Pipeline Factory is a Python server framework to define, manage, schedule and execute custom defined code pipelines and series of pipeline across multiple worker processes.
- Host: GitHub
- URL: https://github.com/izo0x90/pipeline-factory
- Owner: izo0x90
- License: mit
- Created: 2024-07-23T18:59:14.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-07-23T21:18:26.000Z (10 months ago)
- Last Synced: 2025-02-20T03:16:55.442Z (3 months ago)
- Topics: automation, pipeline, pipelines, python, python-framework, server-framework, task-runner, task-scheduler, tasks, worker-pool, workload
- Language: Python
- Homepage: https://pypi.org/project/pipeline-factory/
- Size: 24.4 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
- License: LICENSE
Awesome Lists containing this project
README
::
O
( ))
( )
( ) )
( ) )
( ) )
( )
( )
( )
──┐ ──┐
/ │ / │
/ │ / │
/ │ / │
/ │ / │
/ │/ │
/ │ │
┌─────────┴────────┴┐
│ │
│ ┌──┐ ┌┘ ┌──┐
│ │ │ ┌─┐ │ │ │ ┌──┐
│ └──┘ │ │ └┬─┴──┴──┴──┴───
│ │ │ │* * * * * * * O
└───────────┴─┴─────┘---------------
____ _ ___ ______ __
/ __ \(_)___ ___ / (_)___ ___ / ____/___ ______/ /_____ _______ __
/ /_/ / / __ \/ _ \/ / / __ \/ _ \ / /_ / __ `/ ___/ __/ __ \/ ___/ / / /
/ ____/ / /_/ / __/ / / / / / __/ / __/ / /_/ / /__/ /_/ /_/ / / / /_/ /
/_/ /_/ .___/\___/_/_/_/ /_/\___/ /_/ \__,_/\___/\__/\____/_/ \__, /
/_/ /____/
Pipeline Factory is a Python server framework to define, manage, schedule and execute custom defined code pipelines and series of pipeline across multiple worker processes.
- Based on Fastapi and Python multiprocessing
- Define pipelines of action steps to process data, make api calls, generate artifacts etc.
- Resumable pipelines where steps can define recovery action to avoid redoing work
- Run series of pipelines based on state/ results of previous pipelines
- Cron schedule tasks to auto run or resume pipelines
- Tasks that run pipelines execute on a worker process pool
- Add python callables as custom tasks, to extend pipeline execution capabilities or support execution of arbitrary work on worker pool
Installation
============
::
pip install pipeline-factory
Example usage
=============
`Example app `_