Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ploomber/soopervisor

☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.
https://github.com/ploomber/soopervisor

airflow argo argo-workflows aws data-science kubeflow kubeflow-pipelines kubernetes machine-learning slurm workflow

Last synced: about 21 hours ago
JSON representation

☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.

Host: GitHub
URL: https://github.com/ploomber/soopervisor
Owner: ploomber
License: apache-2.0
Created: 2020-04-29T03:05:01.000Z (over 4 years ago)
Default Branch: master
Last Pushed: 2024-01-25T01:07:28.000Z (10 months ago)
Last Synced: 2024-04-25T17:05:25.356Z (7 months ago)
Topics: airflow, argo, argo-workflows, aws, data-science, kubeflow, kubeflow-pipelines, kubernetes, machine-learning, slurm, workflow
Language: Python
Homepage: https://soopervisor.readthedocs.io
Size: 929 KB
Stars: 42
Watchers: 6
Forks: 17
Open Issues: 24
Metadata Files:
- Readme: README.rst
- Changelog: CHANGELOG.rst
- Contributing: CONTRIBUTING.md
- License: LICENSE

Awesome Lists containing this project

README

        Soopervisor

-----------

.. image:: https://github.com/ploomber/soopervisor/workflows/CI/badge.svg

   :target: https://github.com/ploomber/soopervisor/workflows/CI/badge.svg

   :alt: CI badge

.. image:: https://github.com/ploomber/soopervisor/workflows/CI%20macOS/badge.svg

   :target: https://github.com/ploomber/soopervisor/workflows/CI%20macOS/badge.svg

   :alt: CI macOS badge

.. image:: https://github.com/ploomber/soopervisor/workflows/CI%20Windows/badge.svg

   :target: https://github.com/ploomber/soopervisor/workflows/CI%20Windows/badge.svg

   :alt: CI Windows badge

.. image:: https://img.shields.io/badge/code%20style-black-000000.svg

   :target: https://github.com/psf/black

*Tip: Deploy AI apps for free on* `Ploomber Cloud! `_

Soopervisor runs `Ploomber `_ pipelines

for batch processing (large-scale training or batch serving) or online

inference.

.. code-block:: sh

   pip install soopervisor

Check out the `documentation `_ to learn more.

*Compatible with Python 3.7 and higher.*

Supported platforms

===================

* Batch serving and large-scale training:

  * `Airflow `_

  * `Argo/Kubernetes `_

  * `AWS Batch `_

  * `Kubeflow `_

  * `SLURM `_

* Online inference:

  * `AWS Lambda `_

From notebook to a production pipeline

======================================

We also have `an example `_ that shows how to use our ecosystem of tools to

go **from a monolithic notebook to a pipeline deployed in Kubernetes.**

Usage

=====

Say that you want to train multiple models in a Kubernetes

cluster, you may create a new target environment to execute your pipeline

using Argo Workflows:

.. code-block:: sh

   soopervisor add training --backend argo-workflows

After filling in some basic configuration settings, export the pipeline with:

.. code-block:: sh

   soopervisor export training

Depending on the selected backend (Argo, Airflow, AWS Batch, or AWS Lambda),

configuration details will change, but the API remains the same:

``soopervisor add``, then ``soopervisor export``.

About Ploomber

==============

Ploomber is a big community of data enthusiasts pushing the boundaries of Data Science and Machine Learning tooling.

Whatever your skillset is, you can contribute to our mission. So whether you're a beginner or an experienced professional, you're welcome to join us on this journey!

`Click here to know how you can contribute to Ploomber. `_