{"id":20101587,"url":"https://github.com/ploomber/soopervisor","last_synced_at":"2025-08-21T08:30:36.921Z","repository":{"id":37014765,"uuid":"259810880","full_name":"ploomber/soopervisor","owner":"ploomber","description":"☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.","archived":true,"fork":false,"pushed_at":"2025-03-10T15:54:07.000Z","size":984,"stargazers_count":45,"open_issues_count":0,"forks_count":20,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-08-15T19:41:49.200Z","etag":null,"topics":["airflow","argo","argo-workflows","aws","data-science","kubeflow","kubeflow-pipelines","kubernetes","machine-learning","slurm","workflow"],"latest_commit_sha":null,"homepage":"https://soopervisor.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ploomber.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-29T03:05:01.000Z","updated_at":"2025-03-10T16:00:54.000Z","dependencies_parsed_at":"2024-01-05T15:28:01.019Z","dependency_job_id":"20df7a39-7e0b-4866-a1e4-f2c9b084672a","html_url":"https://github.com/ploomber/soopervisor","commit_stats":{"total_commits":531,"total_committers":17,"mean_commits":"31.235294117647058","dds":0.2994350282485876,"last_synced_commit":"9bca3193b9c2ee4ade55b94c0f21b574c44427a2"},"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"purl":"pkg:github/ploomber/soopervisor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ploomber%2Fsoopervisor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ploomber%2Fsoopervisor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ploomber%2Fsoopervisor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ploomber%2Fsoopervisor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ploomber","download_url":"https://codeload.github.com/ploomber/soopervisor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ploomber%2Fsoopervisor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271449019,"owners_count":24761582,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-21T02:00:08.990Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","argo","argo-workflows","aws","data-science","kubeflow","kubeflow-pipelines","kubernetes","machine-learning","slurm","workflow"],"created_at":"2024-11-13T17:25:52.282Z","updated_at":"2025-08-21T08:30:36.916Z","avatar_url":"https://github.com/ploomber.png","language":"Python","funding_links":[],"categories":["Simplification Tools"],"sub_categories":[],"readme":"**Important:** *This project is no longer maintained.*\n\nSoopervisor\n-----------\n\n.. image:: https://github.com/ploomber/soopervisor/workflows/CI/badge.svg\n   :target: https://github.com/ploomber/soopervisor/workflows/CI/badge.svg\n   :alt: CI badge\n\n.. image:: https://github.com/ploomber/soopervisor/workflows/CI%20macOS/badge.svg\n   :target: https://github.com/ploomber/soopervisor/workflows/CI%20macOS/badge.svg\n   :alt: CI macOS badge\n\n.. image:: https://github.com/ploomber/soopervisor/workflows/CI%20Windows/badge.svg\n   :target: https://github.com/ploomber/soopervisor/workflows/CI%20Windows/badge.svg\n   :alt: CI Windows badge\n\n.. image:: https://img.shields.io/badge/code%20style-black-000000.svg\n   :target: https://github.com/psf/black\n\n\n*Tip: Deploy AI apps for free on* `Ploomber Cloud! \u003chttps://ploomber.io/?utm_medium=github\u0026utm_source=soopervisor\u003e`_\n\n\nSoopervisor runs `Ploomber \u003chttps://github.com/ploomber/ploomber\u003e`_ pipelines\nfor batch processing (large-scale training or batch serving) or online\ninference.\n\n.. code-block:: sh\n\n   pip install soopervisor\n\n\nCheck out the `documentation \u003chttps://soopervisor.readthedocs.io/\u003e`_ to learn more.\n\n*Compatible with Python 3.7 and higher.*\n\nSupported platforms\n===================\n\n* Batch serving and large-scale training:\n\n  * `Airflow \u003chttps://soopervisor.readthedocs.io/en/latest/tutorials/airflow.html\u003e`_\n  * `Argo/Kubernetes \u003chttps://soopervisor.readthedocs.io/en/latest/tutorials/kubernetes.html\u003e`_\n  * `AWS Batch \u003chttps://soopervisor.readthedocs.io/en/latest/tutorials/aws-batch.html\u003e`_\n  * `Kubeflow \u003chttps://soopervisor.readthedocs.io/en/latest/tutorials/kubeflow.html\u003e`_\n  * `SLURM \u003chttps://soopervisor.readthedocs.io/en/latest/tutorials/slurm.html\u003e`_\n\n* Online inference:\n\n  * `AWS Lambda \u003chttps://soopervisor.readthedocs.io/en/latest/tutorials/aws-lambda.html\u003e`_\n\n\nFrom notebook to a production pipeline\n======================================\n\nWe also have `an example \u003chttps://soopervisor.readthedocs.io/en/latest/tutorials/workflow.html\u003e`_ that shows how to use our ecosystem of tools to\ngo **from a monolithic notebook to a pipeline deployed in Kubernetes.**\n\nUsage\n=====\n\nSay that you want to train multiple models in a Kubernetes\ncluster, you may create a new target environment to execute your pipeline\nusing Argo Workflows:\n\n.. code-block:: sh\n\n   soopervisor add training --backend argo-workflows\n\nAfter filling in some basic configuration settings, export the pipeline with:\n\n.. code-block:: sh\n\n   soopervisor export training\n\n\nDepending on the selected backend (Argo, Airflow, AWS Batch, or AWS Lambda),\nconfiguration details will change, but the API remains the same:\n``soopervisor add``, then ``soopervisor export``.\n\n\nAbout Ploomber\n==============\n\nPloomber is a big community of data enthusiasts pushing the boundaries of Data Science and Machine Learning tooling.\n\nWhatever your skillset is, you can contribute to our mission. So whether you're a beginner or an experienced professional, you're welcome to join us on this journey!\n\n`Click here to know how you can contribute to Ploomber. \u003chttps://github.com/ploomber/contributing/blob/main/README.md\u003e`_\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fploomber%2Fsoopervisor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fploomber%2Fsoopervisor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fploomber%2Fsoopervisor/lists"}