Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ploomber/ploomber
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
https://github.com/ploomber/ploomber
data-engineering data-science jupyter jupyter-notebooks machine-learning mlops notebooks papermill pipelines pycharm vscode workflow
Last synced: about 9 hours ago
JSON representation
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
- Host: GitHub
- URL: https://github.com/ploomber/ploomber
- Owner: ploomber
- License: apache-2.0
- Created: 2020-01-20T20:13:06.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2024-09-18T19:11:09.000Z (5 months ago)
- Last Synced: 2025-02-11T13:05:54.774Z (7 days ago)
- Topics: data-engineering, data-science, jupyter, jupyter-notebooks, machine-learning, mlops, notebooks, papermill, pipelines, pycharm, vscode, workflow
- Language: Python
- Homepage: https://docs.ploomber.io
- Size: 6.73 MB
- Stars: 3,541
- Watchers: 30
- Forks: 238
- Open Issues: 107
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
- best-of-python - GitHub - 11% open · ⏱️ 20.02.2024): (Data Pipelines & Streaming)
- awesome-github-repos - ploomber/ploomber - The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️ (Python)
- awesome-llmops - Ploomber - square) | (Large Scale Deployment / Workflow)
- awesome-starred - ploomber/ploomber - The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️ (data-science)
- awesome-cv - Ploomber
- awesome-list - Ploomber - The fastest way to build data pipelines. (Data Management & Processing / Database & Cloud Management)
- awesome-argo - Ploomber - ready pipelines using JupyterLab or any text editor. (Ecosystem Projects)
- awesome-production-machine-learning - Ploomber - The fastest way to build data pipelines. Develop iteratively, deploy anywhere. (Data Pipeline)
- awesome-python-machine-learning-resources - GitHub - 25% open · ⏱️ 26.08.2022): (数据管道和流处理)
- StarryDivineSky - ploomber/ploomber
- jimsghstars - ploomber/ploomber - The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️ (Python)
README
![]()
[](https://github.com/ploomber/ploomber/actions/workflows/ci-unit-linux.yml/badge.svg)
[](https://github.com/ploomber/ploomber/actions/workflows/ci-unit-macos.yml/badge.svg)
[](https://docs.ploomber.io/en/latest/?badge=latest)
[](https://badge.fury.io/py/ploomber)
[](https://anaconda.org/conda-forge/ploomber)
[](https://anaconda.org/conda-forge/ploomber)
[](https://coveralls.io/github/ploomber/ploomber?branch=master)
[](https://twitter.com/intent/user?screen_name=ploomber)
[](https://pepy.tech/project/ploomber)> [!TIP]
> Deploy AI apps for free on [Ploomber Cloud!](https://ploomber.io/?utm_medium=github&utm_source=ploomber)
Join our community
|
Newsletter
|
Contact us
|
Docs
|
Blog
|
Website
|
YouTubePloomber is the fastest way to build data pipelines ⚡️. Use your favorite editor (**[Jupyter](https://docs.ploomber.io/en/latest/user-guide/jupyter.html), [VSCode](https://docs.ploomber.io/en/latest/user-guide/editors.html), [PyCharm](https://docs.ploomber.io/en/latest/user-guide/editors.html)**) to develop interactively and deploy ☁️ without code changes (**[Kubernetes](https://soopervisor.readthedocs.io/en/latest/tutorials/kubernetes.html), [Airflow](https://soopervisor.readthedocs.io/en/latest/tutorials/airflow.html), [AWS Batch](https://soopervisor.readthedocs.io/en/latest/tutorials/aws-batch.html), and [SLURM](https://soopervisor.readthedocs.io/en/latest/tutorials/slurm.html)**). Do you have legacy notebooks? Refactor them into modular pipelines with a single command.
## Installation
*Compatible with Python 3.7 and higher.*
Install with `pip`:
```sh
pip install ploomber
```Or with `conda`:
```sh
conda install ploomber -c conda-forge
```## Getting started
### Try the tutorial:
## Community
* [Join us on Slack](https://ploomber.io/community)
* [Newsletter](https://share.hsforms.com/1E7Qa_OpcRPi_MV-segFsaAe6c2g)
* [YouTube](https://www.youtube.com/channel/UCaIS5BMlmeNQE4-Gn0xTDXQ)
* [Contact the development team](mailto:[email protected])## Main Features
### ⚡️ Get started quickly
A simple YAML API to get started quickly, a powerful Python API for total flexibility.
https://user-images.githubusercontent.com/989250/150660813-fc289c6c-0ed5-432d-b6df-063ce98c0093.mp4
### ⏱ Shorter development cycles
Automatically cache your pipeline’s previous results and only re-compute tasks that have changed since your last execution.
https://user-images.githubusercontent.com/989250/150660820-9a3a0abd-5904-492b-97ff-5494285dfebf.mp4
### ☁️ Deploy anywhere
Run as a shell script in a single machine or distributively in [Kubernetes](https://soopervisor.readthedocs.io/en/latest/tutorials/kubernetes.html), [Airflow](https://soopervisor.readthedocs.io/en/latest/tutorials/airflow.html), [AWS Batch](https://soopervisor.readthedocs.io/en/latest/tutorials/aws-batch.html), or [SLURM](https://soopervisor.readthedocs.io/en/latest/tutorials/slurm.html).
https://user-images.githubusercontent.com/989250/150660830-3f81c9a2-5392-49e5-976d-cb8a38441ecb.mp4
### 📙 Automated migration from legacy notebooks
Bring your old monolithic notebooks, and we’ll automatically convert them into maintainable, modular pipelines.
https://user-images.githubusercontent.com/989250/150660840-b0c12f85-504c-4233-8c3d-6724d291f1aa.mp4
[I want to migrate my notebook.](https://docs.ploomber.io/en/latest/user-guide/refactoring.html)
[Show me a demo.](https://www.youtube.com/watch?v=EJecqsZBr3Q)
## Resources
* [Documentation](https://docs.ploomber.io/)
* [PyData Chicago talk (covers motivation and demo)](https://youtu.be/qUL7QabcKcw)
* [Develop and deploy an ML pipeline in 30 minutes (EuroPython 2021)](https://youtu.be/O8tqiCkIWPs)
* [Guest blog post on the official Jupyter blog](https://blog.jupyter.org/ploomber-maintainable-and-collaborative-pipelines-in-jupyter-acb3ad2101a7)
* [Examples (Machine Learning pipeline, ETL, among others)](https://github.com/ploomber/projects)
* [Blog](https://ploomber.io/)
* [Comparison with other tools](https://ploomber.io/posts/survey)
* [More videos](https://docs.ploomber.io/en/latest/videos.html)## About Ploomber
Ploomber is a big community of data enthusiasts pushing the boundaries of Data Science and Machine Learning tooling.
Whatever your skillset is, you can contribute to our mission. So whether you're a beginner or an experienced professional, you're welcome to join us on this journey!
[Click here to know how you can contribute to Ploomber.](https://github.com/ploomber/contributing/blob/main/README.md)