Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/orchest/orchest

Build data pipelines, the easy way 🛠️
https://github.com/orchest/orchest

airflow cloud dag data-pipelines data-science deployment docker etl etl-pipeline ide jupyter jupyterlab kubernetes machine-learning notebooks orchest pipelines python self-hosted

Last synced: about 1 month ago
JSON representation

Build data pipelines, the easy way 🛠️

Lists

README 

        




  













Notice: we’re no longer actively developing Orchest. We could not find a way to make building a workflow orchestrator commercially viable. Check out Apache Airflow for a robust workflow solution.



## Build data pipelines, the easy way 🙌



No frameworks. No YAML. Just write your data processing code directly in **Python**, **R** or

**Julia**.





  






  💡 Watch the full narrated video to learn more about building data pipelines in Orchest.

 



> **Note**: Orchest is in **beta**.



## Features



- **Visually construct pipelines** through our user-friendly UI

- **Code in Notebooks** and scripts

  ([quickstart](https://docs.orchest.io/en/stable/getting_started/quickstart.html))

- Run any subset of a pipelines directly or periodically

  ([jobs](https://docs.orchest.io/en/stable/fundamentals/jobs.html))

- Easily define your dependencies to run on **any machine**

  ([environments](https://docs.orchest.io/en/stable/fundamentals/environments.html))

- Spin up services whose lifetime spans across the entire pipeline run

  ([services](https://docs.orchest.io/en/stable/fundamentals/services.html))

- Version your projects using git

  ([projects](https://docs.orchest.io/en/stable/fundamentals/projects.html))



**When to use Orchest?** Read it in the

[docs](https://docs.orchest.io/en/stable/getting_started/when_to_use_orchest.html).



👉 Get started with our

[quickstart](https://docs.orchest.io/en/stable/getting_started/quickstart.html) tutorial or have a look at our [video tutorials](https://www.orchest.io/video-tutorials) explaining some of Orchest's core concepts.



## Roadmap



Missing a feature? Have a look at [our public roadmap](https://github.com/orgs/orchest/projects/1)

to see what the team is working on in the short and medium term.

Still missing it? Please [let us know by opening an issue](https://github.com/orchest/orchest/issues/new/choose)!



## Examples



Get started with an example project:



- [Train and compare 3 regression models](https://github.com/orchest/quickstart)

- [Connecting to an external database using SQLAlchemy](https://github.com/astrojuanlu/orchest-sqlalchemy)

- [Run dbt in Orchest for a dbt + Python transform pipeline](https://github.com/ricklamers/orchest-dbt)

- [Use PySpark in Orchest](https://github.com/ricklamers/orchest-hello-spark)



👉 Check out the full list of [example projects](https://github.com/orchest/orchest-examples).



[![Open in Orchest](https://github.com/orchest/orchest-examples/raw/main/imgs/open_in_orchest_large.svg)](https://cloud.orchest.io/)



## Installation



Want to skip [the installation](https://docs.orchest.io/en/stable/getting_started/installation.html)

and jump right in? Then try out our managed service: [Orchest Cloud](https://cloud.orchest.io/signup).



## Slack Community



Join our Slack to chat about Orchest, ask questions, and share tips.



[![Join us on Slack](https://img.shields.io/badge/%20-Join%20us%20on%20Slack-blue?style=for-the-badge&logo=slack&labelColor=5c5c5c)](https://join.slack.com/t/orchest/shared_invite/zt-g6wooj3r-6XI8TCWJrXvUnXKdIKU_8w)



## License



The software in this repository is licensed as follows:



- All content residing under the ` orchest-sdk/` and `orchest-cli/` directories of this repository

  are licensed under the `Apache-2.0` license as defined in `orchest-sdk/LICENSE` and

  `orchest-cli/LICENSE` respectively.

- Content outside of the above mentioned directories is available under the `AGPL-3.0` license.



## Contributing



Contributions are more than welcome! Please see our [contributor

guides](https://docs.orchest.io/en/stable/development/contributing.html) for more details.



Alternatively, you can submit your pipeline to the curated list of [Orchest

examples](https://github.com/orchest/orchest-examples) that are automatically loaded in every

Orchest deployment! 🔥



## Contributors