Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/orchest/orchest
Build data pipelines, the easy way π οΈ
https://github.com/orchest/orchest
airflow cloud dag data-pipelines data-science deployment docker etl etl-pipeline ide jupyter jupyterlab kubernetes machine-learning notebooks orchest pipelines python self-hosted
Last synced: 26 days ago
JSON representation
Build data pipelines, the easy way π οΈ
- Host: GitHub
- URL: https://github.com/orchest/orchest
- Owner: orchest
- License: apache-2.0
- Created: 2020-05-21T17:33:05.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2023-06-06T09:48:26.000Z (over 1 year ago)
- Last Synced: 2024-04-14T12:10:21.818Z (7 months ago)
- Topics: airflow, cloud, dag, data-pipelines, data-science, deployment, docker, etl, etl-pipeline, ide, jupyter, jupyterlab, kubernetes, machine-learning, notebooks, orchest, pipelines, python, self-hosted
- Language: TypeScript
- Homepage: https://orchest.readthedocs.io/en/stable/
- Size: 27.2 MB
- Stars: 4,019
- Watchers: 43
- Forks: 251
- Open Issues: 128
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
Awesome Lists containing this project
- awesome-ccamel - orchest/orchest - Build data pipelines, the easy way π οΈ (TypeScript)
- awesome-starred - orchest/orchest - Build data pipelines, the easy way π οΈ (machine-learning)
- awesome-argo - Orchest
README
Notice: weβre no longer actively developing Orchest. We could not find a way to make building a workflow orchestrator commercially viable. Check out Apache Airflow for a robust workflow solution.
## Build data pipelines, the easy way π
No frameworks. No YAML. Just write your data processing code directly in **Python**, **R** or
**Julia**.
π‘ Watch the full narrated video to learn more about building data pipelines in Orchest.
> **Note**: Orchest is in **beta**.
## Features
- **Visually construct pipelines** through our user-friendly UI
- **Code in Notebooks** and scripts
([quickstart](https://docs.orchest.io/en/stable/getting_started/quickstart.html))
- Run any subset of a pipelines directly or periodically
([jobs](https://docs.orchest.io/en/stable/fundamentals/jobs.html))
- Easily define your dependencies to run on **any machine**
([environments](https://docs.orchest.io/en/stable/fundamentals/environments.html))
- Spin up services whose lifetime spans across the entire pipeline run
([services](https://docs.orchest.io/en/stable/fundamentals/services.html))
- Version your projects using git
([projects](https://docs.orchest.io/en/stable/fundamentals/projects.html))**When to use Orchest?** Read it in the
[docs](https://docs.orchest.io/en/stable/getting_started/when_to_use_orchest.html).π Get started with our
[quickstart](https://docs.orchest.io/en/stable/getting_started/quickstart.html) tutorial or have a look at our [video tutorials](https://www.orchest.io/video-tutorials) explaining some of Orchest's core concepts.## Roadmap
Missing a feature? Have a look at [our public roadmap](https://github.com/orgs/orchest/projects/1)
to see what the team is working on in the short and medium term.
Still missing it? Please [let us know by opening an issue](https://github.com/orchest/orchest/issues/new/choose)!## Examples
Get started with an example project:
- [Train and compare 3 regression models](https://github.com/orchest/quickstart)
- [Connecting to an external database using SQLAlchemy](https://github.com/astrojuanlu/orchest-sqlalchemy)
- [Run dbt in Orchest for a dbt + Python transform pipeline](https://github.com/ricklamers/orchest-dbt)
- [Use PySpark in Orchest](https://github.com/ricklamers/orchest-hello-spark)π Check out the full list of [example projects](https://github.com/orchest/orchest-examples).
[![Open in Orchest](https://github.com/orchest/orchest-examples/raw/main/imgs/open_in_orchest_large.svg)](https://cloud.orchest.io/)
## Installation
Want to skip [the installation](https://docs.orchest.io/en/stable/getting_started/installation.html)
and jump right in? Then try out our managed service: [Orchest Cloud](https://cloud.orchest.io/signup).## Slack Community
Join our Slack to chat about Orchest, ask questions, and share tips.
[![Join us on Slack](https://img.shields.io/badge/%20-Join%20us%20on%20Slack-blue?style=for-the-badge&logo=slack&labelColor=5c5c5c)](https://join.slack.com/t/orchest/shared_invite/zt-g6wooj3r-6XI8TCWJrXvUnXKdIKU_8w)
## License
The software in this repository is licensed as follows:
- All content residing under the ` orchest-sdk/` and `orchest-cli/` directories of this repository
are licensed under the `Apache-2.0` license as defined in `orchest-sdk/LICENSE` and
`orchest-cli/LICENSE` respectively.
- Content outside of the above mentioned directories is available under the `AGPL-3.0` license.## Contributing
Contributions are more than welcome! Please see our [contributor
guides](https://docs.orchest.io/en/stable/development/contributing.html) for more details.Alternatively, you can submit your pipeline to the curated list of [Orchest
examples](https://github.com/orchest/orchest-examples) that are automatically loaded in every
Orchest deployment! π₯## Contributors