Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/flyteorg/flyte
Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://github.com/flyteorg/flyte
data data-analysis data-science dataops declarative fine-tuning flyte golang grpc kubernetes kubernetes-operator llm machine-learning mlops orchestration-engine production production-grade python scale workflow
Last synced: about 2 months ago
JSON representation
Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
- Host: GitHub
- URL: https://github.com/flyteorg/flyte
- Owner: flyteorg
- License: apache-2.0
- Created: 2019-10-21T17:40:04.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2024-05-22T21:31:52.000Z (4 months ago)
- Last Synced: 2024-05-22T23:55:33.198Z (4 months ago)
- Topics: data, data-analysis, data-science, dataops, declarative, fine-tuning, flyte, golang, grpc, kubernetes, kubernetes-operator, llm, machine-learning, mlops, orchestration-engine, production, production-grade, python, scale, workflow
- Language: Go
- Homepage: https://flyte.org
- Size: 296 MB
- Stars: 4,869
- Watchers: 259
- Forks: 525
- Open Issues: 1,005
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG/CHANGELOG-v0.10.0.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: CODEOWNERS
- Security: SECURITY.md
Awesome Lists containing this project
- awesomeLibrary - flyte - Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. (语言资源库 / go)
- awesome-ccamel - flyteorg/flyte - Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. (Go)
- awesome - flyteorg/flyte - Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. (Go)
- awesome-llmops - Flyte - native workflow automation platform for complex, mission-critical data and ML processes at scale. |  | (Large Scale Deployment / Workflow)
- awesome-repositories - flyteorg/flyte - Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. (Go)
- awesome-production-machine-learning - Flyte - Lyft’s Cloud Native Machine Learning and Data Processing Platform - [(Demo)](https://youtu.be/KdUJGSP1h9U?t=1451). (Data Pipeline)
- StarryDivineSky - flyteorg/flyte
- awesome-starred - flyteorg/flyte - Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. (python)
README
Flyte
:building_construction: :rocket: :chart_with_upwards_trend:
Flyte is an open-source orchestrator that facilitates building production-grade data and ML pipelines. It is built for scalability and reproducibility, leveraging Kubernetes as its underlying platform. With Flyte, user teams can construct pipelines using the Python SDK, and seamlessly deploy them on both cloud and on-premises environments, enabling distributed processing and efficient resource utilization.
Build
Write code in Python or any other language and leverage a robust type engine.
Deploy & Scale
Either locally or on a remote cluster, execute your models with ease.
Get Started
·
Documentation
·
Resources
## Table of contents
* [Quick start](#quick-start)
* [Tutorials](#tutorials)
* [Features](#features)
* [Who uses Flyte](#whos-using-flyte)
* [How to stay involved](#how-to-stay-involved)
* [How to contribute](#how-to-contribute)
---
## Quick start
1. Install Flyte's Python SDK
```bash
pip install flytekit
```
2. Create a workflow (see [example](https://github.com/flyteorg/flytesnacks/blob/master/examples/basics/basics/hello_world.py))
3. Run it locally with:
```bash
pyflyte run hello_world.py hello_world_wf
```
**Ready to try a Flyte cluster?**
1. Create a new sandbox cluster, running as a Docker container:
```bash
flytectl demo start
```
2. Now execute your workflows on the cluster:
```bash
pyflyte run --remote hello_world.py hello_world_wf
```
**Do you want to see more but don't want to install anything?**
Head over to https://sandbox.union.ai/. It allows you to experiment with Flyte's capabilities from a hosted Jupyter notebook.
**Ready to productionize?**
Go to the [Deployment guide](https://docs.flyte.org/en/latest/deployment/deployment/index.html) for instructions to install Flyte on different environments
## Tutorials
- [Fine-tune Code Llama on the Flyte codebase](https://github.com/unionai-oss/llm-fine-tuning/tree/main/flyte_llama#readme)
- [Forecast sales with Horovod and Spark](https://docs.flyte.org/en/latest/flytesnacks/examples/forecasting_sales/index.html)
- [Nucleotide Sequence Querying with BLASTX](https://docs.flyte.org/en/latest/flytesnacks/examples/blast/index.html)
## Features
🚀 **Strongly typed interfaces**: Validate your data at every step of the workflow by defining data guardrails using Flyte types.
🌐 **Any language**: Write code in any language using raw containers, or choose [Python](https://github.com/flyteorg/flytekit), [Java](https://github.com/flyteorg/flytekit-java), [Scala](https://github.com/flyteorg/flytekit-java) or [JavaScript](https://github.com/NotMatthewGriffin/pterodactyl) SDKs to develop your Flyte workflows.
🔒 **Immutability**: Immutable executions help ensure reproducibility by preventing any changes to the state of an execution.
🧬 **Data lineage**: Track the movement and transformation of data throughout the lifecycle of your data and ML workflows.
📊 **Map tasks**: Achieve parallel code execution with minimal configuration using [map tasks](https://docs.flyte.org/en/latest/user_guide/advanced_composition/map_tasks.html).
🌎 **Multi-tenancy**: Multiple users can share the same platform while maintaining their own distinct data and configurations.
🌟 **Dynamic workflows**: [Build flexible and adaptable workflows](https://docs.flyte.org/en/latest/user_guide/advanced_composition/dynamic_workflows.html) that can change and evolve as needed, making it easier to respond to changing requirements.
⏯️ [Wait](https://docs.flyte.org/en/latest/user_guide/advanced_composition/waiting_for_external_inputs.html) for **external inputs** before proceeding with the execution.
🌳 **Branching**: [Selectively execute branches](https://docs.flyte.org/en/latest/user_guide/advanced_composition/conditionals.html) of your workflow based on static or dynamic data produced by other tasks or input data.
📈 **Data visualization**: Visualize data, monitor models and view training history through plots.
📂 **FlyteFile & FlyteDirectory**: Transfer [files](https://docs.flyte.org/en/latest/user_guide/data_types_and_io/flytefile.html) and [directories](https://docs.flyte.org/en/latest/user_guide/data_types_and_io/flytedirectory.html) between local and cloud storage.
🗃️ **Structured dataset**: Convert dataframes between types and enforce column-level type checking using the abstract 2D representation provided by [Structured Dataset](https://docs.flyte.org/en/latest/user_guide/data_types_and_io/structureddataset.html).
🛡️ **Recover from failures**: Recover only the failed tasks.
🔁 **Rerun a single task**: Rerun workflows at the most granular level without modifying the previous state of a data/ML workflow.
🔍 **Cache outputs**: Cache task outputs by passing `cache=True` to the task decorator.
🚩 **Intra-task checkpointing**: [Checkpoint progress](https://docs.flyte.org/en/latest/user_guide/advanced_composition/intratask_checkpoints.html) within a task execution.
⏰ **Timeout**: Define a timeout period, after which the task is marked as failure.
🏭 **Dev to prod**: As simple as changing your [domain](https://docs.flyte.org/en/latest/concepts/domains.html) from development or staging to production.
💸 **Spot or preemptible instances**: Schedule your workflows on spot instances by setting `interruptible` to `True` in the task decorator.
☁️ **Cloud-native deployment**: Deploy Flyte on AWS, GCP, Azure and other cloud services.
📅 **Scheduling**: [Schedule](https://docs.flyte.org/en/latest/user_guide/productionizing/schedules.html) your data and ML workflows to run at a specific time.
📢 **Notifications**: Stay informed about changes to your workflow's state by configuring [notifications](https://docs.flyte.org/en/latest/user_guide/productionizing/notifications.html) through Slack, PagerDuty or email.
⌛️ **Timeline view**: Evaluate the duration of each of your Flyte tasks and identify potential bottlenecks.
💨 **GPU acceleration**: Enable and control your tasks’ GPU demands by requesting resources in the task decorator.
🐳 **Dependency isolation via containers**: Maintain separate sets of dependencies for your tasks so no dependency conflicts arise.
🔀 **Parallelism**: Flyte tasks are inherently parallel to optimize resource consumption and improve performance.
💾 **Allocate resources dynamically** at the task level.
## Who's using Flyte
Join the likes of LinkedIn, Spotify, Freenome, Pachama, Warner Bros. and many others in adopting Flyte for mission-critical use cases. For a full list of adopters and information on how to add your organization or project, please visit our [ADOPTERS](https://github.com/flyteorg/community/blob/main/ADOPTERS.md) page.
## How to stay involved
👥 [Monthly community sync](https://www.addevent.com/event/EA7823958): Happening the first Tuesday of every month, this is where the Flyte team provides updates on the project, and community members can share their progress and ask questions.
💬 [Slack](https://slack.flyte.org/): Join the Flyte community on Slack to chat with other users, ask questions, and get help.
⚠️ [Newsletter](https://lists.lfaidata.foundation/g/flyte-announce/join): join this group to receive the Flyte Monthly newsletter.
📹 [Youtube](https://www.youtube.com/channel/UCNduEoLOToNo3nFVly-vUTQ): Tune into panel discussions, customer success stories, community updates and feature deep dives.
📄 [Blog](https://flyte.org/blog): Here, you can find tutorials and feature deep dives to help you learn more about Flyte.
💡 [RFCs](rfc/.): RFCs are used for proposing new ideas and features to improve Flyte. You can refer to them to stay updated on the latest developments and contribute to the growth of the platform.
## How to contribute
There are many ways to get involved in Flyte, including:
- Submitting [bugs](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&template=bug_report.yaml&title=%5BBUG%5D+) and [feature requests](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=enhancement%2Cuntriaged&template=feature_request.yaml&title=%5BCore+feature%5D+) for various components.
- Reviewing [the documentation](https://docs.flyte.org/en/latest/) and submitting [pull requests](https://github.com/flyteorg/flytesnacks) for anything from fixing typos to adding new content.
- Speaking or writing about Flyte or any other ecosystem integration and [letting us know](https://flyte-org.slack.com/archives/C02JMT8KTEE)!
- Taking on a [`help wanted`](https://github.com/flyteorg/flyte/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22+) or [`good-first-issue`](https://github.com/flyteorg/flyte/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) and following the [CONTRIBUTING](https://docs.flyte.org/en/latest/community/contribute.html) guide to submit changes to the codebase.
- Upvoting [popular feature requests](https://github.com/flyteorg/flyte/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement+sort%3Areactions-%2B1-desc) to show your support.
### We :heart: our contributors
[](https://github.com/katrogan)[](https://github.com/lyft-metaservice-3)[](https://github.com/matthewphsmith)[](https://github.com/EngHabu)[](https://github.com/goreleaserbot)[](https://github.com/evalsocket)[](https://github.com/hamersaw)[](https://github.com/flyte-bot)[](https://github.com/honnix)[](https://github.com/anandswaminathan)[](https://github.com/wild-endeavor)[](https://github.com/bnsblue)[](https://github.com/pingsutw)[](https://github.com/iaroslav-ciupin)[](https://github.com/kumare3)[](https://github.com/samhita-alla)[](https://github.com/MorpheusXAUT)[](https://github.com/aliabbasjaffri)[](https://github.com/ckiosidis)[](https://github.com/mayitbeegh)[](https://github.com/alexlipa91)[](https://github.com/brucearctor)[](https://github.com/pmahindrakar-oss)[](https://github.com/Antaxify)[](https://github.com/eapolinario)[](https://github.com/andrewwdye)[](https://github.com/ariefrahmansyah)[](https://github.com/jeevb)[](https://github.com/jonathanburns)[](https://github.com/lu4nm3)[](https://github.com/lyft-metaservice-2)[](https://github.com/2uasimojo)[](https://github.com/veggiemonk)[](https://github.com/schottra)[](https://github.com/supreeth7)[](https://github.com/cosmicBboy)[](https://github.com/migueltol22)[](https://github.com/milton0825)[](https://github.com/slai)[](https://github.com/SmritiSatyanV)[](https://github.com/surindersinghp)[](https://github.com/TheYk98)[](https://github.com/kosigz-lyft)[](https://github.com/chanadian)[](https://github.com/kanterov)[](https://github.com/hanzo)[](https://github.com/igorvalko)[](https://github.com/kennyworkman)[](https://github.com/maximsmol)[](https://github.com/vsbus)[](https://github.com/akhurana001)[](https://github.com/bstadlbauer)[](https://github.com/jerempy)[](https://github.com/tnsetting)[](https://github.com/catalinii)[](https://github.com/ByronHsu)[](https://github.com/chetcode)[](https://github.com/regadas)[](https://github.com/fg91)[](https://github.com/aybidi)[](https://github.com/akashkatipally)[](https://github.com/goyalankit)[](https://github.com/clairemcginty)[](https://github.com/flixr)[](https://github.com/akumor)[](https://github.com/niliayu)[](https://github.com/snyk-bot)[](https://github.com/derwiki)[](https://github.com/th0114nd)[](https://github.com/AlekhyaSasi)[](https://github.com/apps/dependabot)[](https://github.com/asottile)[](https://github.com/SandraGH5)[](https://github.com/sbrunk)[](https://github.com/sonjaer)[](https://github.com/fediazgon)[](https://github.com/rahul-theorem)[](https://github.com/ryankarlos)[](https://github.com/ddhirajkumar)[](https://github.com/max-hoffman)[](https://github.com/AdrianoKF)[](https://github.com/dennisobrien)[](https://github.com/Ln11211)[](https://github.com/aeioulisa)[](https://github.com/michaels-lyft)[](https://github.com/murilommen)[](https://github.com/RobertoRRW)[](https://github.com/bimtauer)[](https://github.com/esadler-hbo)[](https://github.com/ggydush-fn)[](https://github.com/kiliangojek)[](https://github.com/bethebunny)[](https://github.com/aalavian)[](https://github.com/convexquad)[](https://github.com/andresgomezfrr)[](https://github.com/asahalyft)[](https://github.com/apatel-fn)[](https://github.com/arpitbhardwaj)[](https://github.com/lordnodd)[](https://github.com/bryanwweber)[](https://github.com/CalvinLeather)[](https://github.com/YmirKhang)[](https://github.com/franco-bocci)[](https://github.com/frsann)[](https://github.com/hfurkanvural)[](https://github.com/jbrambleDC)[](https://github.com/jcugat)[](https://github.com/madhur-tandon)[](https://github.com/matheusMoreno)[](https://github.com/NotMatthewGriffin)[](https://github.com/myz540)[](https://github.com/tekumara)[](https://github.com/ppiegaze)[](https://github.com/Qiwen-Yu)[](https://github.com/RobinKa)[](https://github.com/rubenbarragan)[](https://github.com/sugatoray)[](https://github.com/sushrut111)[](https://github.com/Tat-V)[](https://github.com/TeoZosa)[](https://github.com/ThomVett)[](https://github.com/datability-io)[](https://github.com/techytushar)[](https://github.com/vchowdhary)[](https://github.com/varshaparthay)[](https://github.com/vvasavada-fn)[](https://github.com/ybubnov)[](https://github.com/Yicheng-Lu-llll)[](https://github.com/palchicz)[](https://github.com/ajsalow)[](https://github.com/ggydush)[](https://github.com/martinlyra)[](https://github.com/mcloney-ddm)[](https://github.com/pbrogan12)[](https://github.com/stef-stripe)[](https://github.com/charlie0220)[](https://github.com/stephen37)[](https://github.com/anrusina)[](https://github.com/service-github-lyft-semantic-release)[](https://github.com/ursucarina)[](https://github.com/jsonporter)[](https://github.com/csirius)[](https://github.com/olga-union)[](https://github.com/Pianist038801)[](https://github.com/james-union)[](https://github.com/eugenejahn)[](https://github.com/4nalog)[](https://github.com/apTalya)[](https://github.com/aviaviavi)[](https://github.com/Professional0321)[](https://github.com/HiromuHota)[](https://github.com/rafaelraposospot)[](https://github.com/daniel-shuy)[](https://github.com/live-wire)[](https://github.com/sisco0)[](https://github.com/jimbobby5)[](https://github.com/pradithya)[](https://github.com/skiptomyliu)[](https://github.com/haoyuez)[](https://github.com/lupasarin)[](https://github.com/Dread1982)[](https://github.com/narape)[](https://github.com/alexapdev)[](https://github.com/3t8)[](https://github.com/zeryx)[](https://github.com/arturdryomov)[](https://github.com/ChickenTarm)[](https://github.com/DavidMertz)[](https://github.com/felixwang9817)[](https://github.com/juandiegopalomino)[](https://github.com/kanyesthaker)[](https://github.com/marc-union)[](https://github.com/michaeltinsley)[](https://github.com/mucahitkantepe)[](https://github.com/oyevtushok)[](https://github.com/trishitapingolia)[](https://github.com/Smartmind12)[](https://github.com/huxuan)[](https://github.com/privatedumbo)[](https://github.com/tjKairos)[](https://github.com/georgesnelling)[](https://github.com/dschaller)[](https://github.com/davidmirror-ops)[](https://github.com/davidxia)[](https://github.com/hoyajigi)[](https://github.com/MrKrishnaAgarwal)[](https://github.com/NitinAgg)[](https://github.com/noobkid2411)[](https://github.com/yongchand)[](https://github.com/nicklofaso)[](https://github.com/mounesi)[](https://github.com/eanakhl)[](https://github.com/adinin)[](https://github.com/anton-malakhov)[](https://github.com/avan-sh)[](https://github.com/kinow)[](https://github.com/Daeruin)[](https://github.com/dav009)[](https://github.com/idivyanshbansal)[](https://github.com/fvde)[](https://github.com/Lundez)[](https://github.com/hasukmistry)[](https://github.com/rokrokss)[](https://github.com/jeremydonahue)[](https://github.com/jonasdebeukelaer)[](https://github.com/jmcarp)[](https://github.com/kazesberger)[](https://github.com/lsena)[](https://github.com/mishmanners)[](https://github.com/paravatha)[](https://github.com/uschi2000)[](https://github.com/rodrigobaron)[](https://github.com/ronaldosaheki)[](https://github.com/shahwar9)[](https://github.com/shihgianlee)[](https://github.com/SKalt)[](https://github.com/samuel-sujith)[](https://github.com/ilikedata)[](https://github.com/orf)[](https://github.com/vijaysaravana)[](https://github.com/yubofredwang)[](https://github.com/fsz285)[](https://github.com/gigi-at-zymergen)[](https://github.com/hampusrosvall)[](https://github.com/hitarth01)[](https://github.com/jcourteau)[](https://github.com/jw0515)[](https://github.com/leorleor)[](https://github.com/moose007)[](https://github.com/sanjaychouhan-adf)[](https://github.com/v01dXYZ)[](https://github.com/wanderer163)[](https://github.com/kylewaynebenson)[](https://github.com/Gui11aum3)[](https://github.com/JakeNeyer)[](https://github.com/abhijeet007rocks8)[](https://github.com/mouuff)[](https://github.com/Juneezee)[](https://github.com/goodgravy)[](https://github.com/radiantly)[](https://github.com/nishantwrp)[](https://github.com/sighingnow)[](https://github.com/vglocus)[](https://github.com/RustedBones)[](https://github.com/pablocasares)[](https://github.com/evdokim)[](https://github.com/stormy-ua)[](https://github.com/marschall)[](https://github.com/gdungca-fn)[](https://github.com/ttanay)[](https://github.com/Abdullahi-Ahmed)[](https://github.com/amaleelhamri)[](https://github.com/pradyunsg)[](https://github.com/apps/pre-commit-ci)[](https://github.com/jdknight)[](https://github.com/kmike)[](https://github.com/hugovk)[](https://github.com/sirosen)[](https://github.com/humitos)[](https://github.com/bastimeyer)[](https://github.com/asmeurer)[](https://github.com/drewyh)[](https://github.com/polyzen)[](https://github.com/dvarrazzo)[](https://github.com/dbitouze)[](https://github.com/idryzhov)[](https://github.com/pauloxnet)[](https://github.com/ichard26)[](https://github.com/sethmlarson)[](https://github.com/graingert)[](https://github.com/stonecharioteer)[](https://github.com/yeraydiazdiaz)[](https://github.com/eviau-sat)[](https://github.com/rozsasarpi)[](https://github.com/estan)[](https://github.com/pseudomuto)[](https://github.com/htdvisser)[](https://github.com/jacobtolar)[](https://github.com/ezimanyi)[](https://github.com/lpabon)[](https://github.com/ArcEye)[](https://github.com/mingrammer)[](https://github.com/aschrijver)[](https://github.com/panzerfahrer)[](https://github.com/glasser)[](https://github.com/murph0)[](https://github.com/zetaron)[](https://github.com/sunfmin)[](https://github.com/guozheng)[](https://github.com/suusan2go)[](https://github.com/mhaberler)[](https://github.com/s4ichi)[](https://github.com/dreampuf)[](https://github.com/UnicodingUnicorn)[](https://github.com/philiptzou)[](https://github.com/timabell)[](https://github.com/jasonhancock)[](https://github.com/matryer)[](https://github.com/piotrrojek)[](https://github.com/jasonsattler)[](https://github.com/sbward)[](https://github.com/Pisush)[](https://github.com/tamalsaha)[](https://github.com/marianina8)[](https://github.com/ernesto-jimenez)[](https://github.com/jsteenb2)[](https://github.com/darwayne)[](https://github.com/naysayer)[](https://github.com/AgrimPrasad)[](https://github.com/dahernan)[](https://github.com/jtarchie)[](https://github.com/jdtobe)[](https://github.com/alrs)[](https://github.com/urisimchoni)[](https://github.com/Xercoy)[](https://github.com/marbergq)[](https://github.com/anothrNick)[](https://github.com/fermoya)[](https://github.com/sbe-arg)[](https://github.com/PeerXu)[](https://github.com/reececomo)[](https://github.com/dmerrick)[](https://github.com/andrewcole)[](https://github.com/phish108)[](https://github.com/endrjuskr)[](https://github.com/bevans-HD)[](https://github.com/gukoff)[](https://github.com/lovromazgon)[](https://github.com/117)[](https://github.com/tomsolem)[](https://github.com/vq-ambiata)[](https://github.com/sjauld)[](https://github.com/adam-berrio)[](https://github.com/zsedem)[](https://github.com/imdanielsp)[](https://github.com/fabricepipart)[](https://github.com/ivanpk)[](https://github.com/JeremyLWright)[](https://github.com/OskarStark)[](https://github.com/pavyarov)[](https://github.com/pellared)[](https://github.com/cuttingedge1109)[](https://github.com/okozachenko1203)[](https://github.com/zero-below)[](https://github.com/asford)[](https://github.com/bmcconeghy)[](https://github.com/conda-forge-admin)[](https://github.com/regro-cf-autotick-bot)[](https://github.com/apps/conda-forge-curator)[](https://github.com/apps/github-actions)[](https://github.com/conda-forge-linter)[](https://github.com/fellhorn)
## License
Flyte is available under the Apache License 2.0. Use it wisely.
