{"id":19534466,"url":"https://github.com/flyteorg/flytelab","last_synced_at":"2025-09-03T13:06:22.172Z","repository":{"id":42049097,"uuid":"344845328","full_name":"flyteorg/flytelab","owner":"flyteorg","description":"Machine Learning Projects with Flytekit","archived":false,"fork":false,"pushed_at":"2023-05-23T03:55:07.000Z","size":2525,"stargazers_count":36,"open_issues_count":8,"forks_count":44,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-26T14:42:21.106Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flyteorg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-03-05T15:02:48.000Z","updated_at":"2025-03-22T15:46:58.000Z","dependencies_parsed_at":"2025-04-26T14:35:54.244Z","dependency_job_id":"18a9f5a2-07cf-4070-8ebe-9bdfa7b01623","html_url":"https://github.com/flyteorg/flytelab","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/flyteorg/flytelab","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyteorg%2Fflytelab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyteorg%2Fflytelab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyteorg%2Fflytelab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyteorg%2Fflytelab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flyteorg","download_url":"https://codeload.github.com/flyteorg/flytelab/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyteorg%2Fflytelab/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268043843,"owners_count":24186491,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-31T02:00:08.723Z","response_time":66,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T02:14:27.064Z","updated_at":"2025-07-31T13:04:03.344Z","avatar_url":"https://github.com/flyteorg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003chtml\u003e\n    \u003cp align=\"center\"\u003e \n        \u003cimg src=\"https://github.com/flyteorg/flyte/blob/master/rsts/images/flyte_circle_gradient_1_4x4.png\" alt=\"Flyte Logo\" width=\"100\"\u003e\n    \u003c/p\u003e\n    \u003ch1 align=\"center\"\u003e\n        Flytelab\n    \u003c/h1\u003e\n    \u003cp align=\"center\"\u003e\n        The Open Source Repository of Flyte-based Projects\n    \u003c/p\u003e\n    \u003cp align=\"center\"\u003e\n        \u003ca href=\"https://slack.flyte.org\"\u003e\n            \u003cimg src=\"https://img.shields.io/badge/slack-join_chat.svg?logo=slack\u0026style=social\" alt=\"Slack\" /\u003e\n        \u003c/a\u003e\n    \u003c/p\u003e\n\u003c/html\u003e\n\nThe purpose of this repository is to showcase [Flyte's](https://flyte.org/) capabilities in end-to-end\napplications that do some form of data processing or machine learning.\n\nThe source code for each project can be found in the `projects` directory, where each project has its\nown set of dependencies.\n\n## Table of Contents\n\n- [Create a New Project](#-create-a-new-project)\n- [Environment Setup](#-environment-setup)\n- [Deployment](#-deployment)\n  - [Sandbox Deployment](#sandbox-deployment)\n  - [Union.ai Playground Deployment](#unionai-playground-deployment)\n- [Streamlit App [Optional]](#-streamlit-app-optional)\n\n## 🚀 Create a New Project\n\nFork the repo on github, then clone it:\n\n```bash\ngit clone https://github.com/\u003cyour-username\u003e/flytelab\n```\n\n| **📝 Note** |\n|:---------|\n| Make sure you're using `Python \u003e 3.7`|\n\nCreate a new branch for your project:\n\n```bash\ngit checkout -b my_project  # replace this with your project name\n```\n\n| **📝 Note** |\n|:---------|\n| For [MLOps Community Engineering Labs Hackathon](https://flyte.org/hackathon/) participants: Each team will have its own branch on the main `flyteorg/flytelab` repo. If you're part of a team of more than one person, assign *one teammate* to create a project directory and push it into your team's branch. |\n\nWe use `cookiecutter` to manage project templates.\n\nInstall prerequisites:\n\n```\npip install cookiecutter\n```\n\nIn the root of the repo, create a new project:\n\n```bash\ncookiecutter templates/basic -o projects\n```\n\n| **📝 Note** |\n|:---------|\n| There are more templates in the `templates` directory depending on the requirements of your project. |\n\nAnswer the project setup questions:\n\n```\nproject_name: my_project          # replace this with your project name (can only contain alphanumeric characters and `_`)\nproject_author: foobar            # replace this with your name\ngithub_username: my_username      # replace this with your github username\nflyte_project: my_flyte_project   # [optional]\ndescription: project description  # [optional]\n```\n\n| **📝 Note** |\n|:---------|\n| For [MLOps Community Engineering Labs Hackathon](https://flyte.org/hackathon/) participants: `project_author` should be your team name, and `flyte_project` should be left as the default value. |\n\nThe project structure looks like the following:\n```bash\n.\n├── Dockerfile\n├── README.md\n├── dashboard\n│   ├── app.py  # streamlit app\n│   ├── remote.config\n│   └── sandbox.config\n├── deploy.py  # deployment script\n├── my_project\n│   ├── __init__.py\n│   └── workflows.py  # flyte workflows\n├── requirements-dev.txt\n└── requirements.txt\n```\n\n## 🌏 Environment Setup\n\nGo into the project directory, then create your project's virtual environment:\n\n```bash\ncd projects/my_project\n\n# create and activate virtual environment, name the venv whatever you want\npython -m venv ~/venvs/my_project\nsource ~/venvs/my_project/bin/activate\n\n# install requirements\npip install -r requirements.txt -r requirements-dev.txt\n```\n\nRun Flyte workflows locally:\n\n```\npython my_project/workflows.py\n```\n\nYou should see something like this in the output (you can ignore the warnings):\n```\ntrained model: LogisticRegression()\n```\n\nCongrats! You just setup your flytelab project 🌟.\n\nYou can now modify and iterate on the `workflows.py` file to create your very own Flyte\nworkflows using `flytekit`. You can refer to the\n[User Guide](https://docs.flyte.org/projects/cookbook/en/latest/index.html),\n[Tutorials](https://docs.flyte.org/projects/cookbook/en/latest/tutorials.html),\nand [Flytekit API Reference](https://docs.flyte.org/projects/flytekit/en/latest/) to\nlearn more about all of `Flyte`'s capabilities.\n\n## 🚢 Deployment\n\nSo far you've probably been running your workflows locally by invoking `python my_project/workflows.py`.\nThe first step to deploying your workflows to a Flyte cluster is to test it out on a\n[local sandbox cluster](https://docs.flyte.org/en/latest/deployment/sandbox.html).\n\nMake sure you have [docker](https://docs.docker.com/get-docker/) installed.\n\nThen install `flytectl`:\n\n\u003cdetails\u003e\n\n\u003csummary\u003e💻 OSX\u003c/summary\u003e\n\n---\n\n```bash\nbrew install flyteorg/homebrew-tap/flytectl\n```\n\n---\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\n\u003csummary\u003e💻 Other Operating Systems\u003c/summary\u003e\n\n---\n\n```bash\ncurl -sL https://ctl.flyte.org/install | sudo bash -s -- -b /usr/local/bin # You can change path from /usr/local/bin to any file system path\nexport PATH=$(pwd)/bin:$PATH # Only required if user used different path then /usr/local/bin\n```\n\n---\n\n\u003c/details\u003e\n\n### Sandbox Deployment\n\nStart the sandbox cluster from your `projects/my_project` directory:\n\n```bash\nflytectl sandbox start --source .\n```\n\n\u003cdetails\u003e\n\n\u003csummary\u003eℹ Interacting with Flyte sandbox\u003c/summary\u003e\n\n---\n\nGet the status of sandbox:\n\n```\nflytectl sandbox status\n```\n\nTeardown the sandbox:\n\n```\nflytectl sandbox teardown\n```\n\n---\n\n\u003c/details\u003e\n\n| **📝 Note** |\n|:---------|\n| If you're having trouble getting the Flyte sandbox to start, see the [troubleshooting guide](https://docs.flyte.org/en/latest/community/troubleshoot.html#troubleshooting-guide). |\n\nYou should now be able to go to `http://localhost:30081/console` on your browser to see the Flyte UI.\n\n`git commit` your changes, then deploy your project's workflows with:\n\n```bash\npython deploy.py\n```\n\n\u003cdetails\u003e\n\n\u003csummary\u003eℹ Expected output\u003c/summary\u003e\n\n---\n\nYou should see something like:\n\n```\nSuccessfully packaged 4 flyte objects into /Users/nielsbantilan/git/flytelab/projects/my_project/flyte-package.tgz\nRegistering Flyte workflows\n ---------------------------------------------------------------- --------- ------------------------------\n| NAME (4)                                                       | STATUS  | ADDITIONAL INFO              |\n ---------------------------------------------------------------- --------- ------------------------------\n| /tmp/register724861421/0_my_project.workflows.get_dataset_1.pb | Success | Successfully registered file |\n ---------------------------------------------------------------- --------- ------------------------------\n| /tmp/register724861421/1_my_project.workflows.train_model_1.pb | Success | Successfully registered file |\n ---------------------------------------------------------------- --------- ------------------------------\n| /tmp/register724861421/2_my_project.workflows.main_2.pb        | Success | Successfully registered file |\n ---------------------------------------------------------------- --------- ------------------------------\n| /tmp/register724861421/3_my_project.workflows.main_3.pb        | Success | Successfully registered file |\n ---------------------------------------------------------------- --------- ------------------------------\n4 rows\n```\n\n---\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\n---\n\n\u003csummary\u003eℹ What just happened?\u003c/summary\u003e\n\nThe `python deploy.py` command just did the following:\n\n1. Built a docker image specified in your project's `Dockerfile` from within the sandbox docker container.\n2. `flytekit` serializes your tasks and workflows into a `flyte-package.tar.gz` file.\n3. `flytectl` registers those Flyte-compatible artifacts to the playground cluster.\n\n---\n\n\u003c/details\u003e\n\n\nOn the Flyte UI, you'll see a `flytelab-\u003cproject-name\u003e` project namespace on the homepage.\nNavigate to the `my_project.workflows.main` workflow and hit the `Launch Workflow` button, then\nthe `Launch` button on the model form.\n\n🎉 Congrats! You just kicked off your first workflow on your local Flyte sandbox cluster.\n\n#### Fast Deployments\n\nBy default, Flyte uses docker images to encapsulate all the system and python dependencies of\nyour application. If you update those dependencies then you'll need to re-build the docker image.\nHowever, if you want to quickly deploy _code changes_ in your tasks/workflows, you can go through\nfast registration:\n\n```\npython deploy.py --fast\n```\n\n### Union.ai Playground Deployment\n\nThe [Union.ai](https://union.ai/) team maintains a playground Flyte cluster that you can use\nto run your workflows.\n\nWhen you're ready to deploy your workflows to a full-fledged production Flyte cluster, first you'll need to\nrequest an account on the Flyte OSS Slack [`#flytelab` channel](https://flyte-org.slack.com/archives/C032ZU3FSAX).\n\n| **📝 Note** |\n|:---------|\n| For [MLOps Community Engineering Labs Hackathon](https://flyte.org/hackathon/) participants: you will receive these credentials after all teams have been finalized. |\n\nYou'll receive a `username` and `password` to sign into the [Union.ai Playground](https://playground.hosted.unionai.cloud/console), in addition to a `client_id` and `client_secret` if you want to use the [FlyteRemote](https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.html#design-control-plane) object to get the input and output data of your workflow executions from the playground.\n\n#### Hosting Docker Images on Github Container Registry\n\nCreate a [personal access token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) on github.\nMake sure to give your PAT [read and write access to packages](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry)\n\nThen authenticate to the `ghcr.io` registry:\n\n```bash\nexport CONTAINER_REPO_TOKEN=\"\u003cyour-token\u003e\"\necho $CONTAINER_REPO_TOKEN | docker login ghcr.io -u \u003cyour-username\u003e --password-stdin\n```\n\nThen, deploying to the playground is as simple as:\n\n```\npython deploy.py --remote\n```\n\n\u003cdetails\u003e\n\n---\n\n\u003csummary\u003eℹ What just happened?\u003c/summary\u003e\n\nThe `python deploy.py --remote` command just did the following:\n\n1. Built a docker image specified in your project's `Dockerfile`.\n2. Pushed the image to the github container registry under your username's package namespace.\n3. `flytekit` serializes your tasks and workflows into a `flyte-package.tgz` file.\n4. `flytectl` registers those Flyte-compatible artifacts to the playground cluster.\n\n---\n\n\u003c/details\u003e\n\nGo to `https://github.com/users/\u003cyour-username\u003e/packages/container/flytelab/settings`, and then:\n\n1. Click **Add Repository** to link your fork of the `flytelab` repo.\n2. Scroll down to the **Danger Zone**, click **Change visibility**, and make the package public.\n\nFinally, go to https://playground.hosted.unionai.cloud, authenticate with your union.ai playground\n`username` and `password`, where you can navigate to your `flytelab-\u003cproject-name\u003e` project\nto run your workflows.\n\n| **📝 Note** |\n|:---------|\n| Fast registering is currently not enabled in the Union.ai playground. |\n\n\n## 💻 Streamlit App [Optional]\n\nThe `basic` project template ships with a `dashboard/app.py` script that uses\n[`streamlit`](https://streamlit.io/) as a UI for interacting with your model.\n\n```\npip install streamlit\n```\n\n### Run App Locally against Sandbox Cluster\n\n```\nstreamlit run dashboard/app.py\n```\n\n| **📝 Note** |\n|:---------|\n| For the given example, make sure to run the workflow at least once before spinning up the streamlit server. |\n\n### Run App Locally against Union.ai Playground Cluster\n\nTo access the data on the Union.ai playground, first export your `client_id` and `client_secret`\nto your terminal session.\n\n```\nexport FLYTE_CREDENTIALS_CLIENT_ID=\"\u003cclient_id\u003e\"\nexport FLYTE_CREDENTIALS_CLIENT_SECRET=\"\u003cclient_secret\u003e\"\n```\n\nThen start serving your streamlit app with:\n\n```\nstreamlit run dashboard/app.py -- --remote\n```\n\n### Deploying to Streamlit Cloud\n\nIf you want to use [streamlit cloud](https://streamlit.io/cloud) to deploy your app\nto share with the world, push your changes to the remote github branch you're working\nfrom and point streamlit cloud to the streamlit app script:\n\n```\nflytelab/projects/my_project/dashboard/app.py\n```\n\nYou'll need to use their [Secrets management](https://docs.streamlit.io/streamlit-cloud/get-started/deploy-an-app/connect-to-data-sources/secrets-management) system on the streamlit cloud UI\nto add your client id and secret credentials so that it has access to the playground\ncluster:\n\n```bash\nFLYTE_BACKEND = \"remote\"  # point the app to the playground backend\nFLYTE_CREDENTIALS_CLIENT_ID = \"\u003cclient_id\u003e\"  # replace this with your client id\nFLYTE_CREDENTIALS_CLIENT_SECRET = \"\u003cclient_secret\u003e\"  # replace this with your client secret\n```\n\nYou can also add additional secrets to the secrets file if needed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflyteorg%2Fflytelab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflyteorg%2Fflytelab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflyteorg%2Fflytelab/lists"}