{"id":19678898,"url":"https://github.com/emilrex/kubernetes-worker-demo","last_synced_at":"2025-04-29T04:30:53.582Z","repository":{"id":161221973,"uuid":"635955491","full_name":"EmilRex/kubernetes-worker-demo","owner":"EmilRex","description":null,"archived":false,"fork":false,"pushed_at":"2023-07-06T22:43:58.000Z","size":24,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-05T13:01:39.543Z","etag":null,"topics":["kubernetes","prefect"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EmilRex.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-03T20:21:10.000Z","updated_at":"2024-06-22T15:04:43.000Z","dependencies_parsed_at":null,"dependency_job_id":"e4a1d698-2bf7-4de0-afa7-424218166273","html_url":"https://github.com/EmilRex/kubernetes-worker-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EmilRex%2Fkubernetes-worker-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EmilRex%2Fkubernetes-worker-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EmilRex%2Fkubernetes-worker-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EmilRex%2Fkubernetes-worker-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EmilRex","download_url":"https://codeload.github.com/EmilRex/kubernetes-worker-demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251432644,"owners_count":21588621,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kubernetes","prefect"],"created_at":"2024-11-11T17:39:52.674Z","updated_at":"2025-04-29T04:30:53.574Z","avatar_url":"https://github.com/EmilRex.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kubernetes Worker Demo\n\nThis repo provides an overview of the Prefect Kubernetes worker and how to use it. Some parts are specific to AWS, but the concepts should transfer to all major cloud providers.\n\n## Requirements\n\n* AWS account\n* Prefect Cloud account\n* GitHub account\n\n## Resources\n\n- [Introducing Prefect Workers and Projects](https://www.prefect.io/guide/blog/introducing-prefect-workers-and-projects/)\n- [Worker Helm Chart](https://github.com/PrefectHQ/prefect-helm/tree/main/charts/prefect-worker#prefect-worker)\n- [Projects Documentation](https://docs.prefect.io/latest/concepts/projects/)\n- [Worker Pools Documentation](https://docs.prefect.io/latest/concepts/work-pools/)\n\n## Create a cluster\n\nLet's start by creating a new cluster. If you already have one, skip ahead to the next section. We'll use EKS backed by FARGATE,\n\n```bash\neksctl create cluster --fargate --name kubernetes-worker\n```\n\nThe process should take about 15 minutes and produce something like the following output,\n\n```\n2023-05-02 15:19:00 [ℹ]  eksctl version 0.124.0\n2023-05-02 15:19:00 [ℹ]  using region us-east-2\n2023-05-02 15:19:00 [ℹ]  setting availability zones to [us-east-2a us-east-2b us-east-2c]\n2023-05-02 15:19:00 [ℹ]  subnets for us-east-2a - public:192.168.0.0/19 private:192.168.96.0/19\n2023-05-02 15:19:00 [ℹ]  subnets for us-east-2b - public:192.168.32.0/19 private:192.168.128.0/19\n2023-05-02 15:19:00 [ℹ]  subnets for us-east-2c - public:192.168.64.0/19 private:192.168.160.0/19\n2023-05-02 15:19:00 [ℹ]  using Kubernetes version 1.23\n2023-05-02 15:19:00 [ℹ]  creating EKS cluster \"kubernetes-worker\" in \"us-east-2\" region with Fargate profile\n2023-05-02 15:19:00 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-east-2 --cluster=kubernetes-worker'\n2023-05-02 15:19:00 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster \"kubernetes-worker\" in \"us-east-2\"\n2023-05-02 15:19:00 [ℹ]  CloudWatch logging will not be enabled for cluster \"kubernetes-worker\" in \"us-east-2\"\n2023-05-02 15:19:00 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=us-east-2 --cluster=kubernetes-worker'\n2023-05-02 15:19:00 [ℹ]\n2 sequential tasks: { create cluster control plane \"kubernetes-worker\",\n    2 sequential sub-tasks: {\n        wait for control plane to become ready,\n        create fargate profiles,\n    }\n}\n2023-05-02 15:19:00 [ℹ]  building cluster stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:19:01 [ℹ]  deploying stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:19:31 [ℹ]  waiting for CloudFormation stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:20:01 [ℹ]  waiting for CloudFormation stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:21:01 [ℹ]  waiting for CloudFormation stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:22:01 [ℹ]  waiting for CloudFormation stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:23:02 [ℹ]  waiting for CloudFormation stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:24:02 [ℹ]  waiting for CloudFormation stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:25:02 [ℹ]  waiting for CloudFormation stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:26:02 [ℹ]  waiting for CloudFormation stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:27:02 [ℹ]  waiting for CloudFormation stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:28:03 [ℹ]  waiting for CloudFormation stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:29:03 [ℹ]  waiting for CloudFormation stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:30:03 [ℹ]  waiting for CloudFormation stack \"eksctl-kubernetes-worker-cluster\"\n2023-05-02 15:32:04 [ℹ]  creating Fargate profile \"fp-default\" on EKS cluster \"kubernetes-worker\"\n2023-05-02 15:34:15 [ℹ]  created Fargate profile \"fp-default\" on EKS cluster \"kubernetes-worker\"\n2023-05-02 15:34:45 [ℹ]  \"coredns\" is now schedulable onto Fargate\n2023-05-02 15:35:48 [ℹ]  \"coredns\" is now scheduled onto Fargate\n2023-05-02 15:35:48 [ℹ]  \"coredns\" pods are now scheduled onto Fargate\n2023-05-02 15:35:48 [ℹ]  waiting for the control plane to become ready\n2023-05-02 15:35:49 [✔]  saved kubeconfig as \"/Users/emilchristensen/.kube/config\"\n2023-05-02 15:35:49 [ℹ]  no tasks\n2023-05-02 15:35:49 [✔]  all EKS cluster resources for \"kubernetes-worker\" have been created\n2023-05-02 15:35:50 [ℹ]  kubectl command should work with \"/Users/emilchristensen/.kube/config\", try 'kubectl get nodes'\n2023-05-02 15:35:50 [✔]  EKS cluster \"kubernetes-worker\" in \"us-east-2\" region is ready\n```\n\n## Create a service account\n\nIn order to authenticate with Prefect cloud, we'll want to use a service account. Create one with the Developer role in your workspace. Securely store the API key and set it as a Kubernetes secret,\n\n```bash\nkubectl create secret generic prefect-api-key --from-literal=key=\u003cYOUR-KEY-HERE\u003e\n```\n\n## Create a work pool\n\nNext, go to the work pools page and create a new work pool with type `Kubernetes`. Set Finished Job TTL to 60 and Pod Watch Timeout Seconds to 300, but otherwise leave the defaults. We'll name our workpool `kubernetes`.\n\n## Start a worker\n\nNow we're ready to start a worker using the [Prefect Helm chart](https://github.com/prefecthq/prefect-helm#prefect-worker). Create a file called `values.yaml` and fill it in,\n\n```yaml\nworker:\n  cloudApiConfig:\n    accountId: \"\u003cYOUR-ACCOUNT-ID\u003e\"\n    workspaceId: \"\u003cYOUR-WORKSPACE-ID\u003e\"\n  config:\n    workPool: \"kubernetes\"\n```\n\nNow we can install the worker,\n\n```bash\nhelm repo add prefect https://prefecthq.github.io/prefect-helm/\n\nhelm repo update\n\nhelm install prefect-worker prefect/prefect-worker -f values.yaml\n```\n\n## Create an ECR repository\n\nWe'll want a remote image repository to store any custom images we build. If you already have a repository you want to use, skip this step. Here we'll use ECR, making sure to log in,\n\n```bash\n# Create a repo unless you already have one you wish to use\naws ecr create-repository --repository-name custom-flow-image\n\n# Login to ECR\n# Replace the region and account ID with your own values\naws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com\n```\n\n## Create a variable\n\nWe could hardcode the image name our `prefect.yaml` file, but for the sake of demonstration and security, let's use a Prefect variable. Go to the variables page in the UI, set `Name` to `image_name`, and set `Value` to `$AWS_ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/custom-flow-image` or equivalent.\n\n## Deploy and run\n\nAt last we can deploy our flows and run them. The deploy command will actually build the images and push them to our remote repository.\n\n```bash\nprefect deploy --all\n\nprefect deployment run hello/default\nprefect deployment run hello/arthur\nprefect deployment run hello-parallel/default\n```\n\n## Configure GitHub Actions\n\nWe can configure GitHub Actions to build our images and deploy our flows on every push to main. To do so we'll create an IAM user for GitHub to use when pushing images to ECR,\n\n```bash\nIAM_USER_NAME=github-actions-deploy\n\naws iam create-user --user-name $IAM_USER_NAME\n\ncat \u003c\u003cEOF \u003e policy.json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"ecr:GetAuthorizationToken\",\n        \"ecr:BatchCheckLayerAvailability\",\n        \"ecr:GetDownloadUrlForLayer\",\n        \"ecr:BatchGetImage\",\n        \"ecr:PutImage\",\n        \"ecr:InitiateLayerUpload\",\n        \"ecr:UploadLayerPart\",\n        \"ecr:CompleteLayerUpload\"\n      ],\n      \"Resource\": \"*\"\n    }\n  ]\n}\nEOF\n\naws iam put-user-policy --user-name $IAM_USER_NAME --policy-name allow-push-ecr --policy-document file://policy.json\n\naws iam create-access-key --user-name $IAM_USER_NAME\n```\n\nThen set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `PREFECT_API_KEY` as GitHub secrets.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femilrex%2Fkubernetes-worker-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Femilrex%2Fkubernetes-worker-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femilrex%2Fkubernetes-worker-demo/lists"}