{"id":13616619,"url":"https://github.com/matiaslindgren/celery-kubernetes-example","last_synced_at":"2025-04-14T03:31:12.156Z","repository":{"id":45933755,"uuid":"140308928","full_name":"matiaslindgren/celery-kubernetes-example","owner":"matiaslindgren","description":"Small Flask app with scalable, asynchronous backend workers deployed on Kubernetes.","archived":true,"fork":false,"pushed_at":"2022-03-25T15:55:38.000Z","size":358,"stargazers_count":120,"open_issues_count":1,"forks_count":30,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-11-08T01:36:56.179Z","etag":null,"topics":["celery","flask","kubernetes","minikube","producer-consumer","rabbitmq"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matiaslindgren.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-09T15:57:19.000Z","updated_at":"2024-10-04T00:07:20.000Z","dependencies_parsed_at":"2022-09-23T09:32:18.516Z","dependency_job_id":null,"html_url":"https://github.com/matiaslindgren/celery-kubernetes-example","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matiaslindgren%2Fcelery-kubernetes-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matiaslindgren%2Fcelery-kubernetes-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matiaslindgren%2Fcelery-kubernetes-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matiaslindgren%2Fcelery-kubernetes-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matiaslindgren","download_url":"https://codeload.github.com/matiaslindgren/celery-kubernetes-example/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248815514,"owners_count":21165938,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","flask","kubernetes","minikube","producer-consumer","rabbitmq"],"created_at":"2024-08-01T20:01:31.011Z","updated_at":"2025-04-14T03:31:11.801Z","avatar_url":"https://github.com/matiaslindgren.png","language":"Python","funding_links":[],"categories":["HarmonyOS"],"sub_categories":["Windows Manager"],"readme":"# Celery on Kubernetes\n\nToy example of a Kubernetes application with Celery workers.\nThe system consists of a HTTP service which computes long running tasks asynchronously using two different task queues depending on input size.\nThis example is intended for local experimentation with Minikube and is probably not suitable for direct production use.\n\nThe web service is a simple Flask application, deployed in its own pod, along with a single Celery worker for small tasks (two containers in one pod).\nThis system uses RabbitMQ as the Celery message broker and it is deployed as a service in another pod.\nIn addition, a third deployment is created that runs independent, stateless Celery workers for consuming large tasks.\nThe third deployment can be easily scaled.\n\nThe web service is represented by ``myproject`` and long running tasks are simulated with a [poorly implemented](./lcs/lcs/__init__.py) longest common substring algorithm.\n\n## Overview sketch\n\n\u003cimg\n  alt=\"architecture sketch, which shows how the components of this application relate to and interact with each other\"\n  src=\"./celerykube.png\"\n  style=\"max-width: 700px\"\u003e\n\n## Requirements\n\n* [`docker`](https://www.docker.com/get-started) (19.03.12-ce)\n* [`minikube`](https://github.com/kubernetes/minikube) (v1.11.0)\n* [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl) (v1.18.5)\n\nAll application dependencies will be installed into Docker containers.\n\n### Docker Desktop (optional)\n\nYou can probably run all examples without `minikube` if you are using Kubernetes with Docker Desktop.\n\n## Running\n\nAssuming `dockerd` is running and `minikube` is installed, let's deploy the system inside a Minikube cluster.\n\n### Initialization\n\nCreate a Minikube cluster that uses the local `dockerd` environment (skip this step if you are running Kubernetes from Docker Desktop):\n```\nminikube start\neval $(minikube -p minikube docker-env)\n```\nBuild all Docker images:\n```\ndocker build --tag myproject:1 --file myproject/Dockerfile .\ndocker build --tag consumer-small:1 --file consumer-small/Dockerfile .\ndocker build --tag consumer-large:1 --file consumer-large/Dockerfile .\n```\nCheck that the images were created successfully:\n```\ndocker images\n```\nOutput:\n```\nREPOSITORY           TAG       IMAGE ID       CREATED         SIZE\nconsumer-large       1         ddfec2f889ad   3 minutes ago   67.2MB\nconsumer-small       1         99e589f61f63   3 minutes ago   72.4MB\nmyproject            1         bbed507879da   3 minutes ago   72.4MB\n```\n\n### Deploying applications\n\nDeploy the RabbitMQ message broker as a service inside the cluster:\n```\nkubectl create --filename message_queue/rabbitmq-deployment.yaml\nkubectl create --filename message_queue/rabbitmq-service.yaml\n```\nDeploy the `myproject` Flask web service and its `consumer-small` Celery worker:\n```\nkubectl create --filename myproject/deployment.yaml\n```\nThen deploy the `consumer-large` Celery worker for large tasks in its own pod:\n```\nkubectl create --filename consumer-large/deployment.yaml\n```\nCheck that we have 3 pods running:\n```\nkubectl get pods\n```\nOutput:\n```\nNAME                              READY   STATUS    RESTARTS   AGE\nconsumer-large-7f44489db9-9btcf   1/1     Running   0          3s\nmyproject-648fbdff85-kw78t        2/2     Running   0          7s\nrabbitmq-68447cbdf5-ktj4v         1/1     Running   0          14s\n```\nNote that you might have different names for the pods.\nI'll be using the above pod names but you should use the ones printed by `kubectl get pods`.\n\n### Inspecting application logs\n\nCheck that all applications are running and the Celery workers can connect to the broker.\n\nFlask web server:\n```\nkubectl logs myproject-648fbdff85-kw78t --container myproject\n```\nCelery worker for small tasks:\n```\nkubectl logs myproject-648fbdff85-kw78t --container consumer-small\n```\nCelery worker for large tasks:\n```\nkubectl logs consumer-large-7f44489db9-9btcf\n```\nRabbitMQ message broker:\n```\nkubectl logs rabbitmq-68447cbdf5-ktj4v\n```\n\nI prefer to open new terminals or `tmux` for all applications and then use `kubectl logs --follow` to monitor all logs interactively.\n\n### Interacting with the web app\n\nNow everything is running and we can expose the Flask web app port to our local machine:\n```\nkubectl port-forward deployment/myproject 5000:5000\n```\nThen open http://localhost:5000/ in a browser and you should see a simple web UI.\n\nTry copy-pasting some strings and compute the longest common substrings for them.\nE.g. first try short strings and check that the tasks show up in the Celery logs of pod `consumer-small`.\nThen try long strings (over 1000 chars) and check the Celery logs of pod `consumer-large`.\nThe `consumer-large` pods run Celery workers with `--concurrency 2`, so you should be seeing two CPUs being utilized when submitting two or more large tasks at the same time.\n\n### Scaling up\n\nThe `consumer-large` deployment creates stateless Celery worker pods, which can be scaled easily to e.g. 4 pods with:\n```\nkubectl scale deployment/consumer-large --replicas=4\n```\nYou should now have 6 pods running:\n```\nkubectl get pods\n```\nIf you submit several large tasks now, you should see much higher CPU usage.\n\n### Other useful things\n\nGet a shell to the container that is running the Flask app:\n```\nkubectl exec --stdin --tty myproject-648fbdff85-kw78t --container myproject -- /bin/bash\n```\nThen e.g. delete all data from the SQL database:\n```\npython3 -c 'import sqlite3\nconn = sqlite3.connect(\"/data/myproject.sqlite3.db\")\nconn.execute(\"delete from tasks\")\nconn.commit()'\n```\nRefresh the task list and all results should now be empty.\n\nGet a shell to the large tasks Celery worker container:\n```\nkubectl exec --stdin --tty consumer-large-7f44489db9-9btcf -- /bin/bash\n```\nInspect the Celery worker state:\n```\ncelery inspect active_queues --broker=$CELERY_BROKER_URL\ncelery inspect report --broker=$CELERY_BROKER_URL\n```\n\n## Cleanup\nTerminate all pods by removing the deployments:\n```\nkubectl delete deploy myproject consumer-large rabbitmq\nkubectl delete service rabbitmq-service\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatiaslindgren%2Fcelery-kubernetes-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatiaslindgren%2Fcelery-kubernetes-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatiaslindgren%2Fcelery-kubernetes-example/lists"}