{"id":15426454,"url":"https://github.com/tomconte/sample-keda-queue-jobs","last_synced_at":"2025-04-19T16:27:15.965Z","repository":{"id":46163768,"uuid":"220311999","full_name":"tomconte/sample-keda-queue-jobs","owner":"tomconte","description":"This sample shows how to use KEDA to automatically schedule Kubernetes Jobs based on an Azure Storage Queue trigger.","archived":false,"fork":false,"pushed_at":"2021-11-10T01:07:13.000Z","size":5,"stargazers_count":4,"open_issues_count":1,"forks_count":6,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-29T10:05:14.163Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tomconte.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-07T19:20:07.000Z","updated_at":"2024-07-22T18:45:25.000Z","dependencies_parsed_at":"2022-09-17T03:31:28.645Z","dependency_job_id":null,"html_url":"https://github.com/tomconte/sample-keda-queue-jobs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomconte%2Fsample-keda-queue-jobs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomconte%2Fsample-keda-queue-jobs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomconte%2Fsample-keda-queue-jobs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomconte%2Fsample-keda-queue-jobs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tomconte","download_url":"https://codeload.github.com/tomconte/sample-keda-queue-jobs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249738211,"owners_count":21318501,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T17:56:19.454Z","updated_at":"2025-04-19T16:27:15.948Z","avatar_url":"https://github.com/tomconte.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KEDA jobs with Azure Storage Queues\n\nThis sample shows how to use [KEDA](https://github.com/kedacore/keda) to automatically schedule Kubernetes Jobs based on an Azure Storage Queue trigger.\n\n## Create an Azure Storage Queue\n\nUsing the Azure CLI, create a Resource Group, a Storage Account, and a Queue:\n\n```\nSTORAGE_ACCOUNT_NAME=storageaccountname\nexport QUEUE_NAME=keda-queue\naz group create -l westus -n hello-keda\naz storage account create -g hello-keda -n $STORAGE_ACCOUNT_NAME\nexport AZURE_STORAGE_CONNECTION_STRING=$(az storage account show-connection-string --name $STORAGE_ACCOUNT_NAME --query connectionString -o tsv)\naz storage queue create -n $QUEUE_NAME\n```\n\nYou will need to choose a unique name for the `STORAGE_ACCOUNT_NAME` variable.\n\n## Build and push the queue consumer container image\n\nThe `queue-consumer` directory contains a simple Python script that consumes a single message from an Azure Storage Queue and sleeps for 30 seconds, simulating a very simple job.\n\nThe script requires two environment variables:\n\n- `AzureWebJobsStorage`: the Azure Storage connection string you obtained from the previous step.\n- `QUEUE_NAME`: the name of the queue to read from.\n\nTo schedule the jobs, you will need to build and push the Docker image to a container registry. For exanple, to use Docker Hub:\n\n```\nexport REGISTRY=tomconte\ncd queue-consumer/\ndocker build -t queue-consumer .\ndocker tag queue-consumer $REGISTRY/queue-consumer\ndocker push $REGISTRY/queue-consumer\n```\n\nReplace the value for `REGISTRY` with your own Docker hub profile name.\n\n## Install KEDA\n\n[Follow the instructions](https://github.com/kedacore/keda#setup) to deploy KEDA on your Kubernetes cluster.\n\n## Create the KEDA ScaledObject\n\nThe `azurequeue_scaledobject_jobs.yaml` YAML configuration defines the trigger and the specification of the job to run. You will need to check a few values:\n\n- Set the container image name\n- Check that the queue names are correct\n\nYou will also need to create a secret to store the Azure Storage connection string:\n\n```\nkubectl create secret generic secrets --from-literal=AzureWebJobsStorage=$AZURE_STORAGE_CONNECTION_STRING\n```\n\nYou can then create the ScaledObject:\n\n```\nkubectl apply -f azurequeue_scaledobject_jobs.yaml\n```\n\nNothing will happen right away since our queue is empty!\n\n## Send some messages to the queue\n\nThe `send_messages.py` script can be used to send a bunch of messages to the queue. First install the dependencies:\n\n```\npip install -r requirements.txt\n```\n\nYou will also need to configure the `AzureWebJobsStorage` environment variable (assuming you defined `QUEUE_NAME` above):\n\n```\nexport AzureWebJobsStorage=$AZURE_STORAGE_CONNECTION_STRING\n```\n\nLet's send e.g. a hundred messages into the queue:\n\n```\npython send_messages.py 100\n```\n\n## Watch KEDA at work\n\nNow you can watch jobs being automatically scheduled until the queue has been drained:\n\n```\nkubectl get jobs\n```\n\nIf anything goes wrong, it can be useful to check the logs of the KEDA operator to look for any error messages.\n\n```\nKOP=$(kubectl get pods -n keda -l app=keda-operator -o name)\nkubectl logs $KOP keda-operator -n keda\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomconte%2Fsample-keda-queue-jobs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftomconte%2Fsample-keda-queue-jobs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomconte%2Fsample-keda-queue-jobs/lists"}