{"id":21539289,"url":"https://github.com/nextdoor/ndkale","last_synced_at":"2025-04-07T15:08:40.481Z","repository":{"id":29464483,"uuid":"33001010","full_name":"Nextdoor/ndkale","owner":"Nextdoor","description":"Kale is a python task worker library that supports priority queues on Amazon SQS","archived":false,"fork":false,"pushed_at":"2024-01-11T16:21:20.000Z","size":201,"stargazers_count":204,"open_issues_count":10,"forks_count":37,"subscribers_count":124,"default_branch":"master","last_synced_at":"2025-03-31T12:08:37.744Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://engblog.nextdoor.com/ac4f7886957b","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Nextdoor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-03-27T17:54:25.000Z","updated_at":"2025-03-12T22:50:13.000Z","dependencies_parsed_at":"2024-12-31T10:12:22.712Z","dependency_job_id":"b807b090-9be8-4717-9578-57b95404b36c","html_url":"https://github.com/Nextdoor/ndkale","commit_stats":{"total_commits":98,"total_committers":25,"mean_commits":3.92,"dds":0.8367346938775511,"last_synced_commit":"c5782535496c55819fb6966792b923678677a112"},"previous_names":[],"tags_count":29,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nextdoor%2Fndkale","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nextdoor%2Fndkale/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nextdoor%2Fndkale/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nextdoor%2Fndkale/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Nextdoor","download_url":"https://codeload.github.com/Nextdoor/ndkale/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247675597,"owners_count":20977376,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-24T04:14:40.001Z","updated_at":"2025-04-07T15:08:40.460Z","avatar_url":"https://github.com/Nextdoor.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kale: Distributed task worker from Nextdoor\n\n![Apache](https://img.shields.io/hexpm/l/plug.svg) \n[![Build Status](https://travis-ci.org/Nextdoor/ndkale.svg?branch=master)](https://travis-ci.org/Nextdoor/ndkale)\n\nKale is a python task worker library that supports priority queues on Amazon SQS. \n\nCheck out our blog post - [Nextdoor Taskworker: Simple, Efficient \u0026 Scalable](https://engblog.nextdoor.com/ac4f7886957b)\n\n## How does it work?\n\n![Kale-based Taskworker](https://cloud.githubusercontent.com/assets/1719237/16959018/e4fe6378-4d97-11e6-9903-d4f4f576524d.png)\n\nLike other distributed task queue system, publishers send task messages to queues and workers fetch messages from queues. For now, Kale supports only Amazon SQS for the queue.\n\n### Publisher\n\nA publisher can be any python program that imports a Kale-based task class and invokes the publish function of this class. For example, if a task class looks like this:\n\n    # tasks.py\n    \n    class MyTask:\n        def run_task(self, arg1, arg2, *args, **kwargs):\n            # Do something\n            \n\nThen the publisher publishes a task to Amazon SQS, which normally takes 10s miliseconds to return:\n\n    import tasks\n    tasks.MyTask.publish(None, arg1, arg2)\n   \nThe publish() function is a static method of a task class.  Other than the first parameter, which can usually be `None`, it has the same signiture as the `run_task()` method. A worker process, which may run on a different machine, will pick up the message and execute `run_task()` method of the task.\n\nWhile ndkale is usable out of the box, the first parameter in `publish(app_data, *args, *kwargs)` is designed for more complex situations where certain state may need to be passed outside the context of the actual task parameters.  One example of this might be to pass the environment.  The `app_data` must be pickleable so that in can be encoded and inserted into the SQS message.\n\nThe default task object will be populated with an `app_data` attribute, but the default worker will not use it.  You will need to extend the default Worker or Task class to take advantage of `app_data`.\n\n### Worker\n\n![task lifecycle](https://cloud.githubusercontent.com/assets/1719237/16958964/b1a1c38a-4d97-11e6-9ea3-abdc86630732.png)\n\nA worker process runs an infinite loop. For each iteration, it does the following things:\n\n1. It runs a **queue selection algorithm** (select_queue) to decides which queue to fetch tasks from;\n2. It fetches a batch of tasks from a queue (get_messages);\n3. It runs tasks one by one in the same batch (run_task);\n4. Finish up.\n    1. If a task succeeded, it'll be deleted from the queue;\n    2. If a task runs too long (exceeding **time_limit** that is a per task property for task SLA) or it fails,\n      it'll be put back to the queue and other workers will pick it up in the future (if retry is allowed);\n    3. If a batch of tasks runs too long, exceeding **visibility timeout** that is a per queue property for task\n      batch SLA, then unfinished tasks will be put back to the queue and other workers will pick them up in the\n      future.\n5. It exits this iteration and enters next iteration and repeat the above steps.\n\n#### Queue Selection Algorithm\n\nCode: kale/queue_selector.py\n\nA good queue selection algorithm has these requirements:\n\n1. higher priority queues should have more chances to be selected than lower priority queues;\n2. it should not starve low priority queues;\n3. it should not send too many requests to SQS while retrieving no task, which is waste of\n   compute resource and Amazon charges us by the number of requests.\n4. it should not wait on empty queues for too long, avoiding waste of compute resources.\n\nWe experimented and benchmarked quite a few queue selection algorithms. We end up using an\nimproved version of lottery algorithm, **ReducedLottery**, which fulfill the above requirements.\n\nReducedLottery works like this:\n\n        Initialize the lottery pool with all queues\n        while lottery pool is not empty:\n            Run lottery based on queue priority to get a queue who wins the jackpot\n            Short poll SQS to see if the selected queue is empty\n            if the selected queue is not empty:\n                return queue\n            else:\n                Remove this queue from the lottery pool\n        Reset the lottery pool with all queues\n        Return whatever queue who wins the jackpot\n\nThe beauty of **ReducedLottery**:\n\n* It prefers higher priority queues, as higher priority queues get more lottery tickets and have\n  higher chances to win the jackpot. Thus, requirement 1 is fulfilled.\n* It uses randomness to avoid starvation. Lower priority queues still have chance to win the jackpot.\n  Thus, requirement 2 is fulfilled.\n* If the selected queue is empty, SQS will automatically let task worker long poll on the queue,\n  avoiding sending too many requests (short polls). Thus, requirement 3 is fulfilled.\n* It excludes known empty queues from the lottery pool. Only when all queues are empty can it returns\n  an empty queue. So, it's unlikely to long poll on an empty queue. Thus, requirement 4 is fulfilled.\n\n#### Settings\n\nThere are two types of settings, worker config and queue config.\n\n##### Worker config\n\nSettings are specified in settings modules, including AWS confidentials, queues config, queue selection\nalgorithm, ...\n\nSettings modules are loaded in such order:\n\n* kale.default_settings\n* the module specified via KALE\\_SETTINGS\\_MODULE environment variable\n\nHere's an example\n\n        import os\n\n        AWS_REGION = 'us-west-2'\n\n        #\n        # Production settings\n        # (use this for prod to talk to Amazon SQS)\n\n        # MESSAGE_QUEUE_USE_PROXY = False\n        # AWS_ACCESS_KEY_ID = 'AWS KEY ID'\n        # AWS_SECRET_ACCESS_KEY = ''AWS SECRET KEY\n\n        #\n        # Development settings\n        # (use this for dev to talk to ElasticMQ, which is SQS emulator)\n\n        # Using elasticmq to emulate SQS locally\n        MESSAGE_QUEUE_USE_PROXY = True\n        MESSAGE_QUEUE_PROXY_PORT = 9324\n        MESSAGE_QUEUE_PROXY_HOST = os.getenv('MESSAGE_QUEUE_PROXY_HOST', '0.0.0.0')\n        AWS_ACCESS_KEY_ID = 'x'\n        AWS_SECRET_ACCESS_KEY = 'x'\n\n        QUEUE_CONFIG = 'taskworker/queue_config.yaml'\n\n        # SQS limits per message size, bytes\n        # It can be set anywhere from 1024 bytes (1KB), up to 262144 bytes (256KB).\n        # See http://aws.amazon.com/sqs/faqs/\n        SQS_TASK_SIZE_LIMIT = 256000\n\n        QUEUE_SELECTOR = 'kale.queue_selector.ReducedLottery'\n\n\nSettings in the later modules overwrite those in the early-loaded modules.\n\n##### Queue config\n\nAll queues and their properties are in a queues config yaml file whose path is specified in the above\nsettings modules.\n\nHere's an example\n\n        # task SLA: 60/10 = 6 seconds\n        high_priority:\n            name: high_priority\n            priority: 100\n            batch_size: 10\n            visibility_timeout_sec: 60\n            long_poll_time_sec: 1\n            num_iterations: 10\n\n        # task SLA: 60 / 10 = 6 seconds\n            default:\n            name: default\n            priority: 40\n            batch_size: 10\n            visibility_timeout_sec: 60\n            long_poll_time_sec: 1\n            num_iterations: 5\n\n        # task SLA: 60 / 10 = 6 seconds\n        low_priority:\n            name: low_priority\n            priority: 5\n            batch_size: 10\n            visibility_timeout_sec: 60\n            long_poll_time_sec: 5\n            num_iterations: 5\n\n## How to implement a distributed task worker system using Kale\n\n### Install kale\n    \nFrom source code\n    \n    python setup.py install\n    \nUsing pip (from github repo)\n\n    #\n    # Put this in requirements.txt, then run\n    #    pip install -r requirements.txt\n    #\n\n    # If you want the latest build\n    git+https://github.com/Nextdoor/ndkale.git#egg=ndkale\n\n    # Or put this if you want a specific commit\n    git+https://github.com/Nextdoor/ndkale.git@67f873ed7b0a8131cc8d72453d749ffb389d695f\n    \n    #\n    # Run from command line\n    #\n\n    pip install -e git+https://github.com/Nextdoor/ndkale.git#egg=ndkale\n\n(We'll upload the package to PyPI soon.)\n\n### Example implementation\n\nSee code in the example/ directory.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnextdoor%2Fndkale","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnextdoor%2Fndkale","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnextdoor%2Fndkale/lists"}