{"id":20310913,"url":"https://github.com/progrium/miyamoto","last_synced_at":"2025-10-19T21:27:15.700Z","repository":{"id":66709110,"uuid":"1171629","full_name":"progrium/miyamoto","owner":"progrium","description":null,"archived":false,"fork":false,"pushed_at":"2011-04-21T03:28:53.000Z","size":568,"stargazers_count":45,"open_issues_count":1,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-11T16:11:50.178Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/progrium.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2010-12-15T16:38:39.000Z","updated_at":"2022-01-22T16:21:16.000Z","dependencies_parsed_at":"2023-02-20T09:45:32.379Z","dependency_job_id":null,"html_url":"https://github.com/progrium/miyamoto","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/progrium/miyamoto","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/progrium%2Fmiyamoto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/progrium%2Fmiyamoto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/progrium%2Fmiyamoto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/progrium%2Fmiyamoto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/progrium","download_url":"https://codeload.github.com/progrium/miyamoto/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/progrium%2Fmiyamoto/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265558718,"owners_count":23787969,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T17:34:59.365Z","updated_at":"2025-10-19T21:27:10.661Z","avatar_url":"https://github.com/progrium.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Miyamoto Task Queue\n\nMiyamoto is a fast, clusterable task queue inspired by Google App Engine's task \nqueue. This means it speaks HTTP with a RESTful API for enqueuing tasks and \nuses HTTP callbacks (webhooks) for processing tasks. Worker daemons can be any \nweb daemon.\n\nEnd result? Super easy asynchronous processing.\n\nLike the App Engine task queue, Miyamoto features task scheduling and rate \nlimiting. Unlike the App Engine task queue, it provides idempotency semantics\nand helps you avoid duplicate execution of tasks.\n\nMiyamoto is designed for:\n - High throughput (fast)\n - Fault tolerance (highly available)\n - Data replication (minimal lost messages)\n - No major dependencies or client libs (easy to set up and use)\n - Simple design/implementation (easy to hack!)\n\nMiyamoto is bad if you want:\n - Slow disk persistence (but we may add it)\n - Traditional task polling\n - Smart clients\n - AMQP, JMS, STOMP or other interfaces*\n\nNot out of the box, but you can still achieve:\n - Synchronous tasks\n - Return values / result storage\n - Workers behind firewall / NAT\n - Dynamically adding to the worker pool\n \n* = Excluding ZeroMQ, which is used internally, so there are optional ZMQ interfaces\n \nGETTING STARTED\n\nMake sure you have Python 2.6+ installed with gevent. Run on port 8088 with:\n\n    INTERFACE=127.0.0.1 python scripts/start.py\n\nUSAGE\n\nOnce it's running (currently hardcoded to 8088), set up a pretend web server with netcat:\n\n    nc -l 9099\n\nNow let's post a task to it scheduled to run in 5 seconds: \n\n    curl -d \"task.url=http://localhost:9099/worker\u0026task.countdown=5\u0026foo=bar\" http://localhost:8088/queue\n\nYou should get back some JSON:\n\n    {\"status\": \"scheduled\", \"id\": \"7e998f34-bed7-44f4-80cd-49620b0ab562\"}\n\nAnd in 5 seconds, you should see on your listening netcat:\n\n    POST /worker HTTP/1.1\n    Accept-Encoding: identity\n    Content-Length: 7\n    Host: localhost:9099\n    Content-Type: application/x-www-form-urlencoded\n    X-Task-Id: 7e998f34-bed7-44f4-80cd-49620b0ab562\n    Connection: close\n    User-Agent: miyamoto/0.1\n    \n    foo=bar\n\nYou can also add tasks as JSON objects. Here's one without a delay:\n\n    curl -H \"Content-Type: applicatoin/json\" -d '{\"url\": \"http://localhost:9099/worker\", \"params\": {\"foo\": \"bar\"}}' http://localhost:8088/queue\n   \n\nWHAT'S GOING ON?\n\nIn theory, you'd run a cluster of Miyamoto nodes. Each one is exactly the same, \nexcept that one will be the leader. You seed the leader with the first node, but\notherwise, leaders are elected if, say, the leader dies. Internally there is a list\nof hosts in the cluster that form a ring. Whenever you add tasks to any node, it \nwill send the task round robin to another node in the cluster for basic load \nbalancing. But it will do this N times where N is a replication factor. Each replicant\nhas an added delay to execution. If any of them run, the others are canceled.\nIn this way you can schedule tasks and not worry that hosts might go down, achieving\nHA of service and data. \n\nADVERTISED FEATURES NOT YET DONE\n\n- rate limiting\n- idempotency safeguards\n- error handling\n\nINSPIRATION\n\nApp Engine - API, features\nCelery - features, anti-features\nMemcache - ideas around consistent hashing\nMembase - replication, clustering\nZooKeeper - realtime group membership, leader election\nKestrel - speed, simplicity\nZeroMQ - oh, the possibilities\n\nCONTRIBUTORS\n\nEvan Cooke - offset replica scheduling idea\nJJ Behrens - code review, feedback\n\nLICENSE\n\nMIT","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprogrium%2Fmiyamoto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprogrium%2Fmiyamoto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprogrium%2Fmiyamoto/lists"}