{"id":16918268,"url":"https://github.com/vito/executor-pool-spike","last_synced_at":"2025-07-03T09:06:48.163Z","repository":{"id":10887983,"uuid":"13178707","full_name":"vito/executor-pool-spike","owner":"vito","description":"experimenting with self-organizing executor pools","archived":false,"fork":false,"pushed_at":"2014-01-08T23:16:52.000Z","size":268,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-20T22:25:59.474Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"arnokamphuis/HU-Vision-1516-Base","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vito.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-09-28T18:41:38.000Z","updated_at":"2014-01-08T23:16:53.000Z","dependencies_parsed_at":"2022-08-29T19:30:51.464Z","dependency_job_id":null,"html_url":"https://github.com/vito/executor-pool-spike","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vito/executor-pool-spike","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vito%2Fexecutor-pool-spike","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vito%2Fexecutor-pool-spike/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vito%2Fexecutor-pool-spike/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vito%2Fexecutor-pool-spike/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vito","download_url":"https://codeload.github.com/vito/executor-pool-spike/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vito%2Fexecutor-pool-spike/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263296443,"owners_count":23444489,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T19:39:26.358Z","updated_at":"2025-07-03T09:06:48.137Z","avatar_url":"https://github.com/vito.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Executor Pool Experimentation\n\nThis is a bunch of spiked code with the intention of designing\na self-organizing network of Executors for use in Cloud Foundry.\n\n## Running\n\n```\nmkdir gospace/\ncd gospace/\nexport GOPATH=$PWD\ngit clone git@github.com:vito/executor-pool-spike.git src/github.com/vito/executor-pool-spike\ncd src/github.com/vito/executor-pool-spike\ngo get -v ./...\n\n# start nats\nnats-server -d\n\n# start etcd in one window\netcd\n\n# start 10 nodes in another window\nforeman start -c node=10\n\n# in another window, send 100 app.starts:\ngo run spammer/main.go -instances=100\n\n# check the distribution in the logs of 'foreman start'\n```\n\n## Why\n\nCurrently the Executors (the DEAs) must constantly advertise their capacity\nand app placement, and it is left to orchestration components (the CC) to\ndetermine where to place each instance of an app they want to start.\n\nIdeally, the orchestrator is dead simple, and just publishes its intent:\n\"start an instance\", and relies on the system to do the heavy lifting. This\nmeans the business logic of app placement and load balancing has to be done\nelsewhere.\n\nWith this logic in the pool of Executor nodes itself, it becomes much simpler\nto model rolling deploys. Executors can evacuate all of their applications to\nthe rest of the nodes in the pool, without relying on a rube goldberg machine\n(NATS -\u003e Health Manager -\u003e NATS -\u003e Cloud Controller -\u003e CCDB -\u003e Placement Logic\n-\u003e NATS).\n\nIt also becomes simple to model resurrection of crashed applications; when an\napp goes down another node picks it up.\n\nThis is an experiment in self-organizing and loosely-coupled\nExecutor networks to achieve this goal.\n\nThe intention is to come up with a simple, intuitive model that accomplishes\nthree things:\n\n1. Application instances spread across many nodes\n2. Rough balancing of overall application instances\n3. Still fast enough to start 100 instances of a single app in a jiffy\n\n\n## Strategy\n\nThe current strategy is to have the Executors record their intention to start\nan instance as soon as they try, by registering an entry in etcd.\n\n### Glossary\n\n* `volunteering` - a node writing their intention to start an instance\n* `hesitating` - a node waiting before volunteering, to achieve load balancing\n  and spreading of application instances\n\nVolunteering is defined as setting a value at the key `/apps/\u003cguid\u003e/\u003cindex\u003e`.\nIn the simplest model, all nodes try to set the key, and the one that actually\nwrote it knows that he's the one to start it. This is possible because etcd\nreturns a `NewKey` value in the response; this should only be `true` for one\nof them. The rest just drop the request on the floor.\n\nHesitating is a technique used for balancing instances without having to know\nthe layout of the rest of the pool. For example, a node that already has 10\ninstances of an app can know to wait 10 seconds before volunteering. If every\nnode does this, application instances will naturally balance themselves across\nthe pool, at the cost of longer start times for higher instance counts.\n\nWith one node, starting 100 instances will have the following flow (request\nhandling over time moves down, n is some time scale like milliseconds):\n\n```\n-\u003e sleep 0 * n, volunteer for 1, start\n-\u003e sleep 1 * n, volunteer for 2, start\n-\u003e sleep 2 * n, volunteer for 3, start\n-\u003e ...\n-\u003e sleep 99 * n, volunteer for 100, start\n```\n\nAs you can tell, this is not optimal, but it at least rate limits the 100\nstarts, and the node won't fall over under load.\n\nAdding another node dramatically reduces the time it takes, as they'll\neffectively halve the instances they have and thus halve the amount of time\nthey hesitate.\n\n```\nA: sleep 0 * n, volunteer for 1, start\nB: sleep 0 * n, volunteer for 1, fail\n\nA: sleep 1 * n, volunteer for 2, fail\nB: sleep 0 * n, volunteer for 2, start\n\nA: sleep 1 * n, volunteer for 3, start\nB: sleep 1 * n, volunteer for 3, fail\n\nA: sleep 2 * n, volunteer for 4, fail\nB: sleep 1 * n, volunteer for 4, start\n\nA: sleep 2 * n, volunteer for 5, start\nB: sleep 2 * n, volunteer for 5, fail\n\nA: sleep 3 * n, volunteer for 6, fail\nB: sleep 2 * n, volunteer for 6, start\n\n...\n\nA: sleep 50 * n, volunteer for 100, fail\nB: sleep 49 * n, volunteer for 100, start\n```\n\n## Initial Findings\n\nHesitation is the most sensitive part. If you wait too long, it takes minutes\nto start an app. If you don't wait long enough, your instances may be less\nevenly balanced, and more subject to network latency.\n\n### 100 instances, waiting 1 second per instance\n\nThis takes a long time but is pretty uniform.\n\n```\ndistribution:\nnode3.1  | running  10: ▇▇▇▇▇▇▇▇▇▇\nnode2.1  | running  11: ▇▇▇▇▇▇▇▇▇▇▇\nnode1.1  | running  11: ▇▇▇▇▇▇▇▇▇▇▇\nnode4.1  | running  12: ▇▇▇▇▇▇▇▇▇▇▇▇\nnode5.1  | running   9: ▇▇▇▇▇▇▇▇▇\nnode6.1  | running   8: ▇▇▇▇▇▇▇▇\nnode7.1  | running   9: ▇▇▇▇▇▇▇▇▇\nnode8.1  | running  11: ▇▇▇▇▇▇▇▇▇▇▇\nnode10.1 | running  10: ▇▇▇▇▇▇▇▇▇▇\nnode9.1  | running   9: ▇▇▇▇▇▇▇▇▇\n\ntime to start: 8m9.174275195s\n```\n\n### 100 instances, waiting 10 milliseconds per instance\n\nThis was much faster and stayed pretty uniform.\n\n```\ndistribution:\nnode1.1  | running  12: ▇▇▇▇▇▇▇▇▇▇▇▇\nnode2.1  | running   9: ▇▇▇▇▇▇▇▇▇\nnode4.1  | running  10: ▇▇▇▇▇▇▇▇▇▇\nnode5.1  | running   8: ▇▇▇▇▇▇▇▇\nnode3.1  | running  10: ▇▇▇▇▇▇▇▇▇▇\nnode6.1  | running  10: ▇▇▇▇▇▇▇▇▇▇\nnode7.1  | running  10: ▇▇▇▇▇▇▇▇▇▇\nnode10.1 | running   9: ▇▇▇▇▇▇▇▇▇\nnode9.1  | running  11: ▇▇▇▇▇▇▇▇▇▇▇\nnode8.1  | running  11: ▇▇▇▇▇▇▇▇▇▇▇\n\ntime to start: 5.140204818s\n```\n\n### 100 instances, waiting 1 millisecond per instance\n\nThis was much faster without any discernable downsides.\n\n```\ndistribution:\nnode3.1  | running  10: ▇▇▇▇▇▇▇▇▇▇\nnode1.1  | running   9: ▇▇▇▇▇▇▇▇▇\nnode2.1  | running   9: ▇▇▇▇▇▇▇▇▇\nnode7.1  | running   8: ▇▇▇▇▇▇▇▇\nnode4.1  | running  11: ▇▇▇▇▇▇▇▇▇▇▇\nnode5.1  | running   9: ▇▇▇▇▇▇▇▇▇\nnode10.1 | running   9: ▇▇▇▇▇▇▇▇▇\nnode6.1  | running  11: ▇▇▇▇▇▇▇▇▇▇▇\nnode9.1  | running  12: ▇▇▇▇▇▇▇▇▇▇▇▇\nnode8.1  | running  12: ▇▇▇▇▇▇▇▇▇▇▇▇\n\ntime to start: 708.96801ms\n```\n\n### 100 instances, NO DELAY (control)\n\nYeah, don't do that. Half the nodes didn't even get a chance.\n\n```\ndistribution:\nnode2.1  | running   2: ▇▇\nnode3.1  | running   3: ▇▇▇\nnode1.1  | running   4: ▇▇▇▇\nnode6.1  | running  56: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇\nnode8.1  | running  35: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇\n\ntime to start: 399.383626ms\n```\n\n### Interpretation\n\nAt least on a local machine, sleeping for smaller amounts of time does not\nappear to adversely affect distribution, and has the most reward. However\nonce you're down in the millisecond range this may become subject to network\nlatency. The worst case is that you have no delay; doing this got nowhere\nclose to an even spread, and some didn't even get any instances.\n\nAlso, even with the most naive approach of sleeping 1 second per instance, you\nat least immediately have 1 instance starting on every node. This means if you\nhave 10 nodes you can start 10 instances immediately. The rest \"trickle in\"\nover time.\n\n\n## Next Steps\n\n### Rate-Limiting \u0026 Preventing Start Storms\n\n\"Hesitation\" becomes a natural place to rate-limit start requests that are\nperformed by the Executor, to improve stability and remove the possiblity of\na storm of start requests taking down your nodes.\n\n### Evacuation \u0026 Crash Recovery\n\nIf in the act of volunteering, the node wrote to the key enough information\nfor other nodes to start the same instance (like the message the node\nresponded to), this can be trivially extended for evacuation and crashed\ninstances.\n\nOther nodes watch the path that the nodes write to, and respond to DELETE\nevents. If an entry disappeared while in a RUNNING state, all other nodes\ntreat it as if a start was requested: they perform the same hesitation and\nvolunteering flow as starting instances.\n\nThe key can be given a TTL and as long as a node is running an instance, he\nkeeps bumping it.\n\nIf the instance crashes or the node is being evacuated, the node running the\ninstance deletes its key.\n\nIf an instance is shut down by the user, the node writes STOPPED state to the\nkey, so other nodes know not to volunteer (a node seeing a DELETE event will\nsee the value of the key), and then deletes it.\n\nIf a node hard-crashes, the TTL will expire and a DELETE event will trigger.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvito%2Fexecutor-pool-spike","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvito%2Fexecutor-pool-spike","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvito%2Fexecutor-pool-spike/lists"}