{"id":17100753,"url":"https://github.com/cheremin/link-imbalance-metastable-issue-simulation","last_synced_at":"2025-03-23T18:42:35.795Z","repository":{"id":143237923,"uuid":"488662724","full_name":"cheremin/link-imbalance-metastable-issue-simulation","owner":"cheremin","description":"Simulation of \"Link imbalance metastable issue\" ","archived":false,"fork":false,"pushed_at":"2022-05-14T06:41:20.000Z","size":85,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-29T00:49:33.031Z","etag":null,"topics":["kotlin","simulation"],"latest_commit_sha":null,"homepage":"https://dev.cheremin.info/2022/05/mystery-of-link-imbalance-metastable.html","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cheremin.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-04T16:26:22.000Z","updated_at":"2022-05-14T06:24:28.000Z","dependencies_parsed_at":"2023-04-25T05:17:04.833Z","dependency_job_id":null,"html_url":"https://github.com/cheremin/link-imbalance-metastable-issue-simulation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheremin%2Flink-imbalance-metastable-issue-simulation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheremin%2Flink-imbalance-metastable-issue-simulation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheremin%2Flink-imbalance-metastable-issue-simulation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheremin%2Flink-imbalance-metastable-issue-simulation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cheremin","download_url":"https://codeload.github.com/cheremin/link-imbalance-metastable-issue-simulation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245152738,"owners_count":20569399,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kotlin","simulation"],"created_at":"2024-10-14T15:14:36.068Z","updated_at":"2025-03-23T18:42:35.777Z","avatar_url":"https://github.com/cheremin.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"Simulation of \"link imbalance metastable issue\" described in \n * [\"Metastable Failures in Distributed Systems\"](https://doi.org/10.1145/3458336.3465286) Nathan Bronson, Abutalib Aghayev, Aleksey Charapko, Timothy Zhu\n * [\"Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale\"](https://engineering.fb.com/2014/11/14/production-engineering/solving-the-mystery-of-link-imbalance-a-metastable-failure-state-at-scale/)\n \nSimulation is built on [kalasim](https://www.kalasim.org/) -- Kotlin discrete-events simulation engine.\n\n### Link imbalance metastable issue \nBasically, issue was happening in facebook datacenters due to interaction between:\n * MRU strategy in DB connection pools, used by apps\n * Multi-fiber network link between apps and DB server\n * Periodical spikes of load of many small short queries\n\nThese 3 pieces together interact in such a way that almost all network activity is attracted\nto a single fiber, which exceeds its capacity.\n\nMore details:\n * [\"Metastable Failures in Distributed Systems\"](https://doi.org/10.1145/3458336.3465286) Nathan Bronson, Abutalib Aghayev, Aleksey Charapko, Timothy Zhu\n * [\"Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale\"](https://engineering.fb.com/2014/11/14/production-engineering/solving-the-mystery-of-link-imbalance-a-metastable-failure-state-at-scale/)\n\n\n### Simulation structure\nSimulation is built around `Pool`, `Fiber`, `Connection` and `Query` domain objects.\n\"Client\" (which is the source of `Query`es) is modelled by `kalasim`'s `ComponentGenerator`\n\"DB\" behavior in current simulation is just a delay, hence it is not modelled explicitly at all.\n                                                         \nBoth `Pool` implementations and `Fiber` are extending `kalasim` `Resource`, which gives\nacquire/release occupation-controlling behavior for free. It is also quite natural choice\n(both pools and fibers are indeed resources with limited capacity), and allows in \npost-processing to use extended statistical data `kalasim` gathers for `Resource`s \n                                                             \nSimulation domain objects are in `MultiFibersMRUPoolSimulation`. \n\nSimulation scenarios are created as tests, e.g. `MultiFibersMRUPoolSimulationScenarios`. \nSuch tests are probabilistic by nature, i.e. success/failure is usually not guaranteed, \nbut happens with high enough probability, e.g. \u003e90%. Many tests are used `RepeatedTest` \nto give a sense of success/failure odds explicitly. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheremin%2Flink-imbalance-metastable-issue-simulation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcheremin%2Flink-imbalance-metastable-issue-simulation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheremin%2Flink-imbalance-metastable-issue-simulation/lists"}