{"id":20727329,"url":"https://github.com/blue-yonder/mesos-threshold-oversubscription","last_synced_at":"2025-04-23T18:54:01.713Z","repository":{"id":83330297,"uuid":"68820049","full_name":"blue-yonder/mesos-threshold-oversubscription","owner":"blue-yonder","description":"Simple, threshold-based oversubscription modules for Apache Mesos","archived":false,"fork":false,"pushed_at":"2019-11-14T11:41:12.000Z","size":578,"stargazers_count":43,"open_issues_count":0,"forks_count":6,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-03-30T02:33:26.341Z","etag":null,"topics":["mesos","oversubscription","revocable","utilization"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blue-yonder.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-09-21T13:33:07.000Z","updated_at":"2023-08-04T07:20:37.000Z","dependencies_parsed_at":"2023-03-03T18:45:14.697Z","dependency_job_id":null,"html_url":"https://github.com/blue-yonder/mesos-threshold-oversubscription","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blue-yonder%2Fmesos-threshold-oversubscription","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blue-yonder%2Fmesos-threshold-oversubscription/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blue-yonder%2Fmesos-threshold-oversubscription/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blue-yonder%2Fmesos-threshold-oversubscription/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blue-yonder","download_url":"https://codeload.github.com/blue-yonder/mesos-threshold-oversubscription/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250496780,"owners_count":21440228,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mesos","oversubscription","revocable","utilization"],"created_at":"2024-11-17T04:30:01.483Z","updated_at":"2025-04-23T18:54:01.687Z","avatar_url":"https://github.com/blue-yonder.png","language":"C++","funding_links":[],"categories":["Modules"],"sub_categories":["Networking"],"readme":"Threshold-based Mesos Oversubscription\n======================================\n\n[![LICENSE](https://img.shields.io/badge/license-Apache-blue.svg)](https://github.com/blue-yonder/mesos-threshold-oversubscription/blob/master/LICENSE)\n\nThis repository contains two simple\n[Mesos oversubscription](http://mesos.apache.org/documentation/latest/oversubscription/) modules:\n\n* `ThresholdResourceEstimator`: Informs the Mesos master about resources that can be oversubscribed\n  on the associated agent.\n\n  Similar to the `fixed` resource estimator provided by Mesos, it allows operators to configure a\n  fixed amount of oversubscribable revocable resources per agent. However, once system utilization\n  reaches a defined threshold, resources will be cut to the currently used amount, avoiding further\n  scheduling of revocable tasks.\n\n* `ThresholdQoSController`: Informs the Mesos agent that particular corrective actions need to be\n  made to prevent quality-of-service violations (i.e. system overload).\n\n  Following the same principle as the `ThresholdResourceEstimator`, corrective actions are taken\n  whenever the system utilization reaches a configurable threshold. The controller kills one\n  revocable task per iteration and thus slowly corrects resource estimations that turned out to be\n  over-optimistic.\n\n\nIs it any good?\n---------------\n\nThe threshold-based approach enabled us to double the peak CPU and peak memory utilization in\nour Mesos/Aurora clusters. Your mileage may vary significantly, so please take this statement\nwith a grain of salt.\n\n\nMechanics\n---------\n\nResource estimator and QoS controller aim to work in unison to ensure that the system utilization\nis never approaching critical levels that could negativly affect non-revocable tasks.\n\n![threshold mechanics](docs/oversubscription.png)\n\nRevocable resources will only be offered if the system utilization remains below the estimation\nthreshold. If utilization spikes above, no more revocable resources are offered. If it further\nsurpasses the QoS threshold, the controller will begin to kill revocable tasks until the\nutilization drops. Assuming the utilization drops below the QoS but not below the estimation\nthreshold, the freed resources will not be re-offered, thus preventing further overload of the host.\n\n\nInstallation\n------------\n\nThis project uses CMake. Build requires Mesos development headers and a compatible version of GCC.\nThe Vagrant file in this repository creates a proper build environment for Debian Jessie.\n\nBuild and installation follow the usual CMake tripplet. You can use the CXX and CC environment\nvariable to ensure that a compatible compiler is selected.\n\n    mkdir -p build\n    cd build\n    cmake ..\n    make\n    make test\n    make install\n\n\nConfiguration\n-------------\n\nThe estimator and controller implementations assume the Mesos agent is using the following isolation\nmechanims:\n\n```bash\n--isolation=cgroups/cpu,cgroups/mem  # enable cgroup isolation and accounting\n--revocable_cpu_low_priority         # run revocable tasks with minimal CPU shares\n--cgroups_enable_cfs                 # enforce upper limit of usable CPU shares\n                                     # (optional but recommended)\n```\n\nTo enable the oversubscription modules, launch the Mesos agent with the following flags. The given\nexample assumes you have a host with 256000 MB RAM and 40 CPUs that you want to oversubscribe with\nat most 96000 MB RAM and 16 cores:\n\n```json\n--resource_estimator=\"com_blue_yonder_ThresholdResourceEstimator\"\n--qos_controller=\"com_blue_yonder_ThresholdQoSController\"\n--oversubscribed_resources_interval=15secs\n--qos_correction_interval_min=15secs\n\n--modules='{\n  \"libraries\": {\n    \"file\": \"/\u003cpath\u003e/\u003cto\u003e/libthreshold_oversubscription.so\",\n    \"modules\": [\n      {\n        \"name\": \"com_blue_yonder_ThresholdResourceEstimator\",\n        \"parameters\": [\n          { \"key\": \"resources\",\n            \"value\": \"cpus:16;mem:96000\"},\n          { \"key\": \"load_threshold_1min\",\n            \"value\": \"64\"},\n          { \"key\": \"load_threshold_5min\",\n            \"value\": \"48\" },\n          { \"key\": \"load_threshold_15min\",\n            \"value\": \"32\" },\n          { \"key\": \"mem_threshold\",\n            \"value\": \"200000\"}\n        ]\n      },\n      {\n        \"name\": \"com_blue_yonder_ThresholdQoSController\",\n        \"parameters\": [\n          { \"key\": \"load_threshold_5min\",\n            \"value\": \"60\" },\n          { \"key\": \"load_threshold_15min\",\n            \"value\": \"40\" },\n          { \"key\": \"mem_threshold\",\n            \"value\": \"230000\"}\n        ]\n      }\n    ]\n  }\n}'\n```\n\nMake sure to set the memory thresholds low enough so that the operating system can maintain\nsufficiently large file buffers and caches. This will also prevent the Linux OOM from being\ntriggered which could potentially kill a non-revocable task.\n\nBy default, a Mesos framework receives only non-revocable resources. An explicit opt-in is\nrequired to receive revocable resources as well. For example, for\n[oversubscription with Apache Aurora](https://github.com/apache/aurora/blob/master/docs/features/resource-isolation.md#oversubscription)\nthe following scheduler flags are required:\n\n```bash\n-receive_revocable_resources  # opt-in for revocable resource offers\n-enable_revocable_cpus        # schedule revocable jobs using revocable CPU resources\n-enable_revocable_ram         # schedule revocable jobs using revocable RAM resources\n```\n\n\nKnown Limitations\n-----------------\n\nThe main design goals of the threshold modules was a simple, stateless implementation.\nThis comes with a few limitations:\n\n* Being based on coarse-grained load and memory thresholds, the oversubscription technique described\n  here might not be suitable for aggressive oversubscription on systems with very latency sensitive\n  services. If you have such a requirement, either be very conservative or have a look at\n  [Intel/Mesosphere Serenity](https://github.com/mesosphere/serenity).\n\n* If you run your agents with `--cgroups_enable_cfs` you need at least a Linux kernel 3.8 or later.\n  Otherwise the system `load` used by estimator and QoS controller will not correctly account for\n  throttled processes. Further details can be found in this\n  [LWN article](https://lwn.net/Articles/531853/).\n\n* When the CPU is overloaded, a random revocable task is killed rather than the most aggressive\n  one.\n\nWe may feel compelled to address some of these limitations in the future.\nPull requests are welcome as well :-)\n\n\nLicense\n-------\n\nThis repository is licensed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblue-yonder%2Fmesos-threshold-oversubscription","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblue-yonder%2Fmesos-threshold-oversubscription","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblue-yonder%2Fmesos-threshold-oversubscription/lists"}