{"id":19756311,"url":"https://github.com/shibatch/oomstaller","last_synced_at":"2025-04-30T11:33:24.073Z","repository":{"id":260878733,"uuid":"864450295","full_name":"shibatch/oomstaller","owner":"shibatch","description":"A tool for suppressing swap thrashing at build time","archived":false,"fork":false,"pushed_at":"2025-03-26T16:17:23.000Z","size":115,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-06T01:41:16.537Z","etag":null,"topics":["build-tool","linux","make","memory","memory-management","ninja-build","oom","out-of-memory","outofmemory","parallel","ram","single-board-computer","single-board-computers","swapfile","task","thrashing"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shibatch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-28T08:50:43.000Z","updated_at":"2025-03-26T16:12:29.000Z","dependencies_parsed_at":"2024-11-03T10:16:26.238Z","dependency_job_id":"0fc05fa8-912e-425a-8af7-9d9dec3713cb","html_url":"https://github.com/shibatch/oomstaller","commit_stats":null,"previous_names":["shibatch/oomstaller"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibatch%2Foomstaller","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibatch%2Foomstaller/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibatch%2Foomstaller/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibatch%2Foomstaller/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shibatch","download_url":"https://codeload.github.com/shibatch/oomstaller/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251691610,"owners_count":21628356,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["build-tool","linux","make","memory","memory-management","ninja-build","oom","out-of-memory","outofmemory","parallel","ram","single-board-computer","single-board-computers","swapfile","task","thrashing"],"created_at":"2024-11-12T03:15:32.672Z","updated_at":"2025-04-30T11:33:24.062Z","avatar_url":"https://github.com/shibatch.png","language":"C++","readme":"## oomstaller - A tool for suppressing swap thrashing at build time\n\nOriginal distribution site : https://github.com/shibatch/oomstaller\n\n\n### Introduction\n\nModern computers have many CPU cores, but in order to use them\neffectively, a lot of memory is sometimes required. When building a\nlarge application, the build time can become very long due to lack of\nmemory. Usually, most processes in the build do not require much\nmemory, but a very small portion of the build requires a large amount\nof memory. If swapping occurs when performing those processes, the\nbuild will take an extremely long time to complete. This problem can\nbe avoided by setting the number of CPU cores used for builds to a\nsmall number, but this is a waste of valuable CPU. Also, there is no\ngood way to know the best number of CPU cores to use beforehand.\n\nThis tool monitors the memory usage of each process when performing a\nbuild, and suspends processes as necessary to prevent swap thrashing\nfrom occurring. This allows you to build using all CPU cores without\nworrying about swap thrashing.\n\n\n#### How it works\n\nThrashing is a situation in which a CPU takes longer to perform\nswapping than the process that the CPU would normally perform if there\nwere sufficient memory. When thrashing is occurring, it takes longer\nto respond to user input because the memory for not only the process\ntaking up the most memory but also the memory for other processes is\nswapped out. In addition, the OS may kill the process that is\noccupying the largest amount of memory, which can make the system\nunstable.\n\nWhen thrashing occurs while multiple processes are running in\nparallel, the OS may swap out the memory of one running process to\nfree up the physical memory needed for the other processes to continue\nexecution. However, even after swapping out the memory of the running\nprocess, the swapped-out memory is soon needed again to continue\nexecution of that process, resulting in a state of frequent\nswap-in/swap-out.\n\nTo avoid this situation, this tool suspends part of processes and\nstops their execution completely. This ensures that once the memory\nfor a process is swapped out, it will not be swapped in again until\nits execution is resumed, and processes that are still running can\ncontinue their execution with the necessary physical memory allocated.\nBecause the advantage of not having to swap frequently is much greater\nthan the disadvantage of some CPU cores not being utilized, this tool\ncan reduce the total execution time compared to when thrashing is\noccurring.\n\nWhen selecting which processes to suspend and which to execute,\npriority is given first to the execution of the process occupying the\nlargest amount of memory. The process occupying the largest amount of\nmemory will not be suspended under any circumstances. The remaining\nrunning processes are selected based on the time when the process\nstarted running. The earlier a process is started, the more priority\nit will be given to execution. This avoids the same process being\nsuspended and resumed repeatedly, disrupting the build order, or\nsuspending processes that are crucial to the build.\n\nThe target processes to be suspended are selected by referencing the\nparent-child relationship of the processes. Only processes that are\ndescendants of the command invoked as an argument of oomstaller will\nbe suspended, thus processes that are not related to the build will\nnever be suspended. It does not require information about which\nprocesses are involved in the build, and can reliably control only the\nprocesses involved in the build.\n\n\n### How to build\n\n1. Check out the source code from our GitHub repository :\n`git clone https://github.com/shibatch/oomstaller`\n\n2. Run make :\n`cd oomstaller \u0026\u0026 make`\n\n\n### Synopsis\n\n`oomstaller [\u003coptions\u003e] command [arg] ...`\n\n\n### Description\n\nThis tool monitors the memory usage of each process when performing a\nbuild, and suspends processes as necessary to prevent swapping from\noccurring.\n\nTo perform a build using this tool, specify make or ninja as the\nargument of this tool and execute as follows.\n\n```\noomstaller make -j `nproc`\n```\n\n\n### Options\n\n`--max-parallel \u003cnumber of processes\u003e`         default:   0\n\nSuspends processes so that the number of running build processes does\nnot exceed the specified number. 0 means no limit. A process is\ncounted as one process even if it has multiple threads.\n\n`--max-parallel-thrash \u003cnumber of processes\u003e`  default:   1\n\nSpecifies the maximum number of processes to run when thrashing is\ndetected. 0 means no limit.\n\n`--period \u003cseconds\u003e`                           default:   1.0\n\nSpecifies the interval at which memory usage of each process is checked\nand processes are controlled.\n\n`--show-stat`\n\nDisplays statistics when finished.\n\n\n### Tips\n\nAlthough oomstaller is designed primarily to suppress swapping during\nbuilds by suspending the build process, it can also be configured to\nallow swapping while attempting to reduce the build time. The number\nof processes executed when thrashing is detected can be specified with\n--max-parallel-thrash option. Even when this number of processes is\nincreased, the build processes are suspended as necessary depending on\nmemory usage. By increasing this number, it may be possible to reduce\nbuild time by actively swapping processes. However, the actual effect\ndepends greatly on the speed of swap space, and in most cases the\noverhead of swapping outweighs the benefit of processing with more\nthreads. Furthermore, increasing this number of processes may require\na large swap space.\n\nWe recommend specifying \"-j \\`nproc\\`\" option to ninja. ninja usually\nruns jobs with more threads than CPU cores. This is effective to\nreduce build time if there is sufficient memory. However, this will\nonly consume extra memory in situations where there is not enough\nmemory in which you might want to use this tool.\n\nBefore benchmarking, run \"sudo swapoff -a; sudo swapon -a\" to reset\nthe condition of the swap space.\n\nIf you kill this tool with SIGKILL, a large number of build processes\nwill remain suspended with SIGSTOP. To prevent this from happening,\nuse SIGTERM or SIGINT to kill this tool. You can send SIGCONT to all\nprocesses run by you with the following command.\n\n```\nkillall -v -s CONT -u $USER -r '.*'\n```\n\n\n### License\n\nThe software is distributed under the Boost Software License, Version 1.0.\nSee accompanying file LICENSE.txt or copy at :\nhttp://www.boost.org/LICENSE_1_0.txt.\n\nContributions to this project are accepted under the same license.\n\nThe fact that this software is released under an open source license\nonly means that you can use the current version of the software for\nfree. If you want this software to be maintained, you need to\nfinancially support the project. Please see\n[CODE_OF_CONDUCT.md](https://github.com/shibatch/nofreelunch?tab=coc-ov-file).\n\nCopyright Naoki Shibata 2024.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshibatch%2Foomstaller","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshibatch%2Foomstaller","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshibatch%2Foomstaller/lists"}