{"id":17991717,"url":"https://github.com/hakavlad/file-starve","last_synced_at":"2025-04-04T04:16:42.045Z","repository":{"id":145292783,"uuid":"356518180","full_name":"hakavlad/file-starve","owner":"hakavlad","description":"Explore the impact of file LRU lists restriction","archived":false,"fork":false,"pushed_at":"2022-01-01T05:04:07.000Z","size":54,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-09T15:41:30.747Z","etag":null,"topics":["oom","paging","thrashing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hakavlad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-10T08:20:54.000Z","updated_at":"2022-09-15T08:59:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"c1f17d3e-4228-4d0b-afa7-aba29cd95c2a","html_url":"https://github.com/hakavlad/file-starve","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hakavlad%2Ffile-starve","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hakavlad%2Ffile-starve/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hakavlad%2Ffile-starve/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hakavlad%2Ffile-starve/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hakavlad","download_url":"https://codeload.github.com/hakavlad/file-starve/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247117762,"owners_count":20886439,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["oom","paging","thrashing"],"created_at":"2024-10-29T19:23:36.328Z","updated_at":"2025-04-04T04:16:42.022Z","avatar_url":"https://github.com/hakavlad.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# file-starve\n\n[![Total alerts](https://img.shields.io/lgtm/alerts/g/hakavlad/file-starve.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/hakavlad/file-starve/alerts/)\n\nExplore the impact of reducing [page cache](https://www.kernel.org/doc/html/latest/admin-guide/mm/concepts.html#page-cache) size: from moderate [thrashing](https://en.wikipedia.org/wiki/Thrashing_(computer_science)) to a complete UI freeze. This script tries to keep a specified size of file LRU lists (20 MiB by default) within a specified time (60s by default).\n\n## Background\n\nLet's talk about the elephant in the room - the Linux kernel's inability to gracefully handle low memory pressure:\n- https://lore.kernel.org/lkml/d9802b6a-949b-b327-c4a6-3dbca485ec20@gmx.com/\n\n\u003e...in normal operation you will have nearly all of your executables nad libraries sitting in good ol' physical RAM. But when RAM runs low, but not low enough for the out-of-memory killer to be run, these pages are evicted from RAM. So you end up with a situation where pages are evicted -- at first, no problem, because they are evicted least-recently-used first and it kicks out pages you aren't using anyway. But then, it kicks out the ones you are using, just to have to page them right back in moments later. Thrash city.\n\n-- https://serverfault.com/a/319818\n\n## Options\n\n```\n$ file-starve -h\nusage: file-starve [-h] [-t TARGET] [-T TIMEOUT] [-s SCALE] [-c CHUNK] [-i INTERVAL] [-d DEV]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -t TARGET, --target TARGET\n                        target size of file LRU lists in MiB\n  -T TIMEOUT, --timeout TIMEOUT\n                        timeout in seconds after reaching the target\n  -s SCALE, --scale SCALE\n                        MemFree scale factor\n  -c CHUNK, --chunk CHUNK\n                        chunk size in KiB\n  -i INTERVAL, --interval INTERVAL\n                        output interval in seconds\n  -d DEV, --dev DEV     device name to track I/O stats\n```\n\n## Usage\n\nJust run the script. Output example:\n```\n$ file-starve\nstarting file-starve\nW: device name to track I/O stats is not set\nprocess memory locked with MCL_CURRENT | MCL_ONFAULT\n  target file LRU lists size: 20.0M\n  keep starved (timeout):     60.0s\n  MemFree scale factor:       2.0\n  chunk size:                 64K\ntrying to reach the target...\ntarget reached in 6.6s\n  file: 19.8M, free: 65.4M, passed: 0.0s\n  file: 18.3M, free: 129.7M, passed: 10.0s\n  file: 19.1M, free: 129.3M, passed: 20.0s\n  file: 19.6M, free: 129.2M, passed: 30.0s\n  file: 19.8M, free: 129.2M, passed: 40.0s\n  file: 16.9M, free: 116.9M, passed: 50.0s\n  file: 20.3M, free: 111.2M, passed: 60.0s\nkept starved during 60.0s\nfile lists and free memory sizes after reaching the target:\n  file: min=13.4M, max=21.3M, average=18.2M\n  free: min=64.6M, max=131.8M, average=126.7M\nvmstat I/O metrics for the last 60.0s:\n  pgpgin:  559.9M, 9.3M/s\n  pgpgout: 0.0M, 0.0M/s\n  pswpin:  0.0M, 0.0M/s\n  pswpout: 0.0M, 0.0M/s\nPSI metrics for the last 60.0s:\n  some io:     55.3s, 92.2%\n  full io:     49.5s, 82.5%\n  some memory: 27.7s, 46.1%\n  full memory: 26.4s, 44.0%\n```\n\nOptionally, you can specify a device name for tracking I/O stats:\n```\n$ file-starve -t 40 -T 30 -i 15 -c 32 -d sdb -s 1.5\nstarting file-starve\ndevice name to track I/O stats: sdb\nprocess memory locked with MCL_CURRENT | MCL_ONFAULT\n  target file LRU lists size: 40.0M\n  keep starved (timeout):     30.0s\n  MemFree scale factor:       1.5\n  chunk size:                 32K\ntrying to reach the target...\ntarget reached in 8.8s\n  file: 39.9M, free: 54.8M, passed: 0.0s\n  file: 22.4M, free: 77.1M, passed: 15.0s\n  file: 28.1M, free: 74.8M, passed: 30.0s\n  file: 28.1M, free: 74.8M, passed: 30.0s\nkept starved during 30.0s\nfile lists and free memory sizes after reaching the target:\n  file: min=11.7M, max=40.7M, average=31.5M\n  free: min=49.7M, max=86.2M, average=76.3M\nvmstat I/O metrics for the last 30.0s:\n  pgpgin:  264.6M, 8.8M/s\n  pgpgout: 0.0M, 0.0M/s\n  pswpin:  0.0M, 0.0M/s\n  pswpout: 0.0M, 0.0M/s\nPSI metrics for the last 30.0s:\n  some io:     25.3s, 84.2%\n  full io:     24.3s, 81.1%\n  some memory: 13.9s, 46.3%\n  full memory: 13.6s, 45.2%\nI/O statistics for sdb device in the last 30.0s:\n  util: 29.8s, 99.2%\n  read: 263.1M, 8.8M/s\n  read: 6080 requests processed, 202.7 rp/s\n  read: await 946.7s, 3155.6%\n  write: 0.0M, 0.0M/s\n  write: 11 requests processed, 0.4 rp/s\n  write: await 0.7s, 2.2%\n```\n\n## Warnings\n\n- The UI may not respond while the script is running.\n- In some cases, the UI may not respond even after the specified timeout has expired.\n\n## Requirements\n\n- Python \u003e= 3.3\n\n## Installation\n\nInstall\n```bash\n$ git clone https://github.com/hakavlad/file-starve.git \u0026\u0026 cd file-starve\n$ sudo make install\n```\n\nUninstall\n```bash\n$ sudo make uninstall\n```\n\n## See also\n\n- Lock executables and shared libraries in memory to improve system responsiveness under low-memory conditions\n    - https://github.com/hakavlad/prelockd\n- [PATCH] mm: Protect the working set under memory pressure to prevent thrashing, avoid high latency and prevent livelock in near-OOM conditions\n    - https://github.com/hakavlad/le9-patch\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhakavlad%2Ffile-starve","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhakavlad%2Ffile-starve","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhakavlad%2Ffile-starve/lists"}