{"id":20199556,"url":"https://github.com/max0x7ba/thp-usage","last_synced_at":"2025-04-10T11:08:55.082Z","repository":{"id":140636454,"uuid":"392667848","full_name":"max0x7ba/thp-usage","owner":"max0x7ba","description":"A script to show processes that use transparent huge pages on Linux.","archived":false,"fork":false,"pushed_at":"2024-07-13T17:08:14.000Z","size":10,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-24T09:52:53.320Z","etag":null,"topics":["linux","malloc","memory-allocator","performance","python","thp","transparent-huge-pages"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/max0x7ba.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-04T11:46:22.000Z","updated_at":"2025-02-03T09:46:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"e2c27270-0bc7-4a8f-9e95-5d97268db63c","html_url":"https://github.com/max0x7ba/thp-usage","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/max0x7ba%2Fthp-usage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/max0x7ba%2Fthp-usage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/max0x7ba%2Fthp-usage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/max0x7ba%2Fthp-usage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/max0x7ba","download_url":"https://codeload.github.com/max0x7ba/thp-usage/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248208550,"owners_count":21065202,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["linux","malloc","memory-allocator","performance","python","thp","transparent-huge-pages"],"created_at":"2024-11-14T04:38:04.910Z","updated_at":"2025-04-10T11:08:55.058Z","avatar_url":"https://github.com/max0x7ba.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# thp-usage\nA script to show processes that use transparent huge pages on Linux.\n\n## Setup\n### Enable transparent huge pages on your system\n```\n$ sudo ./install-thp-always.sh\n```\n\n### Make Python always use transparent huge pages for memory allocations by using huge page aware [tcmalloc][1]\n```\n$ sudo apt-get --yes install libgoogle-perftools-dev patchelf\n$ sudo cp --preserve=all $(readlink -f $(which python)){,.$(date +%Y%m%dT%H%M%S)~}\n$ sudo patchelf --add-needed /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so $(readlink -f $(which python))\n```\n\nMy use case is machine learning using Ray Tune and PyTorch on CPU, and above change results in 5-15% faster machine learning with no code changes. Your results may differ, benchmark your application before and after applying the above change.\n\n## Example output\n\n```\n$ sudo ./thp-usage.py\n    pid\t   pages\t          MB\tcmdline\n 712676\t   1,268\t       2,536\tray::ImplicitFunc.train_buffered()\n 711989\t   1,070\t       2,140\tray::ImplicitFunc.train_buffered()\n 714214\t   1,068\t       2,136\tray::ImplicitFunc.train_buffered()\n 712422\t   1,064\t       2,128\tray::ImplicitFunc.train_buffered()\n 713737\t   1,064\t       2,128\tray::ImplicitFunc.train_buffered()\n 711327\t     864\t       1,728\tray::ImplicitFunc.train_buffered()\n 713076\t     863\t       1,726\tray::ImplicitFunc.train_buffered()\n 713122\t     863\t       1,726\tray::ImplicitFunc.train_buffered()\n 713498\t     863\t       1,726\tray::ImplicitFunc.train_buffered()\n 714082\t     863\t       1,726\tray::ImplicitFunc.train_buffered()\n 714235\t     863\t       1,726\tray::ImplicitFunc.train_buffered()\n 712584\t     862\t       1,724\tray::ImplicitFunc.train_buffered()\n 713152\t     862\t       1,724\tray::ImplicitFunc.train_buffered()\n 710998\t     861\t       1,722\tray::ImplicitFunc.train_buffered()\n 712577\t     861\t       1,722\tray::ImplicitFunc.train_buffered()\n 711860\t     860\t       1,720\tray::ImplicitFunc.train_buffered()\n 712579\t     860\t       1,720\tray::ImplicitFunc.train_buffered()\n 713716\t     858\t       1,716\tray::ImplicitFunc.train_buffered()\n 712313\t     857\t       1,714\tray::ImplicitFunc.train_buffered()\n 711680\t     854\t       1,708\tray::ImplicitFunc.train_buffered()\n 713125\t     811\t       1,622\tray::ImplicitFunc.train_buffered()\n 714329\t     663\t       1,326\tray::ImplicitFunc.train_buffered()\n 712401\t     661\t       1,322\tray::ImplicitFunc.train_buffered()\n 713097\t     659\t       1,318\tray::ImplicitFunc.train_buffered()\n 710780\t     635\t       1,270\tray::ImplicitFunc.train_buffered()\n 711244\t     568\t       1,136\tray::ImplicitFunc.train_buffered()\n 713823\t     460\t         920\tray::ImplicitFunc.train_buffered()\n 711992\t     459\t         918\tray::ImplicitFunc.train_buffered()\n 712697\t     360\t         720\tray::ImplicitFunc.train_buffered()\n 713819\t     228\t         456\tray::ImplicitFunc.train_buffered()\n   3412\t     197\t         394\t/usr/bin/influxd\n 712180\t     192\t         384\tray::ImplicitFunc.train_buffered()\n 712712\t     192\t         384\tray::ImplicitFunc.train_buffered()\n 670968\t     148\t         296\tpython3 -m es.tune4 --samples 3000\n   4128\t      54\t         108\t/usr/bin/plasmashell\n   4026\t      35\t          70\t/usr/bin/kwin_x11\n 670976\t      22\t          44\t/home/max/anaconda3/envs/torch/lib/python3.8/site-packages/ray/core/src/ray/thirdparty/redis/src/redis-server ...\n 670981\t      19\t          38\t/home/max/anaconda3/envs/torch/lib/python3.8/site-packages/ray/core/src/ray/thirdparty/redis/src/redis-server ...\n 670967\t      18\t          36\txz --best --stdout\n   2219\t      16\t          32\t/usr/lib/xorg/Xorg -nolisten tcp ...\n 671000\t       7\t          14\t/home/max/anaconda3/envs/torch/bin/python3 ...\n 671065\t       6\t          12\t/home/max/anaconda3/envs/torch/bin/python3 ...\n 671027\t       5\t          10\t/home/max/anaconda3/envs/torch/bin/python3 ...\n 670988\t       4\t           8\t/home/max/anaconda3/envs/torch/bin/python3 ...\n   1335\t       3\t           6\t/opt/piavpn/bin/pia-daemon\n   4311\t       3\t           6\t/usr/bin/konsole\n 670986\t       2\t           4\t/home/max/anaconda3/envs/torch/lib/python3.8/site-packages/ray/core/src/ray/gcs/gcs_server ...\n   1261\t       1\t           2\t/usr/sbin/rsyslogd -n -iNONE\n   1298\t       1\t           2\t/usr/lib/udisks2/udisksd\n   4108\t       1\t           2\t/usr/lib/x86_64-linux-gnu/libexec/kactivitymanagerd\n   4136\t       1\t           2\t/usr/bin/xembedsniproxy\n   4168\t       1\t           2\t/usr/lib/x86_64-linux-gnu/libexec/DiscoverNotifier\n   4186\t       1\t           2\t/usr/libexec/at-spi-bus-launcher --launch-immediately\n 668856\t       1\t           2\t/usr/libexec/xdg-desktop-portal\n 671026\t       1\t           2\t/home/max/anaconda3/envs/torch/lib/python3.8/site-packages/ray/core/src/ray/raylet/raylet ...\n 704865\t       1\t           2\temacs\n      0\t  24,884\t      49,768\t\u003ctotal\u003e\n```\n\nTo find out what huges pages are used for in a process, the following command can be used:\n\n```\n$ grep -A10 -B13 \"^AnonHugePages:[[:space:]]*[1-9]\" /proc/670988/smaps\n55c49328c000-55c493f05000 rw-p 00000000 00:00 0                          [heap]\nSize:              12772 kB\nKernelPageSize:        4 kB\nMMUPageSize:           4 kB\nRss:               11508 kB\nPss:               11508 kB\nShared_Clean:          0 kB\nShared_Dirty:          0 kB\nPrivate_Clean:         0 kB\nPrivate_Dirty:     11508 kB\nReferenced:        11508 kB\nAnonymous:         11508 kB\nLazyFree:              0 kB\nAnonHugePages:      8192 kB\nShmemPmdMapped:        0 kB\nFilePmdMapped:         0 kB\nShared_Hugetlb:        0 kB\nPrivate_Hugetlb:       0 kB\nSwap:                  0 kB\nSwapPss:               0 kB\nLocked:                0 kB\nTHPeligible:    1\nProtectionKey:         0\nVmFlags: rd wr mr mw me ac sd\n```\n\nIn the above command, `-A` and `-B` parameters match `smaps` format of Linux-5.10. If the format is different on your system, you may like to adjust these parameters appropriately.\n\n---\n\nCopyright (c) 2021 Maxim Egorushkin. MIT License. See the full licence in file LICENSE.\n\n[1]: https://google.github.io/tcmalloc/temeraire.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmax0x7ba%2Fthp-usage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmax0x7ba%2Fthp-usage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmax0x7ba%2Fthp-usage/lists"}