{"id":37705627,"url":"https://github.com/caffeinism/pfutils","last_synced_at":"2026-01-16T13:08:08.506Z","repository":{"id":54764184,"uuid":"520213183","full_name":"caffeinism/pfutils","owner":"caffeinism","description":"parallelize file commands","archived":false,"fork":false,"pushed_at":"2022-10-21T08:51:37.000Z","size":66,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-08-08T17:43:15.409Z","etag":null,"topics":["cephfs","nfs"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/caffeinism.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-01T18:01:58.000Z","updated_at":"2023-02-28T04:05:54.000Z","dependencies_parsed_at":"2022-08-14T02:10:15.644Z","dependency_job_id":null,"html_url":"https://github.com/caffeinism/pfutils","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/caffeinism/pfutils","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caffeinism%2Fpfutils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caffeinism%2Fpfutils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caffeinism%2Fpfutils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caffeinism%2Fpfutils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/caffeinism","download_url":"https://codeload.github.com/caffeinism/pfutils/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caffeinism%2Fpfutils/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28478922,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cephfs","nfs"],"created_at":"2026-01-16T13:08:08.450Z","updated_at":"2026-01-16T13:08:08.498Z","avatar_url":"https://github.com/caffeinism.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pfutils\n\nBasically, concurrency is not important when using Direct-attached storage(DAS). Reading blocks is a major bottleneck for DAS because metadata is very small and can be read immediately.\n\nBut what about network-attached storage (NAS)? To read metadata from a file system that satisfies concurrency, communication with the metadata server(MDS) is required. The MDS may need to be read by communicating with another NAS. In that case, a round trip time of several milliseconds may occur. This is part of an unnecessary bottleneck, as it introduces another overhead that DAS does not have.\n\nBasic file handling commands such as cp and rm in linux do not meet this requirement because they are made to target the traditional DAS. This application is designed to meet these needs.\n\n# Usage\n\n## Installation\n\nDepending on how python is installed, the command may be slightly different. You can usually install it with one of the commands below.\n\n```\n# If python3-pip is named \"pip3\"\n$ pip3 install pfutils\n# If pip is unnamed but there is pip module in \"python3\"\n$ python3 -m pip install pfutils\n# You can force the implementation of python if you wish\n$ pip3 install pfutils --no-binary=pfutils\n# If it is named without \"3\", remove \"3\" as appropriate...\n```\n\nFollowing the flow of the Python package installer, the pftuils command can be used as follows:\n\n```\n# If you have sudo privileges, the command will be installed in /usr/local/bin and\n# so on, and you can run it right away.\n$ pfutils --help\n\n# If you don't have sudo privileges, the commands will be installed in ~/.local/bin \n# and you need to add this path to $PATH so that it can be executed without path.\n\n# Regardless of the environment variable, if python3 is in the path, it can be \n# executed as follows.\n$ python3 -m pfutils --help\n```\n\n## Commands\n\n```\nUsage: pfutils cp [OPTIONS] SRC DST\n\n  copy files and directories\n\nOptions:\n  -j, --num-workers INTEGER  number of concurrent workers\n  -c, --chunksize INTEGER    size of chunk\n  -r, --recursive            copy directories recursively\n  --help                     Show this message and exit.\n\nUsage: pfutils rm [OPTIONS] [FILE]...\n\n  remove files or directories\n\nOptions:\n  -j, --num-workers INTEGER  number of concurrent workers\n  -r, --recursive            remove directories and their contents recursively\n  -c, --chunksize INTEGER    size of chunk\n  --help                     Show this message and exit.\n```\n\n# Implementation focus\n\nUnder Python's GIL constraints, using threads is known to improve IO performance, but when concurrency is actually granted using ThreadPoolExecutor, you can see that this is not really the case. So I used ProcessPoolExecutor in my implementation of Python.\n \nIn case of using the map method, since the block operation starts after reading all the metadata, operation loss occurs during that time. However, when the submit method is used, the performance is greatly degraded because chunks are not divided. (It is the same as when giving chunksize=1 in the map.) Therefore, I modified the ProcessPoolExecutor so that it can be executed in chunks to give concurrency to the task.\n\nI can give concurrency to reading metadata as well, but there doesn't seem to be a need for that operation to improve performance. However, its implementation may change in the future to work with more metadata.\n\nWhen implemented as a thread in C++ and bound to Python, there was a performance improvement under a small cpu resource due to the difference in overhead between process and thread. I used [https://github.com/bshoshany/thread-pool](https://github.com/bshoshany/thread-pool) for my module's C++ thread pool implementation. The detailed operation (count of progressbar, output of exception handle, etc.) does not completely match the implementation of python and c++, but the basic operation is designed so that there is no problem with each other. Users can force the installation of python by not using the wheel. (Refer to the installation above)\n\n# Performance Benchmark under cephfs\n\n## Cluster configuration\n\n```mermaid\n  graph TD;\n      Client--\u003eOSD_0;\n      Client--\u003eOSD_1;\n      Client--\u003eOSD_2;\n      Client--\u003eMDS;\n      MDS--\u003eOSD_0;\n      MDS--\u003eOSD_1;\n      MDS--\u003eOSD_2;\n```\n\nOSD is used to write blocks in cephfs and MDS is used to perform file system related operations. MDS also writes data into the OSD to safely manage the metadata it uses. I connected Client, OSD, and MDS through L2 switch of 10Gbps, and 3 OSD nodes were configured using ramdisk to sufficiently satisfy 10Gbps switch. No replication is configured.\n\n## Experiment\n\nWriting a large amount to a single file does not reveal the metadata overhead, which is a problem when using a network-based file system when only block IO is used. I will use as an example the copy and delete of imagenet, which may appear in general. In the experiment, I dropped the page cache before executing the instruction. Micron 7300 Pro 1.92TB as DAS and cephfs with 3 ramdisk nodes as NAS.\n\n## cp\n\n### DAS to DAS (different filesystems)\n\n```\n$ time cp  /host/imagenet-2012 /mnt/nvme/imagenet-2012 -r\nreal    7m50.016s\nuser    0m2.656s\nsys     3m3.991s\n\n# approximately 317.3 MiB/s\n```\n\n### DAS to DAS (same filesystem)\n\n```\n$ time cp /mnt/nvme/imagenet-2012/ /mnt/nvme/imagenet-2012-copy -r\n\nreal    6m29.872s\nuser    0m2.746s\nsys     2m47.706s\n\n# approximately 382.5 MiB/s\n```\n\n### DAS to NAS\n\n```\n$ time cp /mnt/nvme/imagenet-2012/ /mnt/cephfs/imagenet-2012 -r\n\nreal    9m47.546s\nuser    0m2.619s\nsys     2m19.726s\n\n# approximately 253.8 MiB/s\n```\n\n### NAS to NAS (same filesystem)\n\n```\n$ time cp /mnt/cephfs/imagenet-2012 /mnt/cephfs/imagenet-2012-copy -r\n\nreal    17m20.175s\nuser    0m3.249s\nsys     2m6.009s\n\n# Haha.. I couldn't finish it to the end due to the RAM limit of my computers\n# (I guess about 30 GiB of uncopied data is left) But you can see it's very\n# slow. Since writes are written to the page cache and writebacks but not \n# reads, sequentially reading from the NAS has a large overhead.\n```\n\n### NAS to DAS\n\n```\n$ time cp /mnt/cephfs/imagenet-2012/ /mnt/nvme/imagenet-2012-copy -r\n\nreal    12m52.895s\nuser    0m3.024s\nsys     2m58.706s\n\n# approximately 192.9 MiB/s\n```\n\n## pfutils cp -j20 -c5000 -r @ c++ thread pool\n\n### DAS to NAS\n\n```\n$ time pfutils cp -j20 -c5000 -r /raid/imagenet-2012/ /mnt/cephfs/imagenet-2012/\n100%|████████████████████████████████████| 267/267 [01:12\u003c00:00,  3.70it/s]\n\nreal    1m54.897s\nuser    0m29.947s\nsys     3m14.417s\n```\n\n## pfutils cp -j20 -c20000 -r @ python process pool\n\n### DAS to NAS\n\n```\n$ time pfutils cp -r -j20 -c20000 /mnt/nvme/imagenet-2012 /mnt/cephfs/imagenet-2012/\n100%|████████████████████████████| 1331167/1331167 [01:24\u003c00:00, 15689.53it/s]\n\nreal    2m22.602s\nuser    2m20.497s\nsys     5m11.242s\n\n# approximately 1046 MiB/s\n```\n\n### NAS to NAS (same filesystem)\n\n```\n$ time pfutils cp -r -j20 -c20000 /mnt/cephfs/imagenet-2012/ /mnt/cephfs/imagenet-2012-copy/\n68%|██████████████████████       | 900000/1331167 [06:27\u003c03:05, 2323.62it/s]\nreal    7m23.195s\nuser    2m27.837s\nsys     4m37.514s\n\n# It stops at the same point as normal cp, but a simple comparison will be possible.\n```\n\n### NAS to DAS\n\n```\n$ time pfutils cp -r -j20 -c20000 /mnt/cephfs/imagenet-2012/ /mnt/nvme/imagenet-2012-copy/\n100%||████████████████████████████| 1331167/1331167 [08:32\u003c00:00, 2596.60it/s]\n\nreal    9m18.036s\nuser    2m28.436s\nsys     4m43.750s\n\n# approximately 267 MiB/s\n```\n\n## rm\n\n### DAS\n\n```\n$ time rm /mnt/nvme/imagenet-2012-copy/ -r\n\nreal    0m48.581s\nuser    0m0.439s\nsys     0m13.494s\n```\n\n### NAS\n\n```\n$ time rm /mnt/cephfs/imagenet-2012/ -r\n\nreal    4m9.142s\nuser    0m0.689s\nsys     0m10.066s\n```\n\n## pfutils rm -j20 -c5000 -r @ c++ thread pool\n\n### NAS\n\n```\n$ time pfutils rm -j20 -c5000 -r /mnt/cephfs/imagenet-2012/ -y\nfile: 100%|██████████████████████████████████| 267/267 [00:05\u003c00:00, 48.71it/s]\ndirectory: 100%|████████████████████████| 2003/2003 [00:00\u003c00:00, 2273.27it/s]\n\nreal    1m37.519s\nuser    0m9.854s\nsys     0m16.984s\n```\n\n## pfutils rm -j20 -c20000 -r @ python process pool\n\n### NAS\n\n```\n$ time pfutils rm -j20 -c20000 -r -y /mnt/cephfs/imagenet-2012/\nfile: 100%|██████████████████████| 1331167/1331167 [00:16\u003c00:00, 78883.35it/s]\ndirectory: 100%|█████████████████████████| 2003/2003 [00:03\u003c00:00, 641.30it/s]\n\nreal    1m37.219s\nuser    0m31.170s\nsys     0m21.037s\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaffeinism%2Fpfutils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcaffeinism%2Fpfutils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaffeinism%2Fpfutils/lists"}