https://github.com/wtsi-hgi/mpistat
Efficiently lstat every file in a filesystem tree in parallel across multiple nodes using MPI.
https://github.com/wtsi-hgi/mpistat
Last synced: 23 days ago
JSON representation
Efficiently lstat every file in a filesystem tree in parallel across multiple nodes using MPI.
- Host: GitHub
- URL: https://github.com/wtsi-hgi/mpistat
- Owner: wtsi-hgi
- Created: 2014-11-07T13:24:02.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2019-04-10T13:44:06.000Z (about 6 years ago)
- Last Synced: 2025-04-15T05:08:34.394Z (23 days ago)
- Language: Python
- Homepage:
- Size: 70.3 KB
- Stars: 3
- Watchers: 19
- Forks: 4
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
mpistat
=======
Efficiently walk a tree in parallel across many nodes using MPI. Based on http://conferences.computer.org/sc/2012/papers/1000a015.pdf. Also see http://jlafon.io/parallel-file-treewalk.html.
There is a C version using libcircle (https://github.com/hpc/libcircle) which is the C implementation used by the authors of the original article.
The python version uses Guy Coates's interpretation (https://github.com/wtsi-ssg/pwalk) of the algorithm discussed in the paper.
There is no significant difference in the time taken by the C and python versions due to the time being dominated by the lstat syscall.