https://github.com/spcl/nopfs
Near-optimal Prefetching System
https://github.com/spcl/nopfs
Last synced: 9 months ago
JSON representation
Near-optimal Prefetching System
- Host: GitHub
- URL: https://github.com/spcl/nopfs
- Owner: spcl
- Created: 2021-06-28T16:03:56.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2021-11-17T17:53:19.000Z (about 4 years ago)
- Last Synced: 2025-02-10T00:57:55.611Z (11 months ago)
- Size: 61.5 KB
- Stars: 33
- Watchers: 18
- Forks: 6
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# NoPFS
NoPFS is the Near-optimal Prefetching System, an I/O middleware that optimizes distributed machine learning I/O at scale.
For more details, check our paper from Supercomputing 2021: [Clairvoyant Prefetching for Distributed Machine Learning I/O](https://arxiv.org/abs/2101.08734).
```
@inproceedings{dryden2021clairvoyant,
title={CLairvoyant Prefetching for Distributed Machine Learning {I/O}},
author={Nikoli Dryden and Roman B\"{o}hringer and Tal Ben-Nun and Torsten Hoefler},
booktitle={Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC)},
year={2021},
eprint={2101.08734},
archivePrefix={arXiv},
primaryClass={cs.DC}
}
```
----
We are currently preparing a more "production-ready" release, which will be out soon! If you are interested in the original "research" code, look at the `old-nopfs` branch.