https://github.com/facebookresearch/spdl

Scalable and Performant Data Loading
https://github.com/facebookresearch/spdl

dl ml

Last synced: 5 months ago
JSON representation

Scalable and Performant Data Loading

• Host: GitHub
• URL: https://github.com/facebookresearch/spdl
• Owner: facebookresearch
• License: bsd-2-clause
• Created: 2024-06-27T19:57:35.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2025-05-14T01:55:54.000Z (about 1 year ago)
• Last Synced: 2025-05-14T03:28:51.322Z (about 1 year ago)
• Topics: dl, ml
• Language: Python
• Homepage: https://facebookresearch.github.io/spdl/main/
• Size: 12.7 MB
• Stars: 258
• Watchers: 10
• Forks: 13
• Open Issues: 4
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md
  • Citation: CITATION.cff
  • Authors: AUTHORS

Awesome Lists containing this project

README

          
# SPDL

SPDL (Scalable and Performant Data Loading) is a library and project to

explore the design of performant data loading.

It provides flexible pipeline abstraction and a set of operations used

for processing array data.

## Documentation

Please checkout [the documentation](https://facebookresearch.github.io/spdl).

## License

SPDL is BSD 2-Clause licensed, as found in the LICENSE file.

## Citation

Please use the following BibTex for citing our project if you find it useful.

```

@misc{hira2025scalableperformantdataloading,

   title={Scalable and Performant Data Loading},

   author={Moto Hira and Christian Puhrsch and Valentin Andrei and Roman Malinovskyy and Gael Le Lan and Abhinandan Krishnan and Joseph Cummings and Miguel Martin and Gokul Gunasekaran and Yuta Inoue and Alex J Turner and Raghuraman Krishnamoorthi},

   year={2025},

   eprint={2504.20067},

   archivePrefix={arXiv},

   primaryClass={cs.DC},

   url={https://arxiv.org/abs/2504.20067},

}

```