https://github.com/ornl/ddstore

Enabling efficient distributed data loading for distributed data parallelism
https://github.com/ornl/ddstore

Last synced: about 2 months ago
JSON representation

Enabling efficient distributed data loading for distributed data parallelism

• Host: GitHub
• URL: https://github.com/ornl/ddstore
• Owner: ORNL
• License: bsd-3-clause
• Created: 2023-10-30T18:54:36.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2025-03-05T23:26:59.000Z (3 months ago)
• Last Synced: 2025-03-24T20:23:06.021Z (2 months ago)
• Language: Python
• Size: 97.7 KB
• Stars: 2
• Watchers: 6
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# DDStore

Enabling efficient distributed data loading for distributed data parallelism



## Installation

```

pip install numpy mpi4py Cython # install pre-requisites

# for use with PYTHONPATH=$PWD:$PYTHONPATH

CC=mpicc CXX=mpicxx python setup.py build_ext --inplace

# or, for installing into your virtual environment using pip

CC=mpicc CXX=mpicxx pip install .

# or, as above but in development mode

CC=mpicc CXX=mpicxx pip install -e .

```

## Test

```

mpirun -n 4 python -m mpi4py test/demo.py

```