https://github.com/cloneofsimo/ptar
https://github.com/cloneofsimo/ptar
Last synced: about 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/cloneofsimo/ptar
- Owner: cloneofsimo
- Created: 2024-06-03T08:06:18.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-03T13:27:16.000Z (about 2 years ago)
- Last Synced: 2024-06-03T15:32:57.344Z (about 2 years ago)
- Language: C++
- Size: 6.84 KB
- Stars: 4
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# ptar: Parallel Tar Reader for Effecient Pytorch DataLoader
TBD
`ptar` is a Python package that provides a parallel tar file reader implemented using C++ and `pybind11`. This package allows efficient reading of multiple tar files containing specific file types in parallel using multiple threads that prefetch data in non-blocking fashion, just like how it is done in PyTorch's `DataLoader`. However, it is done with threads and in c++, that speeds up the process significantly.
# Usage
```python
from ptar import Ptar
shardlist = [f"shard{i}.tar.gz" for i in range(5)]
# tar has 3 files with extensions .txt, .py, .cpp, in order.
schema = ['.txt', '.py', '.cpp']
reader = Ptar(shardlist, schema)
all_data = []
for i in range(5):
data = reader.get_next()
print(data)
# this will return all 3 of them.
# Make the reader iterate from beginning.
reader.reset()
# return all 15 of them all at once, as list of lists.
data = reader.get_next_n(5)
```
## Features.
- **Parallel Reading**: Utilizes multiple threads to read from a list of tar files in parallel.
- **File Type Filtering**: Supports filtering by file extensions (e.g., `.txt`, `.py`, `.cpp`).
- **Python Interface**: Provides a Python interface for ease of use.
## Installation
### Prerequisites
Ensure you have the following dependencies installed:
- Python 3.9
- CMake 3.12 or higher
- `libarchive` development libraries
- `pybind11`
You can install the necessary system packages using:
```sh
sudo apt-get update
sudo apt-get install python3.9 python3.9-dev cmake libarchive-dev
pip install pybind11 setuptools
```
### Virtual Environment Setup
1. **Create and activate a virtual environment**:
```sh
python3 -m venv myenv
source myenv/bin/activate
```
2. **Ensure Python development headers are accessible**:
```sh
mkdir -p myenv/include/python3.9
ln -s /usr/include/python3.9/* myenv/include/python3.9/
mkdir -p myenv/lib
ln -s /usr/lib/x86_64-linux-gnu/libpython3.9.so myenv/lib/
```
### Building and Installing `ptar`
1. **Clone the repository**:
```sh
git clone https://github.com/cloneofsimo/ptar.git
cd ptar
```
2. **Install the package**:
```sh
python setup.py install
```