{"id":30749845,"url":"https://github.com/helmholtz-analytics/hpc_pytorch_loader","last_synced_at":"2025-10-11T06:39:21.409Z","repository":{"id":312099592,"uuid":"1046300872","full_name":"helmholtz-analytics/hpc_pytorch_loader","owner":"helmholtz-analytics","description":"Efficient dataset storage in Python using memory-mapped arrays, Parquet, PyArrow IPC, and HDF5 to reduce inode usage","archived":false,"fork":false,"pushed_at":"2025-09-28T18:39:39.000Z","size":43,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-28T20:40:48.596Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/helmholtz-analytics.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-28T13:38:54.000Z","updated_at":"2025-09-28T18:39:42.000Z","dependencies_parsed_at":"2025-08-28T21:06:02.144Z","dependency_job_id":"13f7ce7b-cbf4-4edb-9c6b-f646d439ad28","html_url":"https://github.com/helmholtz-analytics/hpc_pytorch_loader","commit_stats":null,"previous_names":["helmholtz-analytics/hpc_pytorch_loader"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/helmholtz-analytics/hpc_pytorch_loader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helmholtz-analytics%2Fhpc_pytorch_loader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helmholtz-analytics%2Fhpc_pytorch_loader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helmholtz-analytics%2Fhpc_pytorch_loader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helmholtz-analytics%2Fhpc_pytorch_loader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/helmholtz-analytics","download_url":"https://codeload.github.com/helmholtz-analytics/hpc_pytorch_loader/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helmholtz-analytics%2Fhpc_pytorch_loader/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279006451,"owners_count":26084107,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-04T06:47:06.884Z","updated_at":"2025-10-11T06:39:21.403Z","avatar_url":"https://github.com/helmholtz-analytics.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dataset Conversion Library\n\nThis library provides tools for efficient conversion and reading of large image datasets. It supports multiple formats (e.g., Memmap, HDF5, Parquet) and is designed for both fixed-size and flexible-size datasets. With distributed processing capabilities, it ensures fast and scalable dataset preparation for machine learning workflows.\n\n---\n\n## Features\n\n- **Multi-format Support**:\n  - **Fixed-size**: Memory-mapped (Memmap), HDF5\n  - **Flexible-size**: Parquet, Tar, PyArrow IPC (Inter-Process Communication)\n- **Dataset Compatibility**:\n  - Supports conversion from image folders (e.g., ImageNet1k structure) or any `torch.Dataset`.\n- **Distributed Processing**:\n  - Accelerates dataset conversion using distributed processing.\n- **Custom Data Loaders**:\n  - Efficient reading of converted datasets with PyTorch-compatible loaders.\n- **Image Format Support**:\n  - Handles various image formats and modes (e.g., RGB, grayscale).\n\n---\n\n## Main Components\n\n1. **Converters**:\n   - `MemmapConverter`, `HDF5Converter`, `ParquetConverter`, `IpcConverter`, `TarConverter`\n2. **Readers**:\n   - `MemmapReader`, `HDF5Reader`, `ParquetReader`, `IpcReader`, `TarReader`\n3. **Utilities**:\n   - Custom samplers, preprocessing tools, and distributed processing decorators.\n\n---\n\n## Installation\n\n### Local Setup\n\n1. Ensure Python 3 is installed on your system.\n2. Clone this repository:\n   ```bash\n   git clone https://github.com/helmholtz-analytics/hpc_pytorch_loader.git\n   cd hpc_pytorch_loader\n   ```\n3. Run the setup script:\n   ```bash\n   bash setup_local.sh\n   ```\n   This will:\n   - Create a virtual environment (`dataloadenv`).\n   - Install required dependencies (e.g., PyTorch, torchvision, tqdm, etc.).\n\n4. Activate the virtual environment:\n   ```bash\n   source dataloadenv/bin/activate\n   ```\n\n### Cluster Setup\n\n1. Connect to your cluster and navigate to the project directory.\n2. Run the cluster setup script:\n   ```bash\n   source setup_cluster.sh\n   ```\n   This will:\n   - Load necessary modules (e.g., PyTorch, CUDA).\n   - Create and activate a virtual environment.\n   - Install required dependencies.\n\n**Note**: The `setup_cluster.sh` script uses the `ml` command to load modules. Adjust it based on your cluster's configuration.\n\n---\n\n## Usage\nYou can provide either:\n### Folder Structure\n\nThe input dataset must be organized into a root directory with one subfolder per class. Each subfolder contains the images belonging to that class. Example:\n\n```python\nroot/\n  ├── class1/\n  │     ├── img1.jpg\n  │     ├── img2.jpg\n  ├── class2/\n  │     ├── img3.jpg\n  │     ├── img4.jpg\n```\n\n- The folder name is used as the class label.\n\n- Supported image formats: .jpg, .png, etc.\n\n### Pytorch Dataset\nThe library also supports input as a PyTorch torch.utils.data.Dataset object.\nHowever, when using a FixedSizeConverter (e.g., MemmapConverter, HDF5Converter), preprocessing is required to ensure that all images have the same size and mode (e.g., RGB).\n\nIn this case, it is mandatory to apply the provided ResizeAndConvert transformation:\n\n```python\nfrom utils.utils import ResizeAndConvert\n\ntransform = ResizeAndConvert(size=(224, 224), mode=\"RGB\")\n```\nThis guarantees that the dataset is compatible with fixed-size storage formats.\n\n### Fixed-Size Datasets\n\n#### Example: Using `MemmapConverter`\n\n```python\nfrom datasets.memmap.memmap_converter import MemmapConverter\n\n# Initialize the converter\nconverter = MemmapConverter(\n    input_data=\"path/to/your/dataset\",\n    output_path=\"path/to/output\",\n    images_per_file=1000,\n    batch_size=100,\n    num_workers=4,\n    shape=(224, 224),\n    im_mode=\"RGB\"\n)\n\n# Convert the dataset\nconverter.convert()\n```\n\n**Notes**:\n- Ensure the batch size divides evenly into the number of images per file.\n\n#### Example: Using a Torch Dataset\n\n```python\nfrom datasets.memmap.memmap_converter import MemmapConverter\nfrom utils.utils import ResizeAndConvert\nfrom torchvision.datasets import MNIST\n\n# Prepare the dataset\ndataset = MNIST(\n    root=\"path/to/mnist\",\n    train=True,\n    download=True,\n    transform=ResizeAndConvert((224, 224), \"RGB\")\n)\n\n# Initialize the converter\nconverter = MemmapConverter(\n    input_data=dataset,\n    output_path=\"path/to/output\",\n    images_per_file=1000,\n    batch_size=100,\n    num_workers=4,\n    shape=(224, 224),\n    im_mode=\"RGB\"\n)\n\n# Convert the dataset\nconverter.convert()\n```\n---\n\n### Flexible-Size Datasets\n\n#### Example: Using `IpcConverter`\n\n```python\nfrom datasets.pyarrow_ipc.pyarrow_ipc_converter import IpcConverter\n\n# Initialize the converter\nconverter = IpcConverter(\n    input_data=\"path/to/your/dataset\",\n    output_path=\"path/to/output\",\n    images_per_file=1000,\n    batch_size=32,\n    num_workers=4\n)\n\n# Convert the dataset\nconverter.convert()\n```\n\n**Note**: For flexible-size datasets, image size and type are automatically inferred from the original dataset.\n\n---\n\n### Reading Converted Datasets\n\n#### Example: Using `MemmapReader`\n\n```python\nfrom datasets.memmap.memmap_reader import MemmapReader\nfrom torch.utils.data import DataLoader\nfrom torchvision import transforms\n\n# Define transformations\ntransform = transforms.Compose([\n    transforms.Resize((28, 28)),\n    transforms.ToTensor(),\n    transforms.Normalize((0.1307,), (0.3081,))\n])\n\n# Initialize the reader\nreader = MemmapReader(dataset_path=\"path/to/converted/dataset\")\n\n# Create a DataLoader\ndataloader = DataLoader(\n    reader,\n    batch_size=32,\n    num_workers=4\n)\n\n# Iterate through the dataset\nfor images, labels in dataloader:\n    # Your processing code here\n    pass\n```\n\n**Note**: On Windows, use the `worker_init_fn` method to handle file handler pickling:\n```python\ndataloader = DataLoader(\n    reader,\n    batch_size=32,\n    num_workers=4,\n    worker_init_fn=MemmapReader.worker_init_fn\n)\n```\n\n---\n\n### Distributed Conversion\n\n#### Python Script\n\n```python\nimport torch.distributed as dist\n\ndef main():\n    rank = int(os.environ['SLURM_PROCID'])\n    world_size = int(os.environ['SLURM_NPROCS'])\n    \n    dist.init_process_group('nccl', world_size=world_size, rank=rank)\n    # Initialize the MemmapReader with distributed support\n    dataset = MemmapConverter(dataset_path=\"path/to/dataset\", dist=True, **kwargs)\n```\n\n#### Batch Script\n\n```bash\n#SBATCH --nodes=1\n#SBATCH --ntasks=12\n#SBATCH --ntasks-per-node=12\n#SBATCH --gres=gpu:0\n\nMASTER_ADDR=\"$(scontrol show hostnames \"$SLURM_JOB_NODELIST\" | head -n 1)\"\nMASTER_ADDR=\"${MASTER_ADDR}i\"\nexport MASTER_ADDR=\"$(nslookup \"$MASTER_ADDR\" | grep -oP '(?\u003c=Address: ).*')\"\nexport MASTER_PORT=6000\n```\n\n---\n\n## Extending the Library\n\nTo add support for new formats:\n1. Subclass `ConverterFixedSize` or `ConverterFlexibleSize`.\n2. Implement the `_custom_collate_fn` method for preprocessing batches.\n3. Define methods for saving data to disk (`_write_data_to_disk`) or initializing arrays (`_init_arrays`).\n\n### Example: Extention to Dense Segmentation Labels\n\nTo support datasets with dense segmentation labels (e.g., per-pixel class annotations), we provide an example extension of the library.\nWe created two new subclasses:\n- FixedSizeConverter:\n  - DenseMemmapConverter\n\n  - DenseMemmapReader\n- FlexibleSizeConverter\n  - DenseIpcConverter\n\n  - DenseIpcReader\n  \nThese classes are adapted versions of the standard MemmapConverter and MemmapReader, with the following changes:\n\nLabel Shape:\n- Instead of storing labels as single integers (shape: (num_instances,)), dense labels are stored as arrays with shape (num_instances, height, width, channels).\n\nTarget Transform Support:\n- A target_transform can be applied to the labels during loading to handle preprocessing (e.g., resizing, normalization).\n\n---\n\n## Usage Examples\n\nRefer to the `examples` directory for detailed use cases:\n- **Caltech256**: Local testing.\n- **ImageNet1k**: Cluster environments.\n\n---\n\n## License\n\nThis project is licensed under the MIT License. See the `LICENSE` file for details.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhelmholtz-analytics%2Fhpc_pytorch_loader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhelmholtz-analytics%2Fhpc_pytorch_loader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhelmholtz-analytics%2Fhpc_pytorch_loader/lists"}