{"id":20446914,"url":"https://github.com/eedalong/dpex","last_synced_at":"2025-04-13T01:06:07.379Z","repository":{"id":50101457,"uuid":"370594409","full_name":"eedalong/Dpex","owner":"eedalong","description":"Distributed DataLoader For Pytorch Based On Ray","archived":false,"fork":false,"pushed_at":"2021-11-05T13:11:47.000Z","size":775,"stargazers_count":24,"open_issues_count":3,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-13T01:05:45.809Z","etag":null,"topics":["data-pipeline","dataloader","deep-learning","distributed-systems","pytorch","ray"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eedalong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-25T06:53:59.000Z","updated_at":"2024-07-23T06:10:05.000Z","dependencies_parsed_at":"2022-08-31T05:01:35.857Z","dependency_job_id":null,"html_url":"https://github.com/eedalong/Dpex","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eedalong%2FDpex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eedalong%2FDpex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eedalong%2FDpex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eedalong%2FDpex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eedalong","download_url":"https://codeload.github.com/eedalong/Dpex/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248650508,"owners_count":21139678,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-pipeline","dataloader","deep-learning","distributed-systems","pytorch","ray"],"created_at":"2024-11-15T10:23:40.733Z","updated_at":"2025-04-13T01:06:07.357Z","avatar_url":"https://github.com/eedalong.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Dpex——用户无感知分布式数据预处理组件\n### 一、前言\n\n随着GPU与CPU的算力差距越来越大以及模型训练时的预处理Pipeline变得越来越复杂，CPU部分的数据预处理已经逐渐成为了模型训练的瓶颈所在，这导致单机的GPU配置的提升并不能带来期望的线性加速。预处理性能瓶颈的本质在于每个GPU能够使用的CPU算力受限，\n为了解决这个问题NVIDIA提出了scale up的方案——GPU数据预处理库[DALI](https://github.com/NVIDIA/DALI)，Tensorflow给出了scale out的方案——分布式数据预处理组件[DataService](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/data/service),而在这里我们给出Pytorch生态中的scale out方案——分布式数据预处理组件Dpex。\n\n![](imgs/1.1.png)\n![](imgs/1.2.png)\n\n### 二、架构介绍（介绍Pytorch DataLoader本身的架构以及DistDataLoader的架构）\nDpex的采用了和Pytorch的DataLoader同样的架构设计并借助Ray将数据预处理任务调度至其他机器节点进行计算。\n![](imgs/1.3.png)\n### 三、使用示例\n不仅在设计上，Dpex的实现上也完全兼容Pytorch的DataLoader。当并行数据预处理时，若设置`distribute_mode`为`True`则`DpexDataLoader`使用\n`_RayDataLoaderIter`实现分布式数据预处理，当设置为`False`时`DpexDataLoader`退回到使用Pytorch本身的`_MultiProcessingDataLoaderIter`\n实现并行数据预处理与加载。在Pytorch训练中使用Dpex非常的简单，只需要将使用到Pytorch的`DataLoader`的地方替换为Dpex中的`DpexDataLoader`即可，当你的训练机器本身为Ray集群中的一个节点时，设置 \ndistribute_mode=True可以启用分布式数据预处理。在下面我们给出单卡训练，使用DataParallel进行多卡训练以及使用DDP进行多卡训练时使用Dpex的示例，具体可参考测试文件。    \n ``` python\n    class DpexDataLoader(torch.utils.data.DataLoader):\n        def __init__(self, dataset: Dataset[T_co], distribute_mode: Optional[bool] = False, head_address=\"auto\", batch_size: Optional[int] = 1,\n                     shuffle: bool = False, sampler: Optional[Sampler[int]] = None,\n                     batch_sampler: Optional[Sampler[Sequence[int]]] = None,\n                     num_workers: int = 0, collate_fn: Optional[_collate_fn_t] = None,\n                     pin_memory: bool = False, drop_last: bool = False,\n                     timeout: float = 0, worker_init_fn: Optional[_worker_init_fn_t] = None,\n                     multiprocessing_context=None, generator=None,\n                     *, prefetch_factor: int = 2):\n    \n```\n#### 3.1 单卡训练\n如下我们给出单卡训练时使用DpexDataLoader的示例代码，具体代码细节参见[测试代码文件](https://github.com/eedalong/Dpex/blob/main/tests/test.py).\n``` python    \n    from torchvision import datasets\n    from torchvision.transforms import ToTensor\n    from Dpex import dataloader\n    \n    training_data = datasets.FashionMNIST(\n        root=\"data\",\n        train=True,\n        download=True,\n        transform=ToTensor()\n    )\n    # use DpexDataLoader\n    train_loader = dataloader.DpexDataLoader(training_data, distribute_mode=True, num_workers=10, batch_size=100, shuffle=True)\n    \n    for epoch in range(3):\n        for index, (image, label) in enumerate(train_loader):\n           # your train process\n           pass\n```\n#### 3.2 基于DataParallel的多卡训练\n如下我们给出使用DataParallel并行训练时使用DpexDataLoader的示例代码，具体代码细节参见[测试代码文件](https://github.com/eedalong/Dpex/blob/main/tests/pytorch_data_parallel.py).\n\n``` python\n    import torch\n    import torch.nn as nn\n    from torch.autograd import Variable\n    from torch.utils.data import Dataset\n    from Dpex import dataloader\n    \n    class MyOwnDataset(Dataset):\n         pass\n         \n    # use DpexDataLoader\n    data_loader = dataloader.DpexDataLoader(dataset=RandomDataset(input_size, data_size),\n                                            distribute_mode=True, batch_size=batch_size, shuffle=True, num_workers=10)\n    \n    class Model(nn.Module):\n        pass\n        \n    model = Model()\n    \n    if torch.cuda.is_available():\n        model.cuda()\n    \n    if torch.cuda.device_count() \u003e 1:\n        model = nn.DataParallel(model)\n    \n    for data in data_loader:\n       # train your own model\n       pass\n```\n\n\n#### 3.3 基于DDP的多卡训练\n如下我们给出使用DDP并行训练时使用DpexDataLoader的示例代码，具体代码细节参见[测试代码文件](https://github.com/eedalong/Dpex/blob/main/tests/pytorch_ddp.py).\n    \n ``` python\n    import torch\n    import torch.nn as nn\n    from torch.utils.data import Dataset\n    from Dpex.dataloader import DpexDataLoader\n    from torch.utils.data.distributed import DistributedSampler\n    \n    # start command: CUDA_VISIBLE_DEVICES=1,6,7 python -m torch.distributed.launch --nproc_per_node=2 pytorch_ddp.py\n    # 1) 初始化\n    torch.distributed.init_process_group(backend=\"nccl\")\n    \n    input_size = 5\n    output_size = 2\n    batch_size = 1\n    data_size = 90000\n    \n    # 2） 配置每个进程的gpu\n    local_rank = torch.distributed.get_rank()\n    torch.cuda.set_device(local_rank)\n    device = torch.device(\"cuda\", local_rank)\n    \n    class RandomDataset(Dataset):\n        def __init__(self, size, length):\n            self.len = length\n            self.data = torch.randn(length, size)\n    \n        def __getitem__(self, index):\n            return self.data[index]\n    \n        def __len__(self):\n            return self.len\n    \n    dataset = RandomDataset(input_size, data_size)\n    # 3）使用DistributedSampler\n    rand_loader = DpexDataLoader(dataset=dataset, distribute_mode=True, batch_size=batch_size, sampler=DistributedSampler(dataset), num_workers=10)\n    \n    class Model(nn.Module):\n        def __init__(self, input_size, output_size):\n            super(Model, self).__init__()\n            self.fc = nn.Linear(input_size, output_size)\n    \n        def forward(self, input):\n            output = self.fc(input)\n            print(\"  In Model: input size\", input.size(),\n                  \"output size\", output.size())\n            return output\n    \n    model = Model(input_size, output_size)\n    \n    # 4) 封装之前要把模型移到对应的gpu\n    model.to(device)\n    \n    if torch.cuda.device_count() \u003e 1:\n        print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n        # 5) 封装\n        model = torch.nn.parallel.DistributedDataParallel(model,\n                                                          device_ids=[local_rank],\n                                                          output_device=local_rank)\n    \n    for data in rand_loader:\n        if torch.cuda.is_available():\n            input_var = data\n        else:\n            input_var = data\n    \n        output = model(input_var)\n        print(\"Outside: input size\", input_var.size(), \"output_size\", output.size())\n```\n### 四、Benchmark\n在接下来的Benchamark中我们核心展示两个部分的内容:\n - DpexDataLoader对于模型训练精度的影响\n - DpexDataLoader对于模型训练速度的影响 \n\nDpex只是将单机数据预处理水平扩展到了多机以借助更多的算力来加速数据预处理而不改变数据本身的加载和与处理方式，所以本身对模型的精度不会有负面影响。对于数据预处理较重的情况\n#### 4.1 模型精度Benchmark\n我们在`FashionMNIST`数据集上进行模型训练精度的Benchmark实验，具体代码细节见[测试文件](https://github.com/eedalong/Dpex/tree/main/tests)\n| **Accuracy**(%) | **Loss** | **GPU Settings** | **DpexDataLoader(distributed_mode=?)** | **Epoch** | **Learning rate** | **Batch size** |\n| --------------- | -------- | ---------------- | ------------------------------ | --------- | ----------------- | -------------- |\n| 90.65           | 0.137    | Single GPU       | True                           | 40        | 0.001             | 100            |\n| 91.09           | 0.112    | Single GPU       | False                          | 40        | 0.001             | 100            |\n| 90.67           | 0.016    | DataParallel     | True                           | 40        | 0.001             | 100            |\n| 90.32           | 0.008    | DataParallel     | False                          | 40        | 0.001             | 100            |\n| 88.98           | 0.034    | DDP              | True                           | 40        | 0.001             | 100            |\n| 89.84           | 0.030    | DDP              | False                          | 40        | 0.001             | 100            |\n\n#### 4.2 训练速度Benchmark\n\n### 五、环境依赖:\nDpex借助Ray完成任务的跨机调度，所以若希望使用分布式数据预处理首先需要将你的训练机器构建成Ray的集群。Ray的集群构建细节具体参考[Ray的相关文档](https://github.com/ray-project/)\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feedalong%2Fdpex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feedalong%2Fdpex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feedalong%2Fdpex/lists"}