{"id":13717561,"url":"https://github.com/PistonY/torch-toolbox","last_synced_at":"2025-05-07T07:31:47.575Z","repository":{"id":43642389,"uuid":"185163711","full_name":"PistonY/torch-toolbox","owner":"PistonY","description":"🛠 Toolbox to extend PyTorch functionalities","archived":false,"fork":false,"pushed_at":"2024-05-13T15:12:09.000Z","size":348,"stargazers_count":417,"open_issues_count":4,"forks_count":56,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-10-07T21:41:41.233Z","etag":null,"topics":["arcloss","autoaugment","cosinewarmuplr","cosloss","cutout","cv2","flops","initializers","l2softmax","labelsmoothing","lmdb","lmdb-dataset","lookahead","metrics","mixup","pytorch","swish","switchnorm","teansform","toolbox"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PistonY.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-06T09:17:09.000Z","updated_at":"2024-09-27T07:50:39.000Z","dependencies_parsed_at":"2024-04-16T22:50:48.372Z","dependency_job_id":"d73e85c9-4aae-4b45-b2dd-d6388def43bd","html_url":"https://github.com/PistonY/torch-toolbox","commit_stats":{"total_commits":340,"total_committers":9,"mean_commits":37.77777777777778,"dds":"0.40294117647058825","last_synced_commit":"993a0dfe56de9eda8d12aa08ebdae6ee932545cf"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PistonY%2Ftorch-toolbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PistonY%2Ftorch-toolbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PistonY%2Ftorch-toolbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PistonY%2Ftorch-toolbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PistonY","download_url":"https://codeload.github.com/PistonY/torch-toolbox/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224573505,"owners_count":17333804,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arcloss","autoaugment","cosinewarmuplr","cosloss","cutout","cv2","flops","initializers","l2softmax","labelsmoothing","lmdb","lmdb-dataset","lookahead","metrics","mixup","pytorch","swish","switchnorm","teansform","toolbox"],"created_at":"2024-08-03T00:01:24.089Z","updated_at":"2024-11-14T05:32:12.573Z","avatar_url":"https://github.com/PistonY.png","language":"Python","funding_links":[],"categories":["Pytorch \u0026 related libraries｜Pytorch \u0026 相关库","Python","Pytorch \u0026 related libraries"],"sub_categories":["Other libraries｜其他库:","Other libraries:"],"readme":"# Torch Toolbox\n![](https://github.com/PistonY/torch-toolbox/workflows/Torch-Toolbox-CI/badge.svg)\n\nStable Version: v0.1.5 (recommended)\n\nAutomatic upload to PyPI is finished.\n\nThis project aims to provide the most frequently used tools to help you write more concise and readable PyTorch code. \n\nPull requests and issues are highly encouraged. Please reach out!\n\n- [Pytorch-Toolbox](#pytorch-toolbox)\n  * [Installing](#installing)\n  * [Todo](#todo)\n  * [Usage](#usage)\n    + [Tools](#tools)\n      - [0. Now CV2 transforms have been released.](#0-now-cv2-transforms-have-been-released)\n      - [1. Show your model parameters and FLOPs.](#1-show-your-model-parameters-and-flops)\n      - [2. Metric collection](#2-metric-collection)\n      - [3. Model Initializer](#3-model-initializer)\n      - [4. AdaptiveSequential](#4-adaptivesequential)\n      - [5. Make and Use LMDB dataset.](#5-make-and-use-lmdb-dataset)\n      - [6. Non-Lable dataset](#6-non-lable-dataset)\n      - [7. Activation Layer](#7-activation-layer)\n      - [8. FeatureVerification Metric](#8-featureverification-metric)\n    + [Fashion work](#fashion-work)\n      - [1. LabelSmoothingLoss](#1-labelsmoothingloss)\n      - [2. CosineWarmupLr](#2-cosinewarmuplr)\n      - [3. SwitchNorm2d/3d](#3-switchnorm2d-3d)\n      - [4. Swish activation](#4-swish-activation)\n      - [5. Lookahead optimizer](#5-lookahead-optimizer)\n      - [5. Mixup training](#5-mixup-training)\n      - [6. Cutout](#6-cutout)\n      - [7. No decay bias](#7-no-decay-bias)\n      - [8. Margin based classification loss](#8-margin-based-classification-loss)\n      - [9. DCNv2](#9-dcnv2)\n      - [10. FocalLoss](#10-focalloss)\n      - [11. SigmoidCrossEntropy](#11-sigmoidcrossentropy)\n      - [12. CircleLoss](#12-circleloss)\n      - [13. EvoNrom](#13-evonrom)\n      - [14. Activation Layer](#14-activation-layer)\n      - [15. Zero LastGamma Init](#15-zero-lastgamma-init)\n      - [16. SGD_GC](#16-sgc_gc)\n      - [17. Autoaugment](#17-autoaugment)\n  * [Contribution](#contribution)\n\n## Installing\nAn easy way to install Torch Toolbox is via pip:\n```shell\npip install torchtoolbox\n```\nIf you want to install the nightly version:\n```shell\npip install -U git+https://github.com/PistonY/torch-toolbox.git@master\n```\n## Todo\n- [x] cv2 transforms.\n- [ ] Integrate with [albumentations](https://github.com/albumentations-team/albumentations)\n- [ ] Prepare tensorboard support with metric collection.\n\n## Usage\nToolbox have two main parts:\n1. Auxiliary tools to make using Pytorch easier.\n2. Some fashion work which don't exist in Pytorch core.\n\nMore examples could be found at [ModelZoo.pytorch](https://github.com/PistonY/ModelZoo.pytorch)\n\n### Tools\n#### 0. Now CV2 transforms have been released.\nSupport as list:(need test)\n```python\n__all__ = [\"Compose\", \"ToTensor\", \"ToCVImage\", \"Normalize\", \"Resize\", \"Scale\", \"CenterCrop\", \"Pad\",\n           \"Lambda\", \"RandomApply\", \"RandomChoice\", \"RandomOrder\", \"RandomCrop\", \"RandomHorizontalFlip\",\n           \"RandomVerticalFlip\", \"RandomResizedCrop\", \"RandomSizedCrop\", \"FiveCrop\", \"TenCrop\", \"LinearTransformation\",\n           \"ColorJitter\", \"RandomRotation\", \"RandomAffine\", \"Grayscale\", \"RandomGrayscale\",\n           \"RandomPerspective\", \"RandomErasing\", \"RandomGaussianNoise\", \"RandomPoissonNoise\", \"RandomSPNoise\",\n           \"RandomTransforms\", \"Cutout\"]\n```\nPart of this code refers to [opencv_transforms_torchvision](https://github.com/YU-Zhiyang/opencv_transforms_torchvision)\n- [albumentations](https://github.com/albumentations-team/albumentations) is also recommended as cv2 backend transform tools.\n\n#### 1. Show your model parameters and FLOPs.\n```python\nimport torch\nfrom torchtoolbox.tools import summary\nfrom torchvision.models.mobilenet import mobilenet_v2\nmodel = mobilenet_v2()\nsummary(model, torch.rand((1, 3, 224, 224)))\n``` \nHere are some short outputs.\n```\n        Layer (type)               Output Shape          Params    FLOPs(M+A) #\n================================================================================\n            Conv2d-1          [1, 64, 112, 112]            9408       235225088\n       BatchNorm2d-2          [1, 64, 112, 112]             256         1605632\n              ReLU-3          [1, 64, 112, 112]               0               0\n         MaxPool2d-4            [1, 64, 56, 56]               0               0\n          ...                      ...                      ...              ...\n          Linear-158                  [1, 1000]         1281000         2560000\n     MobileNetV2-159                  [1, 1000]               0               0\n================================================================================\n        Total parameters: 3,538,984  3.5M\n    Trainable parameters: 3,504,872\nNon-trainable parameters: 34,112\nTotal flops(M)  : 305,252,872  305.3M\nTotal flops(M+A): 610,505,744  610.5M\n--------------------------------------------------------------------------------\nParameters size (MB): 13.50\n```\n\n#### 2. Metric collection\nWhen we train a model we usually need to calculate some metrics like accuracy(top1-acc), loss etc.\nNow toolbox support the following:\n1. Accuracy: top-1 acc.\n2. TopKAccuracy: topK-acc.\n3. NumericalCost: This is a number metric collection which support `mean`, `max`, `min` calculate type.\n4. FeatureVerification.\n    - This is widely used in margin based algorithm.\n\n```python\nfrom torchtoolbox import metric\n\n# define first\ntop1_acc = metric.Accuracy(name='Top1 Accuracy')\ntop5_acc = metric.TopKAccuracy(top=5, name='Top5 Accuracy')\nloss_record = metric.NumericalCost(name='Loss')\n\n# reset before using\ntop1_acc.reset()\ntop5_acc.reset()\nloss_record.reset()\n\n...\nmodel.eval()\nfor data, labels in val_data:\n    data = data.to(device, non_blocking=True)\n    labels = labels.to(device, non_blocking=True)\n\n    outputs = model(data)\n    losses = Loss(outputs, labels)\n    # update/record\n    top1_acc.update(outputs, labels)\n    top5_acc.update(outputs, labels)\n    loss_record.update(losses)\n    test_msg = 'Test Epoch {}: {}:{:.5}, {}:{:.5}, {}:{:.5}\\n'.format(\n    epoch, top1_acc.name, top1_acc.get(), top5_acc.name, top5_acc.get(),\n    loss_record.name, loss_record.get())\n\nprint(test_msg)\n``` \nThen you may get outputs like this\n```\nTest Epoch 101: Top1 Accuracy:0.7332, Top5 Accuracy:0.91514, Loss:1.0605\n```\n\n#### 3. Model Initializer\nNow ToolBox support `XavierInitializer` and `KaimingInitializer`.\n```python\nfrom torchtoolbox.nn.init import KaimingInitializer\n\nmodel = XXX\ninitializer = KaimingInitializer()\nmodel.apply(initializer)\n\n```\n#### 4. AdaptiveSequential\nMake Pytorch `nn.Sequential` could handle multi input/output layer.\n```python\nfrom torch import nn\nfrom torchtoolbox.nn import AdaptiveSequential\nimport torch\n\n\nclass n_to_n(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.conv1 = nn.Conv2d(3, 3, 1, 1, bias=False)\n        self.conv2 = nn.Conv2d(3, 3, 1, 1, bias=False)\n\n    def forward(self, x1, x2):\n        y1 = self.conv1(x1)\n        y2 = self.conv2(x2)\n        return y1, y2\n\n\nclass n_to_one(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.conv1 = nn.Conv2d(3, 3, 1, 1, bias=False)\n        self.conv2 = nn.Conv2d(3, 3, 1, 1, bias=False)\n\n    def forward(self, x1, x2):\n        y1 = self.conv1(x1)\n        y2 = self.conv2(x2)\n        return y1 + y2\n\n\nclass one_to_n(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.conv1 = nn.Conv2d(3, 3, 1, 1, bias=False)\n        self.conv2 = nn.Conv2d(3, 3, 1, 1, bias=False)\n\n    def forward(self, x):\n        y1 = self.conv1(x)\n        y2 = self.conv2(x)\n        return y1, y2\n\nseq = AdaptiveSequential(one_to_n(), n_to_n(), n_to_one()).cuda()\ntd = torch.rand(1, 3, 32, 32).cuda()\n\nout = seq(td)\nprint(out.size())\n\n# output\n# torch.Size([1, 3, 32, 32])\n```\n#### 5. Make and Use LMDB dataset.\nIf you meet IO speed limit, you may think about [LMDB](https://lmdb.readthedocs.io/en/release/) format dataset.\nLMDB is a tiny database with some excellent properties.\n\nEasy to generate a LMDB format dataset.\n```python\nfrom torchtoolbox.tools.convert_lmdb import generate_lmdb_dataset, raw_reader\nfrom torchvision.datasets import ImageFolder\n\ndt = ImageFolder(..., loader=raw_reader)\nsave_dir = XXX \ndataset_name = YYY\ngenerate_lmdb_dataset(dt, save_dir=save_dir, name=dataset_name)\n\n```\n\nThen if you use `ImageFolder` like dataset you can easily use `ImageLMDB` to load you dataset.\n```python\nfrom torchtoolbox.data import ImageLMDB\n\ndt = ImageLMDB(db_path=save_dir, db_name=dataset_name, ...)\n```\n\n#### 6. Non-Lable dataset\nThis dataset only return images.\n\nMore details please refers to [codes](https://github.com/PistonY/torch-toolbox/blob/4838af996b972cd666fadb9fb6bd6dab2103ccad/torchtoolbox/data/datasets.py#L13)\n\n#### 7. Activation Layer\nA `Activation` Layer is provided to select a activation by string.\n```python\nfrom torchtoolbox.nn import Activation\n\nrelu = Activation('relu', auto_optimize=True)\n\n```\n\n#### 8. FeatureVerification Metric\nA `FeatureVerification Metric` used to test feature based accuracy.\nMore details refers to code.\n\n### Fashion work\n\n#### 1. LabelSmoothingLoss\n\n```python\nfrom torchtoolbox.nn import LabelSmoothingLoss\n# The num classes of your task should be defined.\nclasses = 10\n# Loss\nLoss = LabelSmoothingLoss(classes, smoothing=0.1)\n\n...\nfor i, (data, labels) in enumerate(train_data):\n    data = data.to(device, non_blocking=True)\n    labels = labels.to(device, non_blocking=True)\n\n    optimizer.zero_grad()\n    outputs = model(data)\n    # just use as usual.\n    loss = Loss(outputs, labels)\n    loss.backward()\n    optimizer.step()\n```\n\n#### 2. CosineWarmupLr\nCosine lr scheduler with warm-up epochs.It's helpful to improve acc for classification models.\n```python\nfrom torchtoolbox.optimizer import CosineWarmupLr\n\noptimizer = optim.SGD(...)\n# define scheduler\n# `batches_pre_epoch` means how many batches(times update/step the model) within one epoch.\n# `warmup_epochs` means increase lr how many epochs to `base_lr`.\n# you can find more details in file.\nlr_scheduler = CosineWarmupLr(optimizer, batches_pre_epoch, epochs,\n                              base_lr=lr, warmup_epochs=warmup_epochs)\n...\nfor i, (data, labels) in enumerate(train_data):\n    ...\n    optimizer.step()\n    # remember to step/update status here.\n    lr_scheduler.step()\n    ...\n```\n\n#### 3. SwitchNorm2d/3d\n```python\nfrom torchtoolbox.nn import SwitchNorm2d, SwitchNorm3d\n```\nJust use it like Batchnorm2d/3d.\nMore details please refer to origin paper \n[Differentiable Learning-to-Normalize via Switchable Normalization](https://arxiv.org/pdf/1806.10779.pdf) \n[OpenSourse](https://github.com/switchablenorms/Switchable-Normalization)\n\n\n#### 4. Swish activation\n```python\nfrom torchtoolbox.nn import Swish\n```\nJust use it like Relu.\nMore details please refer to origin paper \n[SEARCHING FOR ACTIVATION FUNCTIONS](https://arxiv.org/pdf/1710.05941.pdf)\n\n#### 5. Lookahead optimizer\nA wrapper optimizer seems better than Adam. \n[Lookahead Optimizer: k steps forward, 1 step back](https://arxiv.org/abs/1907.08610)\n```python\nfrom torchtoolbox.optimizer import Lookahead\nfrom torch import optim\n\noptimizer = optim.Adam(...)\noptimizer = Lookahead(optimizer)\n```\n\n#### 5. Mixup training\nMixup method to train a classification model.\n[mixup: BEYOND EMPIRICAL RISK MINIMIZATION](https://arxiv.org/pdf/1710.09412.pdf)\n\n```python\nfrom torchtoolbox.tools import mixup_data, mixup_criterion\n\n# set beta distributed parm, 0.2 is recommend.\nalpha = 0.2\nfor i, (data, labels) in enumerate(train_data):\n    data = data.to(device, non_blocking=True)\n    labels = labels.to(device, non_blocking=True)\n\n    data, labels_a, labels_b, lam = mixup_data(data, labels, alpha)\n    optimizer.zero_grad()\n    outputs = model(data)\n    loss = mixup_criterion(Loss, outputs, labels_a, labels_b, lam)\n\n    loss.backward()\n    optimizer.update()\n```\n\n#### 6. Cutout\nA image transform method.\n[Improved Regularization of Convolutional Neural Networks with Cutout](https://arxiv.org/pdf/1708.04552.pdf)\n```python\nfrom torchvision import transforms\nfrom torchtoolbox.transform import Cutout\n\n_train_transform = transforms.Compose([\n    transforms.RandomResizedCrop(224),\n    Cutout(),\n    transforms.RandomHorizontalFlip(),\n    transforms.ColorJitter(0.4, 0.4, 0.4),\n    transforms.ToTensor(),\n    normalize,\n])\n```\n\n#### 7. No decay bias\nIf you train a model with big batch size, eg. 64k, you may need this,\n[Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes](https://arxiv.org/pdf/1807.11205.pdf)\n\n```python\nfrom torchtoolbox.tools import no_decay_bias\nfrom torch import optim\n\nmodel = XXX\nparameters = no_decay_bias(model)\noptimizer = optim.SGD(parameters, ...)\n\n```\n#### 8. Margin based classification loss\nNow support:\n1. ArcLoss\n2. CosLoss\n3. L2Softmax\n\n```python\nfrom torchtoolbox.nn.loss import ArcLoss, CosLoss, L2Softmax\n```\n\nYou could use this like `nn.CrossEntropyLoss`\n\n#### 9. DCNv2\n- [Deformable ConvNets v2: More Deformable, Better Results](https://arxiv.org/abs/1811.11168)\n\n#### 10. FocalLoss\n- [Focal Loss for Dense Object Detection](https://arxiv.org/abs/1708.02002)\n\n#### 11. SigmoidCrossEntropy\n\n#### 12. CircleLoss\n- [Circle Loss: A Unified Perspective of Pair Similarity Optimization](https://arxiv.org/pdf/2002.10857)\n\n#### 13. EvoNrom\n- [Evolving Normalization-Activation Layers](https://arxiv.org/pdf/2004.02967)\n\n#### 14. Activation Layer\n1. Swish\n2. HardSwish\n3. HardSigmoid\nUsage:\n```python\nfrom torchtoolbox.nn import Swish, HardSwish, HardSigmoid\n\nswish = Swish()\nhswish = HardSwish()\nhsigmoid = HardSigmoid()\n\n```\n\n#### 15. Zero LastGamma Init\n```python\nfrom torchtoolbox.nn.init import ZeroLastGamma\n\nmodel == XXX\ninit = ZeroLastGamma(block_name='Bottleneck', bn_name='bn3')\nmodel.apply(init)\n```\n\n#### 16. SGC_GC\n[Gradient Centralization: A New Optimization Technique for Deep Neural Networks](https://arxiv.org/pdf/2004.01461.pdf)\n\n```python\nfrom torchtoolbox.optimizer import SGD_GC\n\noptimizer = SGD_GC(model.parameters())\noptimizer.zero_grad()\nloss.backward()\noptimizer.step()\n\n```\n\n#### 17. Autoaugment\n```python\n# for ImageNet\nfrom torchtoolbox.transform import ImageNetPolicy, Compose, \\\n            RandomResizedCrop, RandomHorizontalFlip, ToTensor\n\ntransforms = Compose([RandomResizedCrop(224), \n                      RandomHorizontalFlip(),\n                      ImageNetPolicy,\n                      ToTensor()])\n```\n\n## Contribution\n\nWelcome pull requests and issues!!!\n ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPistonY%2Ftorch-toolbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPistonY%2Ftorch-toolbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPistonY%2Ftorch-toolbox/lists"}