{"id":27169180,"url":"https://github.com/smilelab-fl/fednoisy","last_synced_at":"2026-03-15T23:47:15.717Z","repository":{"id":173431587,"uuid":"648743789","full_name":"SMILELab-FL/FedNoisy","owner":"SMILELab-FL","description":"Benchmark for federated noisy label learning","archived":false,"fork":false,"pushed_at":"2024-08-28T20:28:38.000Z","size":45602,"stargazers_count":17,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-08-29T06:50:45.660Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SMILELab-FL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-02T17:43:17.000Z","updated_at":"2024-08-28T20:28:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"7dc4f208-ea1e-4fac-b4c3-69548b92d8a2","html_url":"https://github.com/SMILELab-FL/FedNoisy","commit_stats":null,"previous_names":["smilelab-fl/fednoisy"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SMILELab-FL%2FFedNoisy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SMILELab-FL%2FFedNoisy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SMILELab-FL%2FFedNoisy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SMILELab-FL%2FFedNoisy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SMILELab-FL","download_url":"https://codeload.github.com/SMILELab-FL/FedNoisy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247991692,"owners_count":21029754,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-09T06:31:42.250Z","updated_at":"2026-03-15T23:47:15.650Z","avatar_url":"https://github.com/SMILELab-FL.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FedNoisy: Federated Noisy Label Learning Benchmark\n\nBenchmark for federated noisy label learning.\n\n\n[toc]\n\n[![License](https://img.shields.io/github/license/SMILELab-FL/FedNoisy)](https://opensource.org/licenses/Apache-2.0) [![arXiv](https://img.shields.io/badge/arXiv-2306.11650-red.svg)](https://arxiv.org/abs/2306.11650) \n\nFederated learning has gained popularity for distributed learning without aggregating sensitive data from clients. But meanwhile, the distributed and isolated nature of data isolation may be complicated by data quality, making it more vulnerable to noisy labels. Many efforts exist to defend against the negative impacts of noisy labels in centralized or federated settings. However, there is a lack of a benchmark that comprehensively considers the impact of noisy labels in a wide variety of typical FL settings. \n\nIn this work, we serve the first standardized benchmark that can help researchers fully explore potential federated noisy settings. Also, we conduct comprehensive experiments to explore the characteristics of these data settings and unravel challenging scenarios on the federated noisy label learning, which may guide method development in the future. We highlight the 20 basic settings for more than 5 datasets proposed in our benchmark and standardized simulation pipeline for federated noisy label learning. We hope this benchmark can facilitate idea verification in federated learning with noisy labels. \n\n## Environment requirements\n\nPytorch installation via `conda`:\n\n```bash\n$ conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c nvidia -y\n```\n\nExtra dependencies:\n```bash\n$ pip install -r requirements.txt\n```\n\n\n\n## Federated Noisy Labe Settings\n\nCurrent supported datasets:\n\u003ctable\u003e\n\t\u003ctbody\u003e\n\t\t\u003ctr \u003e\n\t\t\t\u003cth style=\"text-align: left; vertical-align: middle;\"\u003eNoise Scene\u003c/th\u003e\n\t\t\t\u003cth style=\"text-align: left; vertical-align: middle;\"\u003eDataset\u003c/th\u003e\n\t\t\t\u003cth style=\"text-align: left; vertical-align: middle;\"\u003e#Train\u003c/th\u003e\n\t\t\t\u003cth style=\"text-align: left; vertical-align: middle;\"\u003e#Validation\u003c/th\u003e\n\t\t\t\u003cth style=\"text-align: left; vertical-align: middle;\"\u003e#Test\u003c/th\u003e\n\t\t\t\u003cth style=\"text-align: left; vertical-align: middle;\"\u003e#Class\u003c/th\u003e\n\t\t\t\u003cth style=\"text-align: left; vertical-align: middle;\"\u003eImageSize\u003c/th\u003e\n\t\t\t\u003cth style=\"text-align: left; vertical-align: middle;\"\u003eNoise Ratio (%)\u003c/th\u003e\n\t\t\u003c/tr\u003e\n\t\t\u003ctr \u003e\n\t\t\t\u003ctd style=\"vertical-align: middle;\" rowspan=\"4\"\u003e\n\t\t\t\t\u003cp\u003eGlobalized\u003c/p\u003e\n\t\t\t\t\u003cp\u003e\u0026amp;\u003c/p\u003e\n\t\t\t\t\u003cp\u003eLocalized\u003c/p\u003e\n\t\t\t\u003c/td\u003e\n\t\t\t\u003ctd style=\"text-align: left; vertical-align: middle;\"\u003eMNIST\u003c/td\u003e\n\t\t\t\u003ctd\u003e60K\u003c/td\u003e\n\t\t\t\u003ctd\u003eN/A\u003c/td\u003e\n\t\t\t\u003ctd\u003e10K\u003c/td\u003e\n\t\t\t\u003ctd\u003e10\u003c/td\u003e\n\t\t\t\u003ctd\u003e28\u0026times;28\u003c/td\u003e\n\t\t\t\u003ctd\u003eN/A\u003c/td\u003e\n\t\t\u003c/tr\u003e\n\t\t\u003ctr \u003e\n\t\t\t\u003ctd style=\"text-align: left; vertical-align: middle;\"\u003eSVHN\u003c/td\u003e\n\t\t\t\u003ctd\u003e73K\u003c/td\u003e\n\t\t\t\u003ctd\u003eN/A\u003c/td\u003e\n\t\t\t\u003ctd\u003e26K\u003c/td\u003e\n\t\t\t\u003ctd\u003e10\u003c/td\u003e\n\t\t\t\u003ctd\u003e32\u0026times;32\u0026times;3\u003c/td\u003e\n\t\t\t\u003ctd\u003eN/A\u003c/td\u003e\n\t\t\u003c/tr\u003e\n\t\t\u003ctr \u003e\n\t\t\t\u003ctd style=\"text-align: left; vertical-align: middle;\"\u003eCIFAR-10\u003c/td\u003e\n\t\t\t\u003ctd\u003e50K\u003c/td\u003e\n\t\t\t\u003ctd\u003eN/A\u003c/td\u003e\n\t\t\t\u003ctd\u003e10K\u003c/td\u003e\n\t\t\t\u003ctd\u003e10\u003c/td\u003e\n\t\t\t\u003ctd\u003e32\u0026times;32\u0026times;3\u003c/td\u003e\n\t\t\t\u003ctd\u003eN/A\u003c/td\u003e\n\t\t\u003c/tr\u003e\n\t\t\u003ctr \u003e\n\t\t\t\u003ctd style=\"text-align: center; vertical-align: middle;\"\u003eCIFAR-100\u003c/td\u003e\n\t\t\t\u003ctd\u003e50K\u003c/td\u003e\n\t\t\t\u003ctd\u003eN/A\u003c/td\u003e\n\t\t\t\u003ctd\u003e10K\u003c/td\u003e\n\t\t\t\u003ctd\u003e100\u003c/td\u003e\n\t\t\t\u003ctd\u003e32\u0026times;32\u0026times;3\u003c/td\u003e\n\t\t\t\u003ctd\u003eN/A\u003c/td\u003e\n\t\t\u003c/tr\u003e\n\t\t\u003ctr \u003e\n\t\t\t\u003ctd style=\"text-align: center; vertical-align: middle;\" rowspan=\"2\"\u003eReal-world\u003c/td\u003e\n\t\t\t\u003ctd style=\"text-align: center; vertical-align: middle;\"\u003eClothing1M\u003c/td\u003e\n\t\t\t\u003ctd\u003e1M\u003c/td\u003e\n\t\t\t\u003ctd\u003e14K\u003c/td\u003e\n\t\t\t\u003ctd\u003e10K\u003c/td\u003e\n\t\t\t\u003ctd\u003e14\u003c/td\u003e\n\t\t\t\u003ctd\u003e224\u0026times;224\u0026times;3\u003c/td\u003e\n\t\t\t\u003ctd\u003e\u0026asymp;39.46\u003c/td\u003e\n\t\t\u003c/tr\u003e\n\t\t\u003ctr \u003e\n\t\t\t\u003ctd\u003eWebVision\u003c/td\u003e\n\t\t\t\u003ctd\u003e2.4M\u003c/td\u003e\n\t\t\t\u003ctd\u003e50K\u003c/td\u003e\n\t\t\t\u003ctd\u003e50K\u003c/td\u003e\n\t\t\t\u003ctd\u003e1000\u003c/td\u003e\n\t\t\t\u003ctd\u003e256\u0026times;256\u0026times;3\u003c/td\u003e\n\t\t\t\u003ctd\u003e\u0026asymp;20.00\u003c/td\u003e\n\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\n\nFederated noise scenes provided in $\\texttt{FedNoisy}$ is shown as below. (Left: _globalized noise_; middle: _localized noise_; right: _real-word noise_. Globalized noise follows a constant noise ratio $\\varepsilon_{global}$ in label corruption, while localized noise draws localized noise ratio $\\varepsilon_{k}$ from a uniform distribution on each client $k$.)\n\n\u003cimg src=\"./imgs/fednoisy-noise-scenes.png\" height=\"400\"\u003e\n\n\n\n### Prepare\n\nRaw dataset should be downloaded in to local folder before data-build process. The folder structure can be:\n\n```shell\n.\n├── FedNoisy\n│   ├── LICENSE\n│   ├── README.md\n│   ├── build_dataset_fed.py\n│   ├── fednoisy\n│   ├── imgs\n│   ├── requirements.txt\n│   └── scripts\n├── rawdata\n│   ├── cifar10\n│   │   ├── cifar-10-batches-py\n│   │   └── cifar-10-python.tar.gz\n│   ├── cifar100\n│   │   ├── cifar-100-python\n│   │   └── cifar-100-python.tar.gz\n│   ├── mnist\n│   │   └── MNIST\n│   │       └── raw\n│   │           ├── t10k-images-idx3-ubyte\n│   │           ...\n│   │           └── train-labels-idx1-ubyte.gz\n│   ├── webvision1.0/\n│   │   ├── info\n│   │   ├── google\n│   │   ├── README.txt\n│   │   └── val_images_256\n│   ├── imagenet_data/\n│   │   └── ILSVRC2012\n│   │       ├── train\n│   │       └── val\n│   ├── clothing1M/\n│   │   ├── category_names_chn.txt\n│   │   ├── category_names_eng.txt\n│   │   ├── clean_label_kv.txt\n│   │   ├── clean_test_key_list.txt\n│   │   ├── clean_train_key_list.txt\n│   │   ├── clean_val_key_list.txt\n│   │   ├── images\n│   │   │   ├── 0\n│   │   │   ├── 1\n│   │   │   ...\n│   │   │   └── 9\n│   │   ├── noisy_label_kv.txt\n│   │   ├── noisy_train_key_list.txt\n│   │   ├── README.md\n│   │   └── venn.png\n│   └── svhn\n│       ├── extra_32x32.mat\n│       ├── test_32x32.mat\n│       └── train_32x32.mat\n├── fedNLLdata  # to store fedNLL data settings \n└── Fed-Noisy-checkpoint  # to store algorithm logging output and final models \n```\n\n- Use `torchvision.dataset` to download of __CIFAR-10__/__CIFAR-100__/__SVHN__/__MNIST__ directly:\n\n  ```python\n  import torchvision\n  \n  mnist_train = torchvision.datasets.MNIST(root=\"rawdata/mnist\", train=True, download=True)\n  mnist_test = torchvision.datasets.MNIST(root=\"rawdata/mnist\", train=False, download=True)\n  \n  cifar10_train = torchvision.datasets.CIFAR10(root=\"rawdata/cifar10\", train=True, download=True)\n  cifar10_test = torchvision.datasets.CIFAR10(root=\"rawdata/cifar10\", train=False, download=True)\n  \n  cifar100_train = torchvision.datasets.CIFAR100(root=\"rawdata/cifar100\", train=True, download=True)\n  cifar100_test = torchvision.datasets.CIFAR100(root=\"rawdata/cifar100\", train=False, download=True)\n  \n  svhn_train = torchvision.datasets.SVHN(root=\"rawdata/svhn\", split=\"train\", download=True)\n  svhn_test = torchvision.datasets.SVHN(root=\"rawdata/svhn\", split=\"test\", download=True)\n  ```\n\n- To download __[Clothing1M](https://github.com/Cysu/noisy_label)__, please contact *tong.xiao.work[at]gmail[dot]com* to get the download link. Untar the images and unzip the annotations under `rawdata/clothing1M`.\n\n- To download __[WebVision1.0](https://data.vision.ee.ethz.ch/cvl/webvision/download.html)__, only use \"Resized Images (small version)\". Unzip the data under `rawdata/webvision1.0`.\n\n\n\n### Build dataset with noisy label\n\nThe basic command usage is\n\n```bash\n$ cd FedNoisy \n\n# under dir FedNoisy/\n$ python build_dataset_fed.py --dataset cifar10 \\\n     --partition iid \\\n     --num_clients 10 \\\n     --globalize \\\n     --noise_mode clean \\\n     --raw_data_dir ../rawdata/cifar10 \\\n     --data_dir ../fedNLLdata/cifar10 \\\n     --seed 1\n```\n\n\n\n#### Noise related setting\n\n- Clean: `--globalize --noise_mode clean` for data setting without noise\n\n- Globalized noise\n  - `--globalize --noise_ratio 0.4 --noise_mode sym` for globalized symmetric noise $\\varepsilon_{global}=0.4$\n  - `--globalize --noise_ratio 0.4 --noise_mode asym` for globalized asymmetric noise $\\varepsilon_{global}=0.4$\n  \n- Localized noise \n  - `--min_noise_ratio 0.3 --max_noise_ratio 0.5 --noise_mode sym` for localized symmetric noise $\\varepsilon_k \\sim \\mathcal{U}(0.3, 0.5)$\n  - `--min_noise_ratio 0.3 --max_noise_ratio 0.5 --noise_mode asym` for localized asymmetric noise $\\varepsilon_k \\sim \\mathcal{U}(0.3, 0.5)$\n  \n- Real noise:\n\n  - `--dataset clothing1m` for Clothing1M, need to specify ` --num_sampels` as number of training samples used for Clothing1M, the default is 64000\n  - `--dataset webvision` for WebVision1.0, need to specify `--raw_imagenet_dir` for ImageNet validation set raw data dir\n  \n  \u003e Once `--dataset` is set with `clothing1m` or `webvision`, the arguement will automatically set `--globalize --noise_mode real` for real world noise.\n  \n  \n\n\n\n#### Dataset partition setting\n\n- MNIST: `--dataset mnist`\n\n  - IID: `--partition iid --num_clients 10`\n  - Non-IID quantity skew: `--partition noniid-quantity --num_clients 10 --dir_alpha 0.1`\n  - Non-IID Dirichlet-based label skew: `--partition noniid-labeldir --dir_alpha 0.1 --num_clients 10`\n  - Non-IID quantity-based label skew: `--partition noniid-#label --major_classes_num 3 --num_clients 10`\n\n- SVHN: `--dataset svhn`\n\n- CIFAR-10: `--dataset cifar10`\n\n- CIFAR-100:  `--dataset cifar100`\n\n- Clothing1M: `--dataset cloting1m`\n  - IID: `--partition iid --num_clients 10`\n  - Non-IID quantity skew: `--partition noniid-quantity --num_clients 10 --dir_alpha 0.1`\n  - Non-IID Dirichlet-based label skew: `--partition noniid-labeldir --dir_alpha 0.1 --num_clients 10`\n  - Non-IID quantity-based label skew: `--partition noniid-#label --major_classes_num 5 --num_clients 10`\n- WebVision: `--dataset webvision`\n  - IID: `--partition iid --num_clients 10`\n  - Non-IID quantity skew: `--partition noniid-quantity --num_clients 10 --dir_alpha 0.1`\n  - Non-IID Dirichlet-based label skew: `--partition noniid-labeldir --dir_alpha 0.6 --num_clients 10`\n  - Non-IID quantity-based label skew: `--partition noniid-#label --major_classes_num 20 --num_clients 10`\n\n\n\n\n\n\n\n### Algorithms\n\n\u003ctable style=\"margin-left: auto; margin-right: auto;\"\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd rowspan=\"2\"\u003e\u003cstrong\u003eFederated Algorithm\u003c/strong\u003e\u003c/td\u003e\n\u003ctd colspan=\"2\"\u003e\u003cstrong\u003eNoisy Label Algorithm\u003c/strong\u003e\u003c/td\u003e\n\u003ctd rowspan=\"2\"\u003e\u003cstrong\u003ePaper\u003c/strong\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003cem\u003eCategory\u003c/em\u003e\u003c/td\u003e\n\u003ctd\u003e\u003cem\u003eMethod\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd rowspan=\"8\"\u003eFedAvg\u003c/td\u003e\n\u003ctd\u003eRobust Regularizaiton\u003c/td\u003e\n\u003ctd\u003eMixup\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://arxiv.org/abs/1710.09412\" target=\"_blank\"\u003e[2018 ICLR] Mixup: Beyond empirical risk minimization\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd rowspan=\"3\"\u003eRobustLoss Function\u003c/td\u003e\n\u003ctd\u003eSCE\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://openaccess.thecvf.com/content_ICCV_2019/papers/Wang_Symmetric_Cross_Entropy_for_Robust_Learning_With_Noisy_Labels_ICCV_2019_paper.pdf\" target=\"_blank\"\u003e[2019 ICCV] Symmetric Cross Entropy for Robust Learning with Noisy Labels\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eGCE\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://proceedings.neurips.cc/paper/2018/hash/f2925f97bc13ad2852a7a551802feea0-Abstract.html\" target=\"_blank\"\u003e[2018 NeurIPS] Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eMAE\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://arxiv.org/abs/1712.09482\" target=\"_blank\"\u003e[2017 AAAI] Robust Loss Functions under Label Noise for Deep Neural Networks\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd rowspan=\"3\"\u003eLoss Adjustment\u003c/td\u003e\n\u003ctd\u003eM-DYR-H\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"http://proceedings.mlr.press/v97/arazo19a/arazo19a.pdf\" target=\"_blank\"\u003e[2019 ICML] Unsupervised Label Noise Modeling and Loss Correction\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eM-DYR-S\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"http://proceedings.mlr.press/v97/arazo19a/arazo19a.pdf\" target=\"_blank\"\u003e[2019 ICML] Unsupervised Label Noise Modeling and Loss Correction\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eDM-DYR-SH\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"http://proceedings.mlr.press/v97/arazo19a/arazo19a.pdf\" target=\"_blank\"\u003e[2019 ICML] Unsupervised Label Noise Modeling and Loss Correction\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eSample Selection\u003c/td\u003e\n\u003ctd\u003eCo-teaching\u003c/td\u003e\n\u003ctd\u003e\u003ca href=\"https://arxiv.org/abs/1804.06872\" target=\"_blank\"\u003e[2018 NeurIPS] Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"3\"\u003eFedNoRo\u003c/td\u003e\n\u003ctd colspan=\"1\"\u003e\u003ca href=\"https://www.ijcai.org/proceedings/2023/0492.pdf\" target=\"_blank\"\u003e[2023 IJCAI] FedNoRo: Towards Noise-Robust Federated Learning by Addressing Class Imbalance and Label Noise Heterogeneity\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\n\n\n\n\n\n- __FedAvg__\n\n  - Vanilla FedAvg\n\n    ```bash\n    # under dir FedNoisy/\n    $ python python fednoisy/algorithms/fedavg/main.py --dataset mnist \\\n        --model SimpleCNN \\\n        --partition iid \\\n        --num_clients 10 \\\n        --globalize \\\n        --noise_mode sym \\\n        --noise_ratio 0.4 \\\n        --seed 1 \\\n        --sample_ratio 1.0 \\\n        --com_round 500 \\\n        --epochs 5 \\\n        --momentum 0.9 \\\n        --lr 0.01 \\\n        --weight_decay 0.0005 \\\n        --data_dir ../fedNLLdata/mnist \\\n        --out_dir ../Fed-Noisy-checkpoint/mnist/\n    ```\n  \n  - FedAvg + Symmetric Cross Entropy\n  \n    ```bash\n    # under dir FedNoisy/\n    $ python fednoisy/algorithms/fedavg/main.py --dataset mnist \\\n        --model SimpleCNN \\\n        --partition iid \\\n        --num_clients 10 \\\n        --globalize \\\n        --noise_mode sym \\\n        --noise_ratio 0.4 \\\n        --data_dir ../fedNLLdata/mnist \\\n        --out_dir ../Fed-Noisy-checkpoint/mnist/ \\\n        --com_round 500 \\\n        --epochs 5 \\\n        --sample_ratio 1.0 \\\n        --lr 0.01 \\\n        --momentum 0.9 \\\n        --weight_decay 0.0005 \\\n        --criterion sce \\\n        --sce_alpha 0.01 \\\n        --sce_beta 1.0 \\\n        --seed 1\n    ```\n  \n  - FedAvg + GCE\n  \n    ```bash\n    # under dir FedNoisy/\n    $ python fednoisy/algorithms/fedavg/main.py --dataset mnist \\\n        --model SimpleCNN \\\n        --partition iid \\\n        --num_clients 10 \\\n        --globalize \\\n        --noise_mode sym \\\n        --noise_ratio 0.4 \\\n        --data_dir ../fedNLLdata/mnist \\\n        --out_dir ../Fed-Noisy-checkpoint/mnist/ \\\n        --com_round 500 \\\n        --epochs 5 \\\n        --sample_ratio 1.0 \\\n        --lr 0.01 \\\n        --momentum 0.9 \\\n        --weight_decay 0.0005 \\\n        --criterion gce \\\n        --gce_q 0.7 \\\n        --seed 1\n    ```\n  \n  - FedAvg + MAE\n  \n    ```bash\n    # under dir FedNoisy/\n    $ python fednoisy/algorithms/fedavg/main.py --dataset mnist \\\n        --model SimpleCNN \\\n        --partition iid \\\n        --num_clients 10 \\\n        --globalize \\\n        --noise_mode sym \\\n        --noise_ratio 0.4 \\\n        --data_dir ../fedNLLdata/mnist \\\n        --out_dir ../Fed-Noisy-checkpoint/mnist/ \\\n        --com_round 500 \\\n        --epochs 5 \\\n        --sample_ratio 1.0 \\\n        --lr 0.01 \\\n        --momentum 0.9 \\\n        --weight_decay 0.0005 \\\n        --criterion mae \\\n        --loss_scale 1.0 \\\n        --seed 1\n    ```\n  \n  - FedAvg + M-DYR-H\n  \n    ```bash\n    # under dir FedNoisy/\n    $ python fednoisy/algorithms/fedavg/main.py --dataset mnist \\\n        --model SimpleCNN \\\n        --partition iid \\\n        --num_clients 10 \\\n        --globalize \\\n        --noise_mode sym \\\n        --noise_ratio 0.4 \\\n        --data_dir ../fedNLLdata/mnist \\\n        --out_dir ../Fed-Noisy-checkpoint/mnist/ \\\n        --com_round 500 \\\n        --epochs 5 \\\n        --sample_ratio 1.0 \\\n        --lr 0.01 \\\n        --momentum 0.9 \\\n        --weight_decay 0.0005 \\\n        --dynboot \\\n        --dynboot_alpha 32 \\\n        --dynboot_mixup static \\\n        --dynboot_bootbeta hard \\\n        --dynboot_reg 1.0 \\\n        --seed 1\n    ```\n  \n  - FedAvg + M-DYR-S\n  \n    ```bash\n    # under dir FedNoisy/\n    $ python fednoisy/algorithms/fedavg/main.py --dataset mnist \\\n        --model SimpleCNN \\\n        --partition iid \\\n        --num_clients 10 \\\n        --globalize \\\n        --noise_mode sym \\\n        --noise_ratio 0.4 \\\n        --data_dir ../fedNLLdata/mnist \\\n        --out_dir ../Fed-Noisy-checkpoint/mnist/ \\\n        --com_round 500 \\\n        --epochs 5 \\\n        --sample_ratio 1.0 \\\n        --lr 0.01 \\\n        --momentum 0.9 \\\n        --weight_decay 0.0005 \\\n        --dynboot \\\n        --dynboot_alpha 32 \\\n        --dynboot_mixup static \\\n        --dynboot_bootbeta soft \\\n        --dynboot_reg 1.0 \\\n        --seed 1\n    ```\n  \n  - FedAvg + DM-DYR-SH\n  \n    ```bash\n    # under dir FedNoisy/\n    $ python fednoisy/algorithms/fedavg/main.py --dataset mnist \\\n        --model SimpleCNN \\\n        --partition iid \\\n        --num_clients 10 \\\n        --globalize \\\n        --noise_mode sym \\\n        --noise_ratio 0.4 \\\n        --data_dir ../fedNLLdata/mnist \\\n        --out_dir ../Fed-Noisy-checkpoint/mnist/ \\\n        --com_round 500 \\\n        --epochs 5 \\\n        --sample_ratio 1.0 \\\n        --lr 0.01 \\\n        --momentum 0.9 \\\n        --weight_decay 0.0005 \\\n        --dynboot \\\n        --dynboot_alpha 32 \\\n        --dynboot_mixup dynamic \\\n        --dynboot_reg 1.0 \\\n        --seed 1\n    ```\n  \n  - FedAvg + Mixup\n  \n    ```bash\n    # under dir FedNoisy/\n    $ python fednoisy/algorithms/fedavg/main.py --dataset mnist \\\n        --model SimpleCNN \\\n        --partition iid \\\n        --num_clients 10 \\\n        --globalize \\\n        --noise_mode sym \\\n        --noise_ratio 0.4 \\\n        --data_dir ../fedNLLdata/mnist \\\n        --out_dir ../Fed-Noisy-checkpoint/mnist/ \\\n        --com_round 500 \\\n        --epochs 5 \\\n        --sample_ratio 1.0 \\\n        --lr 0.01 \\\n        --momentum 0.9 \\\n        --weight_decay 5e-4 \\\n        --mixup \\\n        --mixup_alpha 1.0 \\\n        --seed 1\n    ```\n  \n  - FedAvg + Co-teaching\n  \n    ```bash\n    # under dir FedNoisy/\n    $ python fednoisy/algorithms/fedavg/main.py --dataset mnist \\\n        --model SimpleCNN \\\n        --partition iid \\\n        --num_clients 10 \\\n        --globalize \\\n        --noise_mode sym \\\n        --noise_ratio 0.4 \\\n        --data_dir ../fedNLLdata/mnist  \\\n        --out_dir ../Fed-Noisy-checkpoint/mnist/ \\\n        --com_round 500 \\\n        --epochs 5 \\\n        --sample_ratio 1.0 \\\n        --lr 0.01 \\\n        --momentum 0.9 \\\n        --weight_decay 5e-4 \\\n        --coteaching \\\n        --coteaching_forget_rate 0.4 \\  # depends on noise ratio, possible value can be noise ratio it self \n        --coteaching_num_gradual 25 \\  # default setting is 10 for 200 epochs, here we set similar ratio with number of global round\n        --seed 1\n    ```\n    \n  - FedNoRo\n  \n    ```bash\n    # under dir FedNoisy/\n    python fednoisy/algorithms/fedavg/main.py --dataset mnist \\\n        --model SimpleCNN \\\n        --partition iid \\\n        --num_clients 10 \\\n        --globalize \\\n        --noise_mode sym \\\n        --noise_ratio 0.4 \\\n        --data_dir ../fedNLLdata/mnist  \\\n        --out_dir ../Fed-Noisy-checkpoint/mnist/ \\\n        --com_round 500 \\\n        --epochs 5 \\\n        --sample_ratio 1.0 \\\n        --lr 0.01 \\\n        --momentum 0.9 \\\n        --weight_decay 5e-4 \\\n        --fednoro \\\n        --fednoro_opt sgd \\\n        --fednoro_warmup 50 \\\n        --fednoro_begin 50 \\\n        --fednoro_end 250 \\\n        --fednoro_a 0.8 \\\n        --seed 1\n    ```\n    \n    \n\nFor more scripts, please check [scripts](./scripts/) folder.\n\n\n\n## Development plan\n\n### Datasets\n\n- [ ] Include mini-ImageNet (synthetic noise)\n- [x] Include WebVision (real-world noise)\n- [ ] Include Food-101N (real-world noise)\n- [ ] Include ANIMAL-10N (real-world noise)\n\n\n\n### FNLL baselines\n\n- [x] Implement FedNoRo\n- [ ] Implement FedNed (working on...)\n- [ ] Implement FedCorr (a little bit complicated)\n\n\n\n### Others\n\n- [ ] Mixed noise modes over different clients (easy to implement)\n- [ ] Text datasets (long term goal)\n- [ ] Graph node classification datasets (long term goal)\n\n\n\n## Contributing to FedNoisy\n\n### Reporting bugs\n\nWe use GitHub issues to track all bugs and feature requests. Feel free to open an issue if you have found a bug or wish to see a feature implemented.\n\nIn case you experience issues using this package, do not hesitate to submit a ticket to the [Bug Tracker](https://github.com/SMILELab-FL/FedNoisy/issues). You are also welcome to post feature requests or pull requests.\n\n\n\n### Contributing code\n\nYou’re welcome to contribute to this project through **Pull Request**. By contributing, you agree that your contributions will be licensed under [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0.html).\n\n\n\nWe encourage you to contribute to the improvement of FedNoisy or the FedNoisy implementation of existing FNLL methods. The preferred workflow for contributing to FedNoisy is to fork the main repository on GitHub, clone, and develop on a branch. Steps as follow:\n\n1. Fork the project repository by clicking on the `Fork`. For contributing new features, please fork FedNoisy [core repo](https://github.com/SMILELab-FL/FedNoisy).\n\n2. Clone your fork of repo from your GitHub to your local:\n\n   ```shell\n   $ git clone git@github.com:YourLogin/FedLab.git\n   $ cd FedNoisy\n   ```\n\n3. Create a new branch to save your changes:\n\n   ```shell\n   $ git checkout -b my-feature\n   ```\n\n4. Develop the feature on your branch.\n\n   ```shell\n   $ git add modified_files\n   $ git commit\n   ```\n\n\n\n### Pull request checklist\n\n- Please follow the file structure below for new features or create new file if there are something new.\n\n  ```shell\n  fednoisy\n  ├── __init__.py\n  ├── algorithms\n  │   ├── __init__.py\n  │   └── fedavg\n  │       ├── __init__.py\n  │       ├── client.py               # FL client class definition\n  │       ├── main.py                 # FL algorithm main script\n  │       ├── misc.py                 \n  │       ├── server.py               # FL server class definition\n  │       └── standalone.py           # FL standalone pipeline class definition\n  ├── data\n  │   ├── NLLData\n  │   │   ├── BaseNLL                 # Base class for datasets, for download and preprocess\n  │   │   ├── CentrNLL                # CNLL class for datasets, including adding noise \n  │   │   ├── FedNLL                  # FNLL class for datasets, including adding noise and partition\n  │   │   ├── __init__.py\n  │   │   └── functional.py           # Utility functions for noisy label datasets\n  │   ├── __init__.py\n  │   ├── dataset.py\n  │   ├── ...\n  │   └── functional.py\n  ├── models\n  │   ├── __init__.py\n  │   ├── build_model.py               # model entries\n  │   ├── cnn.py                       # model architecture files\n  │   ├── .....\n  │   └── wideresnet.py\n  ├── utils                            # Utitlity functions for different algorithms  \n  └── visual.py                        # Visualization functions for FNLL \u0026 CNLL data\n  ```\n\n- Please provide shell script to launch new functions or new algorithms\n\n- All public methods should have informative docstrings with sample usage presented as doctests when appropriate. Docstring and code should follow Google Python Style Guide: [中文版](https://zh-google-styleguide.readthedocs.io/en/latest/google-python-styleguide/python_style_rules/) | [English](https://google.github.io/styleguide/pyguide.html).\n\n  \u003e __Note:__ We understand it can be a burden to add detailed docstring for code. Please have a try if it is convenient for you.\n\n\n\n## Citation\n\nPlease cite __FedNoisy__ in your publications if it helps your research:\n\n```latex\n@article{liang2023fednoisy,\n  title={FedNoisy: Federated noisy label learning benchmark},\n  author={Liang, Siqi and Huang, Jintao and Hong, Junyuan and Zeng, Dun and Zhou, Jiayu and Xu, Zenglin},\n  journal={arXiv preprint arXiv:2306.11650},\n  year={2023}\n}\n```\n\n\n\n\n## References\n\n\u003ca id=\"1\"\u003e[1]\u003c/a\u003e LeCun, Y. (1998). The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/.\n\n\u003ca id=\"2\"\u003e[2]\u003c/a\u003e Krizhevsky, A., \u0026 Hinton, G. (2009). Learning multiple layers of features from tiny images.\n\n\u003ca id=\"3\"\u003e[3]\u003c/a\u003e Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., \u0026 Ng, A. Y. (2011). Reading digits in natural images with unsupervised feature learning.\n\n\u003ca id=\"4\"\u003e[4]\u003c/a\u003e Xiao, T., Xia, T., Yang, Y., Huang, C., \u0026 Wang, X. (2015). Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2691-2699).\n\n\u003ca id=\"5\"\u003e[5]\u003c/a\u003e Li, W., Wang, L., Li, W., Agustsson, E., \u0026 Van Gool, L. (2017). Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmilelab-fl%2Ffednoisy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmilelab-fl%2Ffednoisy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmilelab-fl%2Ffednoisy/lists"}