{"id":16039525,"url":"https://github.com/lishenghui/blades","last_synced_at":"2025-04-10T03:53:21.753Z","repository":{"id":37464302,"uuid":"481229225","full_name":"lishenghui/blades","owner":"lishenghui","description":"⚔️ Blades: A Unified Benchmark Suite for Attacks and Defenses in Federated Learning","archived":false,"fork":false,"pushed_at":"2025-02-16T22:27:34.000Z","size":13714,"stargazers_count":141,"open_issues_count":2,"forks_count":22,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-10T03:53:12.790Z","etag":null,"topics":["byzantine-fault-tolerance","distributed-systems","fedavg","federated","federated-learning","federated-learning-framework","federated-learning-simulator","model-poisoning-attack","robust-federated-learning","robust-machine-learning","robust-optimization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lishenghui.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-04-13T13:32:50.000Z","updated_at":"2025-03-13T04:35:42.000Z","dependencies_parsed_at":"2023-01-30T22:15:58.054Z","dependency_job_id":"cc0cbb0d-ebec-4b78-98e3-ad98e2173b95","html_url":"https://github.com/lishenghui/blades","commit_stats":{"total_commits":40,"total_committers":3,"mean_commits":"13.333333333333334","dds":0.5,"last_synced_commit":"5d3320155b8c98a9ef03aa4e9230bb67a902ba4f"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lishenghui%2Fblades","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lishenghui%2Fblades/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lishenghui%2Fblades/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lishenghui%2Fblades/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lishenghui","download_url":"https://codeload.github.com/lishenghui/blades/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248154998,"owners_count":21056542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["byzantine-fault-tolerance","distributed-systems","fedavg","federated","federated-learning","federated-learning-framework","federated-learning-simulator","model-poisoning-attack","robust-federated-learning","robust-machine-learning","robust-optimization"],"created_at":"2024-10-08T23:05:11.823Z","updated_at":"2025-04-10T03:53:21.724Z","avatar_url":"https://github.com/lishenghui.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":".. raw:: html\n\n   \u003cdiv style=\"text-align: center;\"\u003e\n   \u003ca href=\"https://arxiv.org/pdf/2206.05359.pdf\"\u003e\n       \u003cimg alt=\"Tests Status\" src=\"https://img.shields.io/badge/arXiv-2206.05359-red?logo=arxiv\u0026style=flat-square\u0026link=https%3A%2F%2Farxiv.org%2Fpdf%2F2206.05359.pdf\"/\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://github.com/lishenghui/blades\"\u003e\n       \u003cimg alt=\"Build Status\" src=\"https://img.shields.io/github/last-commit/lishenghui/blades/master?logo=Github\"/\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://github.com/lishenghui/blades/actions/workflows/unit-tests.yml\"\u003e\n       \u003cimg alt=\"Tests Status\" src=\"https://github.com/lishenghui/blades/actions/workflows/unit-tests.yml/badge.svg?branch=master\"/\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://blades.readthedocs.io/en/latest/?badge=latest\"\u003e\n       \u003cimg alt=\"Docs\" src=\"https://readthedocs.org/projects/blades/badge/?version=latest\"/\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://pytorch.org/get-started/pytorch-2.0/\"\u003e\n       \u003cimg alt=\"Docs\" src=\"https://img.shields.io/badge/Pytorch-2.0-brightgreen?logo=pytorch\u0026logoColor=red\"/\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://docs.ray.io/en/releases-2.9.0/\"\u003e\n       \u003cimg alt=\"Docs\" src=\"https://img.shields.io/badge/Ray-2.9-brightgreen?logo=ray\u0026logoColor=blue\"/\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://github.com/lishenghui/blades/blob/master/LICENSE\"\u003e\n       \u003cimg alt=\"License\" src=\"https://img.shields.io/github/license/lishenghui/blades?logo=apache\u0026logoColor=red\"/\u003e\n   \u003c/a\u003e\n   \u003c/div\u003e\n\n..\n    .. image:: https://img.shields.io/github/last-commit/lishenghui/blades/master?logo=Github\n        :alt: GitHub last commit (branch)\n        :target: https://github.com/lishenghui/blades\n    .. image:: https://github.com/lishenghui/blades/actions/workflows/unit-tests.yml/badge.svg?branch=master\n       :alt: GitHub Workflow Status (with event)\n\n    .. container:: badges\n\n        .. image:: https://img.shields.io/github/last-commit/lishenghui/blades/master?logo=Github\n           :alt: GitHub last commit (branch)\n           :target: https://github.com/lishenghui/blades\n\n        .. image:: https://github.com/lishenghui/blades/actions/workflows/unit-tests.yml/badge.svg?branch=master\n           :alt: GitHub Workflow Status (with event)\n\n        .. image:: https://img.shields.io/badge/Pytorch-2.0-brightgreen?logo=pytorch\u0026logoColor=red\n           :alt: Static Badge\n           :target: https://pytorch.org/get-started/pytorch-2.0/\n\n        .. image:: https://img.shields.io/badge/Ray-2.8-brightgreen?logo=ray\u0026logoColor=blue\n           :alt: Static Badge\n           :target: https://docs.ray.io/en/releases-2.8.0/\n\n        .. image:: https://readthedocs.org/projects/blades/badge/?version=latest\n            :target: https://blades.readthedocs.io/en/latest/?badge=latest\n            :alt: Documentation Status\n\n        .. image:: https://img.shields.io/github/license/lishenghui/blades?logo=apache\u0026logoColor=red\n            :alt: GitHub\n            :target: https://github.com/lishenghui/blades/blob/master/LICENSE\n\n        .. image:: https://img.shields.io/badge/arXiv-2206.05359-red?logo=arxiv\u0026style=flat-square\u0026link=https%3A%2F%2Farxiv.org%2Fpdf%2F2206.05359.pdf\n            :alt: Static Badge\n            :target: https://arxiv.org/pdf/2206.05359.pdf\n\n\n.. raw:: html\n\n   \u003cp align=center\u003e\n        \u003cimg src=\"https://github.com/lishenghui/blades/blob/master/docs/source/images/client_pipeline.png\" width=\"1000\" alt=\"Blades Logo\"\u003e\n   \u003c/p\u003e\n\nInstallation\n==================================================\n\n.. code-block:: bash\n\n    git clone https://github.com/lishenghui/blades\n    cd blades\n    pip install -v -e .\n    # \"-v\" means verbose, or more output\n    # \"-e\" means installing a project in editable mode,\n    # thus any local modifications made to the code will take effect without reinstallation.\n\n\n.. code-block:: bash\n\n    cd blades/blades\n    python train.py file ./tuned_examples/fedsgd_cnn_fashion_mnist.yaml\n\n\n**Blades** internally calls `ray.tune \u003chttps://docs.ray.io/en/latest/tune/tutorials/tune-output.html\u003e`_; therefore, the experimental results are output to its default directory: ``~/ray_results``.\n\nExperiment Results\n==================================================\n\n.. image:: https://github.com/lishenghui/blades/blob/master/docs/source/images/fashion_mnist.png\n\n.. image:: https://github.com/lishenghui/blades/blob/master/docs/source/images/cifar10.png\n\n\n\n\nCluster Deployment\n===================\n\nTo run **blades** on a cluster, you only need to deploy ``Ray cluster`` according to the `official guide \u003chttps://docs.ray.io/en/latest/cluster/user-guide.html\u003e`_.\n\n\nBuilt-in Implementations\n==================================================\nIn detail, the following strategies are currently implemented:\n\n\n\nAttacks\n---------\n\nGeneral Attacks\n^^^^^^^^^^^^^^^^^\n+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+\n| Strategy           | Description                                                                                                                                                                                              | Sourse                                                                                                    |\n+====================+==========================================================================================================================================================================================================+===========================================================================================================+\n| **Noise**          |  Put random noise to the updates.                                                                                                                                                                        | `Sourse \u003chttps://github.com/lishenghui/blades/blob/master/blades/adversaries/noise_adversary.py\u003e`_        |\n+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+\n| **Labelflipping**  | `Fang et al. Local Model Poisoning Attacks to Byzantine-Robust Federated Learning \u003chttps://www.usenix.org/conference/usenixsecurity20/presentation/fang\u003e`_, *USENIX Security' 20*                        | `Sourse \u003chttps://github.com/lishenghui/blades/blob/master/blades/adversaries/labelflip_adversary.py\u003e`_    |\n+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+\n| **Signflipping**   | `Li et al. RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets \u003chttps://ojs.aaai.org/index.php/AAAI/article/view/3968\u003e`_, *AAAI' 19*               | `Sourse \u003chttps://github.com/lishenghui/blades/blob/master/blades/adversaries/signflip_adversary.py\u003e`_     |\n+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+\n| **ALIE**           | `Baruch et al. A little is enough: Circumventing defenses for distributed learning \u003chttps://proceedings.neurips.cc/paper/2019/hash/ec1c59141046cd1866bbbcdfb6ae31d4-Abstract.html\u003e`_ *NeurIPS' 19*       | `Sourse \u003chttps://github.com/lishenghui/blades/blob/master/blades/adversaries/alie_adversary.py\u003e`_         |\n+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+\n| **IPM**            | `Xie et al. Fall of empires: Breaking byzantine- tolerant sgd by inner product manipulation \u003chttps://arxiv.org/abs/1903.03936\u003e`_, *UAI' 20*                                                              | `Sourse \u003chttps://github.com/lishenghui/blades/blob/master/blades/adversaries/ipm_adversary.py\u003e`_          |\n+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+\n\nAdaptive Attacks\n^^^^^^^^^^^^^^^^^\n+--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------+\n| Strategy                 | Description                                                                                                                                                                         | Sourse                                                                                                          |\n+==========================+=====================================================================================================================================================================================+=================================================================================================================+\n| **DistanceMaximization** |  `Shejwalkar et al. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning \u003chttps://par.nsf.gov/servlets/purl/10286354\u003e`_, *NDSS' 21*   | `Sourse \u003chttps://github.com/lishenghui/blades/blob/master/blades/adversaries/minmax_adversary.py\u003e`_             |\n+--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------+\n\n\n.. | **FangAttack**           |  `Fang et al. Local Model Poisoning Attacks to Byzantine-Robust Federated Learning \u003chttps://www.usenix.org/conference/usenixsecurity20/presentation/fang\u003e`_, *USENIX Security' 20*  | `Sourse \u003chttps://github.com/bladesteam/blades/blob/master/src/blades/attackers/fangattackclient.py\u003e`_           |\n.. +--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------+\n\n\nDefenses\n---------\n\nRobust Aggregation\n^^^^^^^^^^^^^^^^^^^\n\n+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| Strategy              | Descriptions                                                                                                                                                                                                                                                | Source                                                                                                   |\n+=======================+=============================================================================================================================================================================================================================================================+==========================================================================================================+\n| **MultiKrum**         | `Blanchard et al. Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent \u003chttps://proceedings.neurips.cc/paper/2017/hash/f4b9ec30ad9f68f89b29639786cb62ef-Abstract.html\u003e`_, *NIPS'17*                                                       | `Source \u003chttps://github.com/lishenghui/blades/blob/master/blades/aggregators/multikrum.py\u003e`_              |\n+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| **GeoMed**            | `Chen et al. Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent \u003chttps://arxiv.org/abs/1705.05491\u003e`_, *POMACS'18*                                                                                                 | `Source \u003chttps://github.com/lishenghui/blades/blob/master/blades/aggregators/aggregators.py\u003e`_            |\n+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| **Median**            | `Yin et al. Byzantine-robust distributed learning: Towards optimal statistical rates \u003chttps://proceedings.mlr.press/v80/yin18a\u003e`_, *ICML'18*                                                                                                                | `Source \u003chttps://github.com/lishenghui/blades/blob/master/blades/aggregators/aggregators.py\u003e`_            |\n+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| **TrimmedMean**       | `Yin et al. Byzantine-robust distributed learning: Towards optimal statistical rates \u003chttps://proceedings.mlr.press/v80/yin18a\u003e`_, *ICML'18*                                                                                                                | `Source \u003chttps://github.com/lishenghui/blades/blob/master/blades/aggregators/aggregators.py\u003e`_            |\n+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| **CenteredClipping**  | `Karimireddy et al. Learning from History for Byzantine Robust Optimization \u003chttp://proceedings.mlr.press/v139/karimireddy21a.html\u003e`_, *ICML'21*                                                                                                            | `Source \u003chttps://github.com/lishenghui/blades/blob/master/blades/aggregators/centeredclipping.py\u003e`_       |\n+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| **Clustering**        | `Sattler et al. On the byzantine robustness of clustered federated learning \u003chttps://ieeexplore.ieee.org/abstract/document/9054676\u003e`_, *ICASSP'20*                                                                                                          | `Source \u003chttps://github.com/lishenghui/blades/blob/master/blades/aggregators/clippedclustering.py\u003e`_      |\n+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| **ClippedClustering** | `Li et al. An Experimental Study of Byzantine-Robust Aggregation Schemes in Federated Learning \u003chttps://ieeexplore.ieee.org/abstract/document/10018261\u003e`_, *IEEE TBD'23*                                                                                    | `Source \u003chttps://github.com/lishenghui/blades/blob/master/blades/aggregators/clippedclustering.py\u003e`_      |\n+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| **DnC**               | `Shejwalkar et al. Manipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning \u003chttps://par.nsf.gov/servlets/purl/10286354\u003e`_, *NDSS'21*                                                                             | `Source \u003chttps://github.com/lishenghui/blades/blob/master/blades/aggregators/aggregators.py\u003e`_            |\n+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| **SignGuard**         | `Xu et al. SignGuard: Byzantine-robust Federated Learning through Collaborative Malicious Gradient Filtering \u003chttps://arxiv.org/abs/2109.05872\u003e`_, *ICDCS'22*                                                                                               | `Source \u003chttps://github.com/lishenghui/blades/blob/master/blades/aggregators/signguard.py\u003e`_              |\n+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n\n\nData Partitioners:\n==================================================\n\nDirichlet Partitioner\n----------------------\n\n.. image:: https://github.com/lishenghui/blades/blob/master/docs/source/images/dirichlet_partition.png\n\nSharding Partitioner\n----------------------\n\n.. raw:: html\n\n    \u003cimg src=\"https://github.com/lishenghui/blades/blob/master/docs/source/images/shard_partition.png\" alt=\"Shard Partition\" /\u003e\n\n\n\nCitation\n=========\n\nPlease cite our `paper \u003chttps://arxiv.org/abs/2206.05359\u003e`_ (and the respective papers of the methods used) if you use this code in your own work:\n\n::\n\n    @inproceedings{li2024blades,\n    title={Blades: A Unified Benchmark Suite for Byzantine Attacks and Defenses in Federated Learning},\n    author={Li, Shenghui and Ngai, Edith and Ye, Fanghua and Ju, Li and Zhang, Tianru and Voigt, Thiemo},\n    booktitle={2024 IEEE/ACM Ninth International Conference on Internet-of-Things Design and Implementation (IoTDI)},\n    year={2024}\n    }\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flishenghui%2Fblades","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flishenghui%2Fblades","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flishenghui%2Fblades/lists"}