https://github.com/lishenghui/blades
⚔️ Blades: A Unified Benchmark Suite for Attacks and Defenses in Federated Learning
https://github.com/lishenghui/blades
byzantine-fault-tolerance distributed-systems fedavg federated federated-learning federated-learning-framework federated-learning-simulator model-poisoning-attack robust-federated-learning robust-machine-learning robust-optimization
Last synced: 10 months ago
JSON representation
⚔️ Blades: A Unified Benchmark Suite for Attacks and Defenses in Federated Learning
- Host: GitHub
- URL: https://github.com/lishenghui/blades
- Owner: lishenghui
- License: apache-2.0
- Created: 2022-04-13T13:32:50.000Z (almost 4 years ago)
- Default Branch: master
- Last Pushed: 2025-02-16T22:27:34.000Z (12 months ago)
- Last Synced: 2025-04-10T03:53:12.790Z (10 months ago)
- Topics: byzantine-fault-tolerance, distributed-systems, fedavg, federated, federated-learning, federated-learning-framework, federated-learning-simulator, model-poisoning-attack, robust-federated-learning, robust-machine-learning, robust-optimization
- Language: Python
- Homepage:
- Size: 13.1 MB
- Stars: 141
- Watchers: 2
- Forks: 22
- Open Issues: 2
-
Metadata Files:
- Readme: README.rst
- License: LICENSE
Awesome Lists containing this project
README
.. raw:: html
..
.. image:: https://img.shields.io/github/last-commit/lishenghui/blades/master?logo=Github
:alt: GitHub last commit (branch)
:target: https://github.com/lishenghui/blades
.. image:: https://github.com/lishenghui/blades/actions/workflows/unit-tests.yml/badge.svg?branch=master
:alt: GitHub Workflow Status (with event)
.. container:: badges
.. image:: https://img.shields.io/github/last-commit/lishenghui/blades/master?logo=Github
:alt: GitHub last commit (branch)
:target: https://github.com/lishenghui/blades
.. image:: https://github.com/lishenghui/blades/actions/workflows/unit-tests.yml/badge.svg?branch=master
:alt: GitHub Workflow Status (with event)
.. image:: https://img.shields.io/badge/Pytorch-2.0-brightgreen?logo=pytorch&logoColor=red
:alt: Static Badge
:target: https://pytorch.org/get-started/pytorch-2.0/
.. image:: https://img.shields.io/badge/Ray-2.8-brightgreen?logo=ray&logoColor=blue
:alt: Static Badge
:target: https://docs.ray.io/en/releases-2.8.0/
.. image:: https://readthedocs.org/projects/blades/badge/?version=latest
:target: https://blades.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
.. image:: https://img.shields.io/github/license/lishenghui/blades?logo=apache&logoColor=red
:alt: GitHub
:target: https://github.com/lishenghui/blades/blob/master/LICENSE
.. image:: https://img.shields.io/badge/arXiv-2206.05359-red?logo=arxiv&style=flat-square&link=https%3A%2F%2Farxiv.org%2Fpdf%2F2206.05359.pdf
:alt: Static Badge
:target: https://arxiv.org/pdf/2206.05359.pdf
.. raw:: html
Installation
==================================================
.. code-block:: bash
git clone https://github.com/lishenghui/blades
cd blades
pip install -v -e .
# "-v" means verbose, or more output
# "-e" means installing a project in editable mode,
# thus any local modifications made to the code will take effect without reinstallation.
.. code-block:: bash
cd blades/blades
python train.py file ./tuned_examples/fedsgd_cnn_fashion_mnist.yaml
**Blades** internally calls `ray.tune `_; therefore, the experimental results are output to its default directory: ``~/ray_results``.
Experiment Results
==================================================
.. image:: https://github.com/lishenghui/blades/blob/master/docs/source/images/fashion_mnist.png
.. image:: https://github.com/lishenghui/blades/blob/master/docs/source/images/cifar10.png
Cluster Deployment
===================
To run **blades** on a cluster, you only need to deploy ``Ray cluster`` according to the `official guide `_.
Built-in Implementations
==================================================
In detail, the following strategies are currently implemented:
Attacks
---------
General Attacks
^^^^^^^^^^^^^^^^^
+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+
| Strategy | Description | Sourse |
+====================+==========================================================================================================================================================================================================+===========================================================================================================+
| **Noise** | Put random noise to the updates. | `Sourse `_ |
+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+
| **Labelflipping** | `Fang et al. Local Model Poisoning Attacks to Byzantine-Robust Federated Learning `_, *USENIX Security' 20* | `Sourse `_ |
+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+
| **Signflipping** | `Li et al. RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets `_, *AAAI' 19* | `Sourse `_ |
+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+
| **ALIE** | `Baruch et al. A little is enough: Circumventing defenses for distributed learning `_ *NeurIPS' 19* | `Sourse `_ |
+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+
| **IPM** | `Xie et al. Fall of empires: Breaking byzantine- tolerant sgd by inner product manipulation `_, *UAI' 20* | `Sourse `_ |
+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+
Adaptive Attacks
^^^^^^^^^^^^^^^^^
+--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------+
| Strategy | Description | Sourse |
+==========================+=====================================================================================================================================================================================+=================================================================================================================+
| **DistanceMaximization** | `Shejwalkar et al. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning `_, *NDSS' 21* | `Sourse `_ |
+--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------+
.. | **FangAttack** | `Fang et al. Local Model Poisoning Attacks to Byzantine-Robust Federated Learning `_, *USENIX Security' 20* | `Sourse `_ |
.. +--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------+
Defenses
---------
Robust Aggregation
^^^^^^^^^^^^^^^^^^^
+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+
| Strategy | Descriptions | Source |
+=======================+=============================================================================================================================================================================================================================================================+==========================================================================================================+
| **MultiKrum** | `Blanchard et al. Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent `_, *NIPS'17* | `Source `_ |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+
| **GeoMed** | `Chen et al. Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent `_, *POMACS'18* | `Source `_ |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+
| **Median** | `Yin et al. Byzantine-robust distributed learning: Towards optimal statistical rates `_, *ICML'18* | `Source `_ |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+
| **TrimmedMean** | `Yin et al. Byzantine-robust distributed learning: Towards optimal statistical rates `_, *ICML'18* | `Source `_ |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+
| **CenteredClipping** | `Karimireddy et al. Learning from History for Byzantine Robust Optimization `_, *ICML'21* | `Source `_ |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+
| **Clustering** | `Sattler et al. On the byzantine robustness of clustered federated learning `_, *ICASSP'20* | `Source `_ |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+
| **ClippedClustering** | `Li et al. An Experimental Study of Byzantine-Robust Aggregation Schemes in Federated Learning `_, *IEEE TBD'23* | `Source `_ |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+
| **DnC** | `Shejwalkar et al. Manipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning `_, *NDSS'21* | `Source `_ |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+
| **SignGuard** | `Xu et al. SignGuard: Byzantine-robust Federated Learning through Collaborative Malicious Gradient Filtering `_, *ICDCS'22* | `Source `_ |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+
Data Partitioners:
==================================================
Dirichlet Partitioner
----------------------
.. image:: https://github.com/lishenghui/blades/blob/master/docs/source/images/dirichlet_partition.png
Sharding Partitioner
----------------------
.. raw:: html
Citation
=========
Please cite our `paper `_ (and the respective papers of the methods used) if you use this code in your own work:
::
@inproceedings{li2024blades,
title={Blades: A Unified Benchmark Suite for Byzantine Attacks and Defenses in Federated Learning},
author={Li, Shenghui and Ngai, Edith and Ye, Fanghua and Ju, Li and Zhang, Tianru and Voigt, Thiemo},
booktitle={2024 IEEE/ACM Ninth International Conference on Internet-of-Things Design and Implementation (IoTDI)},
year={2024}
}