{"id":20215927,"url":"https://github.com/thudm/grb","last_synced_at":"2025-07-24T10:01:57.907Z","repository":{"id":40685163,"uuid":"346257001","full_name":"THUDM/grb","owner":"THUDM","description":"Graph Robustness Benchmark: A scalable, unified, modular, and reproducible benchmark for evaluating the adversarial robustness of Graph Machine Learning.","archived":false,"fork":false,"pushed_at":"2023-11-06T12:31:03.000Z","size":13341,"stargazers_count":91,"open_issues_count":7,"forks_count":17,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-12-08T02:13:05.795Z","etag":null,"topics":["adversarial-attacks","deep-learning","graph-neural-networks","machine-learning","pytorch"],"latest_commit_sha":null,"homepage":"https://cogdl.ai/grb/home","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/THUDM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-10T06:35:08.000Z","updated_at":"2024-08-12T20:11:04.000Z","dependencies_parsed_at":"2024-11-14T06:26:59.859Z","dependency_job_id":"c6253e1e-a694-473c-b51e-22efbbfddb6d","html_url":"https://github.com/THUDM/grb","commit_stats":{"total_commits":154,"total_committers":4,"mean_commits":38.5,"dds":"0.40909090909090906","last_synced_commit":"fd9893a172cc679bc1268e4581c4758320270593"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUDM%2Fgrb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUDM%2Fgrb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUDM%2Fgrb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUDM%2Fgrb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/THUDM","download_url":"https://codeload.github.com/THUDM/grb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230400615,"owners_count":18219831,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-attacks","deep-learning","graph-neural-networks","machine-learning","pytorch"],"created_at":"2024-11-14T06:25:29.932Z","updated_at":"2024-12-19T08:08:19.305Z","avatar_url":"https://github.com/THUDM.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![GRB](https://github.com/THUDM/grb/blob/master/docs/source/_static/grb_logo.png)\n\n[![PyPi Latest Release](https://badge.fury.io/py/grb.svg)](https://pypi.org/project/grb/)\n[![Documentation Status](https://readthedocs.org/projects/grb/badge/?version=latest)](https://grb.readthedocs.io/en/latest/?badge=latest)\n[![License](https://img.shields.io/github/license/THUDM/grb)](./LICENSE)\n\n**[Homepage](https://cogdl.ai/grb/home)** | **[Paper](https://arxiv.org/abs/2111.04314)** | **[Datasets](https://cogdl.ai/grb/datasets)** | **[Leaderboard](https://cogdl.ai/grb/leaderboard)** | **[Documentation](https://grb.readthedocs.io/en/latest)**\n\n**Graph Robustness Benchmark (GRB)** provides _scalable_, _unified_, _modular_, and _reproducible_ evaluation on the adversarial robustness of graph machine learning models. GRB has **elaborated datasets**, **unified evaluation pipeline**, **modular coding framework**, and **reproducible leaderboards**, which facilitate the developments of graph adversarial learning, summarizing existing progress and generating insights into future research. \n\n\u003cdiv align=center\u003e\u003cimg width=\"700\" src=https://github.com/THUDM/grb/blob/master/docs/source/_static/grb_key_features.png/\u003e\u003c/div\u003e\n\n## Updates\n\n* [08/11/2021] The final version of our paper is now available in [arxiv](https://arxiv.org/abs/2111.04314), there is also a representation [video](https://recorder-v3.slideslive.com/?share=55351\u0026s=c61f55db-27d4-4c4e-8844-ad30c0e2776b) for brief introduction of GRB.\n* [11/10/2021] GRB is accepted by **NeurIPS 2021 Datasets and Benchmarks Track**! Find [our paper](https://openreview.net/pdf?id=NxWUnvwFV4) in OpenReview.\n* [26/09/2021] Add support for graph classification task! See [tutorials](https://github.com/THUDM/grb/blob/master/examples/graph_classification/) in ``examples/``.\n* [16/09/2021] Add a [paper list](https://github.com/THUDM/grb/blob/master/paperlist/README.md) of state-of-the-art researches about adversarial robustness in graph machine learning (Keep Updating).\n* [27/08/2021] Add support for modification attacks! 7 implementations and tutorials:\n    * [Implementations](https://github.com/THUDM/grb/blob/master/grb/attack/modification/)\n    * [Modification Attack](https://github.com/THUDM/grb/blob/master/examples/node_classification/modification_attack.ipynb)\n* [17/08/2021] Add AutoML function based on [optuna](https://github.com/optuna/optuna) for training models:\n    * ``AutoTrainer`` in [grb.trainer.trainer](https://github.com/THUDM/grb/blob/master/grb/utils/trainer.py)\n    * Tutorial: [Training models with AutoML](https://github.com/THUDM/grb/blob/master/examples/training_automl.ipynb)\n* [14/08/2021] Add tutorials based on jupyter notebook in ``examples/``: \n    * [Load Datasets](https://github.com/THUDM/grb/blob/master/examples/node_classification/load_datasets.ipynb)\n    * [Training models](https://github.com/THUDM/grb/blob/master/examples/node_classification/training.ipynb)\n    * [Injection Attack](https://github.com/THUDM/grb/blob/master/examples/node_classification/injection_attack.ipynb)\n## Get Started\n\n### Installation\n\nInstall grb via _pip_ (current version v0.1.0):\n```bash\npip install grb\n```\nInstall grb via _git_ (for the newest version):\n```bash\ngit clone git@github.com:THUDM/grb.git\ncd grb\npip install -e .\n```\n\n### Preparation\n\nGRB provides all necessary components to ensure the reproducibility of evaluation results.\nGet datasets from [link](https://cloud.tsinghua.edu.cn/f/21d3244b99a54a038b44/) or download them by running the following script:\n```bash\ncd ./scripts\nsh download_dataset.sh\n```\nGet attack results (adversarial adjacency matrix and features) from [link](https://cloud.tsinghua.edu.cn/f/cf400543c12d424bbaf1/) or download them by running the following script:\n```bash\nsh download_attack_results.sh\n```\nGet saved models (model weights) from [link](https://cloud.tsinghua.edu.cn/f/32d6c53f914b4809b238/) or download them by running the following script:\n```bash\nsh download_saved_models.sh\n```\n\n## Usage of GRB Modules\n\n### Training a GML model\n\nAn example of training Graph Convolutional Network ([GCN](https://arxiv.org/abs/1609.02907)) on _grb-cora_ dataset. \n\n```python\nimport torch  # pytorch backend\nfrom grb.dataset import Dataset\nfrom grb.model.torch import GCN\nfrom grb.trainer.trainer import Trainer\n\n# Load data\ndataset = Dataset(name='grb-cora', mode='easy',\n                  feat_norm='arctan')\n# Build model\nmodel = GCN(in_features=dataset.num_features,\n            out_features=dataset.num_classes,\n            hidden_features=[64, 64])\n# Training\nadam = torch.optim.Adam(model.parameters(), lr=0.01)\ntrainer = Trainer(dataset=dataset, optimizer=adam,\n                  loss=torch.nn.functional.nll_loss)\ntrainer.train(model=model, n_epoch=200, dropout=0.5,\n              train_mode='inductive')\n```\n\n### Adversarial attack\n\nAn example of applying Topological Defective Graph Injection Attack ([TDGIA](https://github.com/THUDM/tdgia)) on trained GCN model.\n\n```python\nfrom grb.attack.injection.tdgia import TDGIA\n\n# Attack configuration\ntdgia = TDGIA(lr=0.01, \n              n_epoch=10,\n              n_inject_max=20, \n              n_edge_max=20,\n              feat_lim_min=-0.9, \n              feat_lim_max=0.9,\n              sequential_step=0.2)\n# Apply attack\nrst = tdgia.attack(model=model,\n                   adj=dataset.adj,\n                   features=dataset.features,\n                   target_mask=dataset.test_mask)\n# Get modified adj and features\nadj_attack, features_attack = rst\n```\n\n## GRB Evaluation\n\n### Evaluation scenarios (Injection attack as an example)\n\n![GRB](https://github.com/THUDM/grb/blob/master/docs/source/_static/grb_scenario.png)\n\nGRB provides unified evaluation scenarios for fair comparisons between attacks and defenses. The example scenario is **Black-box**, **Evasion**, **Inductive**, **Injection**. \n\n* **Black-box**: Both the attacker and the defender have no knowledge about the applied methods each other uses.\n* **Evasion**: Models are already trained in trusted data (e.g. authenticated users), which are untouched by the attackers but might have natural noises. Thus, attacks will only happen during the inference phase. \n* **Inductive**: Models are used to classify unseen data (e.g. new users), i.e. validation or test data are unseen during training, which requires models to generalize to out of distribution data.\n* **Injection**: The attackers can only inject new nodes but not modify the target nodes directly. Since it is usually hard to hack into users' accounts and modify their profiles. However, it is easier to create fake accounts and connect them to existing users.\n\n### GRB Leaderboards\n\nGRB maintains [leaderboards](https://cogdl.ai/grb/leaderboard/) that permits a fair comparision across various attacks and defenses. To ensure the reproducibility, we provide all necessary information including datasets, attack results, saved models, etc. Besides, all results on the leaderboards can be easily reproduced by running the following scripts (e.g., [leaderboard for _grb-cora_ dataset](https://cogdl.ai/grb/leaderboard/cora, compatible with v0.1.0)):\n```bash\nsh run_leaderboard_pipeline.sh -d grb-cora -g 0 -s ./leaderboard -n 0\nUsage: run_leaderboard_pipeline.sh [-d \u003cstring\u003e] [-g \u003cint\u003e] [-s \u003cstring\u003e] [-n \u003cint\u003e]\nPipeline for reproducing leaderboard on the chosen dataset.\n    -h      Display help message.\n    -d      Choose a dataset.\n    -s      Set a directory to save leaderboard files.\n    -n      Choose the number of an attack from 0 to 9.\n    -g      Choose a GPU device. -1 for CPU.\n```\n\n## Submission\n\nWe welcome researchers to submit new methods including attacks, defenses, or new GML models to enrich the GRB leaderboard. For future submissions, one should follow the [GRB Evaluation Rules](https://cogdl.ai/grb/intro/rules) and respect the reproducibility. \n\nPlease submit your methods via the google form [GRB submission](https://docs.google.com/forms/d/e/1FAIpQLSfJaUK-SXYFnlSqTEEwTOwsqzA5JnpXyvZe8E24hlLE7scRcA/viewform). Our team will verify the result within a week.\n\n## Requirements\n\n* scipy==1.5.2\n* numpy==1.19.1\n* torch==1.8.0\n* networkx==2.5\n* pandas~=1.2.3\n* cogdl~=0.3.0.post1\n* scikit-learn~=0.24.1\n\n## Citing GRB\n\nIf you find GRB useful for your research, please cite [our paper](https://arxiv.org/abs/2111.04314):  \n\n```\n@article{zheng2021grb,\n  title={Graph Robustness Benchmark: Benchmarking the Adversarial Robustness of Graph Machine Learning},\n  author={Zheng, Qinkai and Zou, Xu and Dong, Yuxiao and Cen, Yukuo and Yin, Da and Xu, Jiarong and Yang, Yang and Tang, Jie},\n  journal={Neural Information Processing Systems Track on Datasets and Benchmarks 2021},\n  year={2021}\n}\n```\n\n## Contact\n\nIn case of any problem, please contact us via email: cogdl.grbteam@gmail.com. We also welcome researchers to join our [Google Group](https://groups.google.com/g/graph-robustness-benchmark) for further discussion on the adversarial robustness of graph machine learning.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthudm%2Fgrb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthudm%2Fgrb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthudm%2Fgrb/lists"}