{"id":15358067,"url":"https://github.com/antonior92/advtrain-linreg","last_synced_at":"2025-08-19T00:32:19.248Z","repository":{"id":196504225,"uuid":"696261393","full_name":"antonior92/advtrain-linreg","owner":"antonior92","description":"Explore properties adversarial training in linear models. Companion code to the paper \"Regularization properties of adversarially-trained linear regression\"","archived":false,"fork":false,"pushed_at":"2023-12-07T13:51:33.000Z","size":1488,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-31T01:13:42.533Z","etag":null,"topics":["adversarial-attacks","adversarial-machine-learning","linear-models","linear-regression","linear-regression-python","machine-learning"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2310.10807","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/antonior92.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-25T12:11:50.000Z","updated_at":"2024-04-27T11:19:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"e196650b-5b2c-430e-8554-f1b06c3820f5","html_url":"https://github.com/antonior92/advtrain-linreg","commit_stats":{"total_commits":21,"total_committers":1,"mean_commits":21.0,"dds":0.0,"last_synced_commit":"d7ef6b5a050828d69d4ca9005197c0331cb0cd02"},"previous_names":["antonior92/advtrain-linreg"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonior92%2Fadvtrain-linreg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonior92%2Fadvtrain-linreg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonior92%2Fadvtrain-linreg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonior92%2Fadvtrain-linreg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/antonior92","download_url":"https://codeload.github.com/antonior92/advtrain-linreg/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230293977,"owners_count":18203779,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-attacks","adversarial-machine-learning","linear-models","linear-regression","linear-regression-python","machine-learning"],"created_at":"2024-10-01T12:39:52.641Z","updated_at":"2024-12-18T15:23:42.985Z","avatar_url":"https://github.com/antonior92.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Regularization properties of adversarially-trained linear regression\n\n\n\n\nCompanion code to the [paper](https://arxiv.org/abs/2310.10807): \n```\nRegularization properties of adversarially-trained linear regression\nAntônio H Ribeiro, Dave Zachariah, Francis Bach, Thomas B. Sch\\\"on\nNeurIPS 2023 (spotlight)\n```\n\nPaper: [arXiv:2310.10807](https://arxiv.org/abs/2310.10807)\n\nOther info:\n[open review](https://openreview.net/forum?id=K8gLHZIgVW) - \n[video](https://recorder-v3.slideslive.com/?share=86229\u0026s=006e4a99-1e12-463e-b7f1-6767feb64b7e) - \n[poster](https://antonior92.github.io/pdfs/posters/2023-Neurips.pdf) - \n[slides](https://antonior92.github.io/pdfs/slides/2023-NeurIPS.pdf) -\n[NeurIPS poster page](https://nips.cc/virtual/2023/poster/72028) -\n[summary tweet](https://twitter.com/ahortaribeiro/status/1732429927784292772)\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003ePaper Abstract:\u003c/b\u003e\u003c/summary\u003e\n\u003ci\u003eState-of-the-art machine learning models can be vulnerable to very small input\nperturbations that are adversarially constructed. Adversarial training is an \neffective approach to defend against it. Formulated as a min-max problem, it\nsearches for the best solution when the training data were corrupted by the \nworst-case attacks. Linear models are among the simple models where vulnerabilities \ncan be observed and are the focus of our study. In this case, adversarial training \nleads to a convex optimization problem which can be formulated as the minimization \nof a finite sum. We provide a comparative analysis between the solution of adversarial \ntraining in linear regression and other regularization methods. Our main findings are \nthat: (A) Adversarial training yields the minimum-norm interpolating solution in the \noverparameterized regime (more parameters than data), as long as the maximum disturbance\nradius is smaller than a threshold. And, conversely, the minimum-norm interpolator is \nthe solution to adversarial training with a given radius. (B) Adversarial training can \nbe equivalent to parameter shrinking methods (ridge regression and Lasso). This happens\nin the underparametrized region, for an appropriate choice of adversarial radius and \nzero-mean symmetrically distributed covariates. (C) For $\\ell_\\infty$-adversarial \ntraining---as in square-root Lasso---the choice of adversarial radius for optimal \nbounds does not depend on the additive noise variance. We confirm our theoretical \nfindings with numerical examples.\u003c/i\u003e\n\u003c/details\u003e\n\n\n\n\n\n## Description\n\nGiven pair of input-output samples $(x_i, y_i), i = 1, \\dots, n$, adversarial training in linear regression is \nformulated as  a min-max optimization problem:\n\n$$\\min_\\beta \\frac{1}{n} \\sum_{i=1}^n \\max_{||\\Delta x_i|| \\le \\delta} (y_i - \\beta^\\top(x_i+ \\Delta x_i))^2$$\n\nThe paper analyse the properties of this problem and this repository contain code for reproducing the experiements\nin the paper.\n\n\n## Jupyter notebooks\nWe provide jupiter notebooks with minimal examples that can be used to quickly reproduce some of the paper main results:\n\n*We try to keep these notebooks as simple as possible, removing some of the plot configurations or\nrunning the experiment only once instead of multiple times.*\n\n\n| Figure   | Jupyter Notebook | Colab | \n| ----- | ---- | ---- |\n| Fig. 1 | [![Fig1](https://img.shields.io/badge/Fig1-Notebook-f37626?logo=jupyter\u0026style=flat)](notebooks/fig1.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/antonior92/advtrain-linreg/blob/main/notebooks/fig1.ipynb) |\n| Fig. 2 |  [![Fig2](https://img.shields.io/badge/Fig2-Notebook-f37626?logo=jupyter\u0026style=flat)](notebooks/fig2.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/antonior92/advtrain-linreg/blob/main/notebooks/fig2.ipynb) |\n| Fig. 3 |  [![Fig3](https://img.shields.io/badge/Fig3-Notebook-f37626?logo=jupyter\u0026style=flat)](notebooks/fig3.ipynb)  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/antonior92/advtrain-linreg/blob/main/notebooks/fig3.ipynb) |\n| Fig. 4 |   [![Fig4](https://img.shields.io/badge/Fig4-Notebook-f37626?logo=jupyter\u0026style=flat)](notebooks/fig4.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/antonior92/advtrain-linreg/blob/main/notebooks/fig4.ipynb) |\n\n\n## Requirements\n\nWe use `cvxpy` to solve adversarial training optimization problem. We also use standard scientific python packages\n`numpy`, `scipy`, `scikit-learn`, `pandas`, `seaborn` and `matplotlib`. See `requirements.txt`.\n\n\n## Implementing adversarial training\n\n\nThe script **advtrain.py** implement adversarial training using python. It uses the reformulation provided in Proposition 1 in the paper. \n```python\nimport cvxpy as cp\nimport numpy as np\n\n\ndef compute_q(p):\n    if p != np.Inf and p \u003e 1:\n        q = p / (p - 1)\n    elif p == 1:\n        q = np.Inf\n    else:\n        q = 1\n    return q\n\n\nclass AdversarialTraining:\n    def __init__(self, X, y, p):\n        m, n = X.shape\n        q = compute_q(p)\n        # Formulate problem\n        param = cp.Variable(n)\n        param_norm = cp.pnorm(param, p=q)\n        adv_radius = cp.Parameter(name='adv_radius', nonneg=True)\n        abs_error = cp.abs(X @ param - y)\n        adv_loss = 1 / m * cp.sum((abs_error + adv_radius * param_norm) ** 2)\n        prob = cp.Problem(cp.Minimize(adv_loss))\n        self.prob = prob\n        self.adv_radius = adv_radius\n        self.param = param\n        self.warm_start = False\n\n    def __call__(self, adv_radius, **kwargs):\n        try:\n            self.adv_radius.value = adv_radius\n            self.prob.solve(warm_start=self.warm_start, **kwargs)\n            v = self.param.value\n        except:\n            v = np.zeros(self.param.shape)\n        return v\n```\n\n\n## Experiments\n\nThe folders  `minnorm`, `varying_regularization`, `random_projections`, `varying_regularization`, `comparing_methods`, `magic`, `misc`\ncontain the scripts for different experiment and for generating the figures in the paper\n\n\n*These allow to reproduce the figures in the paper exactly.\nThe jupyter notebooks above might be easier to understand, since they contain simplified code.*\n\n- `minnorm/`: Evaluate minimum norm interpolator. Plot Fig. 2 and S4, S5, S6.\n- `regulariazation_paths/`: Regularization paths. Plot Fig. 1, 3 and S1, S2, S3.\n- `varying_regularization/`: Varying regularization strength. Plot Fig. 4 and S7, S8, S9.\n- `misc/`: other scripts. Plot Fig. 5.\n- `random_projections/`: Random projections. Plot Fig. S.10.\n- `magic/`: Experiment with Magic dataset.  Plot Fig. 6.\n- `Comparing methods (magic)`:  Compare methods with and without cross-validation. Plot Fig. S.11\n  \nInside some of this folder, we include:\n- `plots/`: containing the pdf plots generated from the experiment.\n- `results/`: containing results in csv format.\n\n\n### Running all experiments\nYou can use **run.sh** to generate all figures in the paper.\n\n```sh\n# Description: Script to run all experiments in the paper\n\n### Download data ###\nwget http://mtweb.cs.ucl.ac.uk/mus/www/MAGICdiverse/MAGIC_diverse_FILES/BASIC_GWAS.tar.gz\ntar -xvf BASIC_GWAS.tar.gz\n\n### Evaluate minimum norm interpolator ###\n# ---- Plot Fig. 2 and S4, S5, S6 ---- #\n( cd minnorm \u0026\u0026 sh run.sh )\n\n###  regularization paths ###\n# ---- Plot Fig. 1, 3 and S1, S2, S3 ---- #\n( cd regularization_paths \u0026\u0026 sh run.sh )\n\n### varying regularization strength  ###\n# ---- Plot Fig. 4 and S7, S8, S9 ---- #\n( cd varying_regularization \u0026\u0026 sh run.sh )\n\n### Random projections ###\n# ---- Plot Fig. S.10---- #\n( cd random_projections \u0026\u0026 sh run.sh )\n\n### Other ###\n# ---- Plot Fig. 5 ---- #\n( cd misc \u0026\u0026 sh run.sh )\n\n### MAGIC ###\n# ---- Plot Fig. 6 ---- #\n( cd magic \u0026\u0026 sh run.sh )\n\n### Comparing methods (magic) ###\n# ---- Plot Fig. S.11---- #\n( cd comparing_methods \u0026\u0026 sh run.sh )\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantonior92%2Fadvtrain-linreg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fantonior92%2Fadvtrain-linreg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantonior92%2Fadvtrain-linreg/lists"}