{"id":19091612,"url":"https://github.com/juliagusak/neural-ode-metasolver","last_synced_at":"2025-04-30T11:10:23.327Z","repository":{"id":86498472,"uuid":"348058096","full_name":"juliagusak/neural-ode-metasolver","owner":"juliagusak","description":"Supplementary code for the paper  \"Meta-Solver for Neural Ordinary Differential Equations\"  https://arxiv.org/abs/2103.08561","archived":false,"fork":false,"pushed_at":"2021-03-30T16:59:27.000Z","size":15821,"stargazers_count":24,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-30T15:36:15.234Z","etag":null,"topics":["adversarial-attacks","neural-network","neural-ode","ordinary-differential-equations","parametrized","pytorch","robustness","runge-kutta","solver"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/juliagusak.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-15T17:12:40.000Z","updated_at":"2025-01-10T00:00:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"60509b9d-ac8e-4209-b56c-e13f3f32f9f0","html_url":"https://github.com/juliagusak/neural-ode-metasolver","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juliagusak%2Fneural-ode-metasolver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juliagusak%2Fneural-ode-metasolver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juliagusak%2Fneural-ode-metasolver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juliagusak%2Fneural-ode-metasolver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/juliagusak","download_url":"https://codeload.github.com/juliagusak/neural-ode-metasolver/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249494043,"owners_count":21281659,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-attacks","neural-network","neural-ode","ordinary-differential-equations","parametrized","pytorch","robustness","runge-kutta","solver"],"created_at":"2024-11-09T03:13:56.367Z","updated_at":"2025-04-18T12:33:10.721Z","avatar_url":"https://github.com/juliagusak.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Meta-Solver for Neural Ordinary Differential Equations\nTowards robust neural ODEs using parametrized solvers.\n\n# Main idea\nEach *Runge-Kutta (RK)* solver with `s` stages and of the `p`-th order is defined by a table of coefficients (*Butcher tableau*). For `s=p=2`, `s=p=3` and `s=p=4` all coefficient in the table can be parametrized with no more than two variables [1]. \n\nUsually, during neural ODE training RK solver with fixed Butcher tableau is used, and only the *right-hand side (RHS)* function is trained. We propose to use the whole parametric family of RK solvers to improve robustness of neural ODEs. \n\n# Requirements\n- pytorch==1.7\n- apex==0.1 (for training)\n\n# Examples\nFor CIFAR-10 and MNIST demo, please,  check  `examples` folder.\n\n# Meta Solver Regimes\nIn the notebook `examples/cifar10/Evaluate model.ipynb` we show how to perform the forward pass through the Neural ODE using different types of Meta Solver regimes, namely\n- Standalone\n- Solver switching/smoothing\n- Solver ensembling\n- Model ensembling\n\nIn more details, usage of different regimes means\n- **Standalone**\n    - Use one solver during  inference.\n    - This regime is applied in the training and testing stages.\n     \n    \n    \n- **Solver switching / smoothing**\n    - For each batch one solver is chosen from a group of solvers with finite (in switching regime) or infinite (in smoothing regime) number of candidates.\n    - This regime is applied in the training stage\n    \n    \n- **Solver ensembling**\n    - Use several solvers durung inference.\n    - Outputs of ODE Block (obtained with different solvers) are averaged before propagating through the next layer.\n    - This regime is applied in the training and testing stages.\n    \n    \n- **Model ensembling**\n    - Use several solvers durung inference.\n    - Model probabilites obtained via propagation with different solvers are averaged to get the final result.\n    - This regime is applied in the training and testing stages.\n    \n# Selected results\n## Different solver parameterizations yield different robustness\nWe have trained a neural ODE model several times, using different ``u`` values in parametrization of the 2-nd order Runge-Kutta solver. The image below depicts robust accuracies for the MNIST classification task. We use PGD attack (eps=0.3, lr=2/255 and iters=7). The mean values of robust accuracy (bold lines) and +- standard error mean (shaded region) computed across 9 random seeds are shown in this image.\n\n\u003cimg src=\"examples/mnist/assets/mnist_adv.png\" width=\"40%\"\u003e\n\n## Solver smoothing improves robustness\nWe compare results of neural ODE adversarial training on CIFAR-10 dataset with meta-solver in standalone, switching or smoothing regimes. We choose 8-steps RK2 solvers for this experiment.\n- We perform training using FGSM random technique described in https://arxiv.org/abs/2001.03994 (with eps=8/255, alpha=10/255). \n- We use cyclic learning rate schedule with one cycle (36 epochs, max_lr=0.1, base_lr=1e-7).\n- We measure robust accuracy of resulting models after FGSM (eps=8/255) and PGD (eps=8/255, lr=2/255, iters=7) attacks.\n- We use `premetanode10` architecture from `sopa/src/models/odenet_cifar10/layers.py` that has the following form \n`Conv -\u003e PreResNet block -\u003e ODE block -\u003e PreResNet block -\u003e ODE block -\u003e  GeLU -\u003e Average Pooling -\u003e Fully Connected`\n- We compute mean and standard error across 3 random seeds.\n\n\u003c!---\n![](examples/cifar10/assets/fgsm_random_train_fgsm_eps_8_255_test.png)\n![](examples/cifar10/assets/fgsm_random_train_pgd_eps_8_255_lr_2_255_iters_7_test.png)\n--\u003e\n\n![](examples/cifar10/assets/fgsm_random_train.png)\n\n# References\n[1] [Wanner, G., \u0026 Hairer, E. (1993). Solving ordinary differential equations I. Springer Berlin Heidelberg](https://www.springer.com/gp/book/9783540566700)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuliagusak%2Fneural-ode-metasolver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjuliagusak%2Fneural-ode-metasolver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuliagusak%2Fneural-ode-metasolver/lists"}