{"id":15362518,"url":"https://github.com/patrick-kidger/fasterneuraldiffeq","last_synced_at":"2025-11-17T15:35:02.104Z","repository":{"id":39254190,"uuid":"287221765","full_name":"patrick-kidger/FasterNeuralDiffEq","owner":"patrick-kidger","description":"Code for \"'Hey, that's not an ODE:' Faster ODE Adjoints via Seminorms\" (ICML 2021)","archived":false,"fork":false,"pushed_at":"2022-10-19T00:21:18.000Z","size":650,"stargazers_count":87,"open_issues_count":1,"forks_count":9,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-15T08:59:40.239Z","etag":null,"topics":["controlled-differential-equations","deep-learning","deep-neural-networks","differential-equations","dynamical-systems","machine-learning","neural-differential-equations","numerical-analysis","numerical-methods","ordinary-differential-equations","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/patrick-kidger.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-13T08:19:06.000Z","updated_at":"2025-03-25T07:05:39.000Z","dependencies_parsed_at":"2023-01-19T17:04:45.524Z","dependency_job_id":null,"html_url":"https://github.com/patrick-kidger/FasterNeuralDiffEq","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/patrick-kidger/FasterNeuralDiffEq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrick-kidger%2FFasterNeuralDiffEq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrick-kidger%2FFasterNeuralDiffEq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrick-kidger%2FFasterNeuralDiffEq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrick-kidger%2FFasterNeuralDiffEq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/patrick-kidger","download_url":"https://codeload.github.com/patrick-kidger/FasterNeuralDiffEq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patrick-kidger%2FFasterNeuralDiffEq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284911788,"owners_count":27083425,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-17T02:00:06.431Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["controlled-differential-equations","deep-learning","deep-neural-networks","differential-equations","dynamical-systems","machine-learning","neural-differential-equations","numerical-analysis","numerical-methods","ordinary-differential-equations","pytorch"],"created_at":"2024-10-01T13:02:05.832Z","updated_at":"2025-11-17T15:35:02.074Z","avatar_url":"https://github.com/patrick-kidger.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align='center'\u003e \"Hey, that's not an ODE\": Faster ODE Adjoints via Seminorms\u003cbr\u003e(ICML 2021)\u003cbr\u003e[\u003ca href=\"https://arxiv.org/abs/2009.09457\"\u003earXiv\u003c/a\u003e]\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cimg align=\"middle\" src=\"./imgs/nfe.png\" width=\"666\" /\u003e\n\u003c/p\u003e\n\nOne simple-to-implement trick dramatically improves the speed at which [Neural ODEs](https://arxiv.org/abs/1806.07366) and [Neural CDEs](https://arxiv.org/abs/2005.08926) can be trained. (As much as doubling the speed.)\n\nBackpropagation through a Neural ODE/CDE can be performed via the \"adjoint method\", which involves solving another differential equation backwards in time. However it turns out that default numerical solvers are unnecessarily stringent when solving the adjoint equation, and take too many steps, that are too small.\n\nTweaking things slightly reduces the number of function evaluations on the backward pass **by as much as 62%**. (Exact number will be problem-dependent, of course.)\n\n_[torchdiffeq](https://github.com/rtqichen/torchdiffeq) now supports this feature natively!_\n\n---\n\n## Summary:\nIf you're using [torchdiffeq](https://github.com/rtqichen/torchdiffeq) (at least version 0.1.0) then replace\n```python\nimport torchdiffeq\n\nfunc = ...\ny0 = ...\nt = ...\ntorchdiffeq.odeint_adjoint(func=func, y0=y0, t=t)\n```\nwith\n```python\nimport torchdiffeq\n\ndef rms_norm(tensor):\n    return tensor.pow(2).mean().sqrt()\n\ndef make_norm(state):\n    state_size = state.numel()\n    def norm(aug_state):\n        y = aug_state[1:1 + state_size]\n        adj_y = aug_state[1 + state_size:1 + 2 * state_size]\n        return max(rms_norm(y), rms_norm(adj_y))\n    return norm\n\nfunc = ...    \ny0 = ...\nt = ...\ntorchdiffeq.odeint_adjoint(func=func, y0=y0, t=t, \n                           adjoint_options=dict(norm=make_norm(y0)))\n```\nThat's it.\n\n## Reproducing experiments\nThe code for the Neural CDE and Symplectic ODE-Net experiments is available.\n\n### Requirements\nPyTorch \u003e= 1.6    \n[torchdiffeq](https://github.com/rtqichen/torchdiffeq) \u003e= 0.1.0  \n[torchcde](https://github.com/patrick-kidger/torchcde) \u003e= 0.1.0  \n[torchaudio](pytorch.org/audio/) \u003e= 0.6.0  \n[sklearn](https://scikit-learn.org/stable/) \u003e= 0.23.1  \n[gym](https://github.com/openai/gym) \u003e= 0.17.2  \n[tqdm](https://github.com/tqdm/tqdm) \u003e= 4.47.0\n\nIn summary:\n```bash\nconda install pytorch torchaudio -c pytorch\npip install torchdiffeq scikit-learn gym tqdm\npip install git+https://github.com/patrick-kidger/torchcde.git\n```\n\n### Neural CDEs\n```bash\npython\n\u003e\u003e\u003e import speech_commands\n\u003e\u003e\u003e device = 'cuda'\n\u003e\u003e\u003e norm = False  # don't use our trick\n\u003e\u003e\u003e norm = True   # use our trick\n\u003e\u003e\u003e rtol = 1e-3\n\u003e\u003e\u003e atol = 1e-5\n\u003e\u003e\u003e results = speech_commands.main(device, norm, rtol, atol)\n\u003e\u003e\u003e print(results.keys())  # inspect results object\n\u003e\u003e\u003e print(results.test_metrics.accuracy)  # query results object\n```\n\n### Symplectic ODE-Net\n```bash\npython\n\u003e\u003e\u003e import acrobot\n\u003e\u003e\u003e device = 'cuda'\n\u003e\u003e\u003e norm = False  # don't use our trick\n\u003e\u003e\u003e norm = True   # use our trick\n\u003e\u003e\u003e results = acrobot.main(device, norm)\n\u003e\u003e\u003e print(results.keys())  # inspect results object\n\u003e\u003e\u003e print(results.test_metrics.loss)  # query results object\n```\n---\n\n## Citation\n```bibtex\n@article{kidger2021hey,\n    author={Kidger, Patrick and Chen, Ricky T. Q. and Lyons, Terry},\n    title={{``Hey, that's not an ODE'': Faster ODE Adjoints via Seminorms}},\n    year={2021},\n    journal={International Conference on Machine Learning}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatrick-kidger%2Ffasterneuraldiffeq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpatrick-kidger%2Ffasterneuraldiffeq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatrick-kidger%2Ffasterneuraldiffeq/lists"}