{"id":29630913,"url":"https://github.com/epfml/grad-norm-smooth","last_synced_at":"2025-07-21T11:07:10.243Z","repository":{"id":299580090,"uuid":"1003151995","full_name":"epfml/grad-norm-smooth","owner":"epfml","description":"Official implementation of \"Gradient-Normalized Smoothness for Optimization with Approximate Hessians\"","archived":false,"fork":false,"pushed_at":"2025-06-17T08:12:38.000Z","size":624,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-17T09:30:10.685Z","etag":null,"topics":["inexact-hessians","second-order-optimization"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2506.13710","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/epfml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-16T17:49:23.000Z","updated_at":"2025-06-17T08:12:52.000Z","dependencies_parsed_at":"2025-06-17T09:31:00.578Z","dependency_job_id":"1701d6b0-5738-4f0c-bbe5-4a99f7d62f60","html_url":"https://github.com/epfml/grad-norm-smooth","commit_stats":null,"previous_names":["epfml/grad-norm-smooth"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/epfml/grad-norm-smooth","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Fgrad-norm-smooth","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Fgrad-norm-smooth/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Fgrad-norm-smooth/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Fgrad-norm-smooth/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/epfml","download_url":"https://codeload.github.com/epfml/grad-norm-smooth/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Fgrad-norm-smooth/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266287824,"owners_count":23905461,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["inexact-hessians","second-order-optimization"],"created_at":"2025-07-21T11:07:09.569Z","updated_at":"2025-07-21T11:07:10.231Z","avatar_url":"https://github.com/epfml.png","language":"Jupyter Notebook","readme":"# Supplementary code for \"Gradient-Normalized Smoothness for Optimization with Approximate Hessians\"\n[![arXiv](https://img.shields.io/badge/arXiv-2401.06766-b31b1b.svg)](https://arxiv.org/abs/2506.13710)\n\nThis code comes jointly with reference:\n\n\u003e Andrei Semenov, Martin Jaggi, Nikita Doikov.\n\nDate:    June 2025\n\n## Structure\n\n```sh\nsrc/\n    methods.py         # Algorithm 1 from the paper, algorithms with other adaptive search schemes, gradient methods\n    oracles.py         # LogSumExp, Nonlinear Equations with linear operator and Chebyshev polynomials,  Rosenbrock function, etc.\n    approximations.py  # code for Hessian approximations for different oracles\n    utils.py           # code for plotting graphs\n    data/\n        mushrooms.txt  # example of a dataset; you can add here more\nnotebooks/\n    examples.ipynb     # examples of approximations and comparison of methods\n```\n\n## Quickstart\n\nSimply run the ```examples.ipynb``` notebook.\nAt the beginning of the notebook, we provide practical approximations for each oracle.\nAll of them are compatible with our theory.\nIn particular, we investigated the following approximations.\n\n| Problem | Naming in the paper | Approximation | Code reference in ```src/approximations.py```|\n|---------|------|--------|-----|\n| LogSumExp | Weighted Gauss-Newton |  $\\frac{1}{\\mu}\\mathbf{A}^\\top Diag\\left(\\mathrm{softmax}\\left(\\mathbf{A}, x\\right)\\right)\\mathbf{A}$ | [```approx_hess_fn_logsumexp```](https://github.com/epfml/hess-approx/blob/0d294d9b65dc6bffb1434994abad8fba5a3aa7dd/src/approximations.py#L6) |\n| Equations with linear operator | Fisher Term of $\\mathbf{H}$ | $\\frac{p-2}{\\lVert u(x) \\rVert^p} \\nabla f(x) \\nabla f(x)^\\top$ | [```approx_hess_fn_fisher_term```](https://github.com/epfml/hess-approx/blob/0d294d9b65dc6bffb1434994abad8fba5a3aa7dd/src/approximations.py#L16) |\n| Nonlinear Equations \u0026 Rosenbrock | Inexact Hessian | $\\lVert u(x)\\rVert^{p - 2} \\nabla u(x)^\\top \\mathbf{B} \\nabla u(x) + \\frac{p - 2}{\\lVert u(x) \\rVert^p} \\nabla f(x) \\nabla f(x)^{\\top}$ | [```approx_hess_nonlinear_equations```](https://github.com/epfml/hess-approx/blob/0d294d9b65dc6bffb1434994abad8fba5a3aa7dd/src/approximations.py#L31)|\n|Nonlinear Equations \u0026 Chebyshev polynomials | Inexact Hessian | $\\lVert u(x) \\rVert^{p - 2} \\nabla u(x)^\\top \\mathbf{B} \\nabla u(x) + \\frac{p - 2}{\\lVert u(x) \\rVert^p} \\nabla f(x) \\nabla f(x)^{\\top}$ | [```approx_hess_fn_chebyshev```](https://github.com/epfml/hess-approx/blob/0d294d9b65dc6bffb1434994abad8fba5a3aa7dd/src/approximations.py#L51) |\n\nYou can also use a fast implementation of our algorithm, which corresponds to ```grad_norm_smooth_for_rank_one``` function in ```examples.py```.\nThus, you could obtain the following nice examples:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/fisher-rd-p4.png\" alt=\"Fisher\" width=\"45%\" style=\"display:inline-block; margin-right: 10px;\"/\u003e\n  \u003cimg src=\"assets/fisher-rd-p4-time.png\" alt=\"Fisher time\" width=\"47.5%\" style=\"display:inline-block;\"/\u003e\n\u003c/p\u003e\n\n\n**We believe the details provided are clear enough to reproduce the main findings of our paper.**\n\n```bib\n@misc{semenov2025gradientnormalizedsmoothnessoptimizationapproximate,\n      title={Gradient-Normalized Smoothness for Optimization with Approximate Hessians}, \n      author={Andrei Semenov and Martin Jaggi and Nikita Doikov},\n      year={2025},\n      eprint={2506.13710},\n      archivePrefix={arXiv},\n      primaryClass={math.OC},\n      url={https://arxiv.org/abs/2506.13710}, \n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepfml%2Fgrad-norm-smooth","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fepfml%2Fgrad-norm-smooth","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepfml%2Fgrad-norm-smooth/lists"}