{"id":13473184,"url":"https://github.com/kach/gradient-descent-the-ultimate-optimizer","last_synced_at":"2025-04-05T11:07:11.958Z","repository":{"id":61571689,"uuid":"545626496","full_name":"kach/gradient-descent-the-ultimate-optimizer","owner":"kach","description":"Code for our NeurIPS 2022 paper","archived":false,"fork":false,"pushed_at":"2023-01-13T10:09:13.000Z","size":6,"stargazers_count":366,"open_issues_count":1,"forks_count":21,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-29T10:05:18.090Z","etag":null,"topics":["deep-learning","differentiable-programming","machine-learning","neurips","neurips-2022","pytorch"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1909.13371","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kach.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-04T17:49:21.000Z","updated_at":"2025-03-27T21:43:25.000Z","dependencies_parsed_at":"2023-02-09T15:15:32.412Z","dependency_job_id":null,"html_url":"https://github.com/kach/gradient-descent-the-ultimate-optimizer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kach%2Fgradient-descent-the-ultimate-optimizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kach%2Fgradient-descent-the-ultimate-optimizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kach%2Fgradient-descent-the-ultimate-optimizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kach%2Fgradient-descent-the-ultimate-optimizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kach","download_url":"https://codeload.github.com/kach/gradient-descent-the-ultimate-optimizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247325693,"owners_count":20920714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","differentiable-programming","machine-learning","neurips","neurips-2022","pytorch"],"created_at":"2024-07-31T16:01:01.506Z","updated_at":"2025-04-05T11:07:11.919Z","avatar_url":"https://github.com/kach.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Gradient Descent: The Ultimate Optimizer\n\n![gdtuo_turtles](https://user-images.githubusercontent.com/31300675/193727211-bff82331-998c-4d44-b675-03d1fd222e0e.png)\n# Abstract\nWorking with any gradient-based machine learning algorithm involves the tedious task of tuning the optimizer's hyperparameters, such as the step size. Recent work has shown how the step size can itself be \"learned\" on-line by gradient descent, by manually deriving expressions for \"hypergradients\" ahead of time.\n\nWe show how to *automatically* compute hypergradients with a simple and elegant modification to backpropagation. This allows us to apply the method to other hyperparameters besides the step size, such as the momentum coefficient. We can even recursively apply the method to its own *hyper*-hyperparameters, and so on *ad infinitum*. As these towers of optimizers grow taller, they become less sensitive to the initial choice of hyperparameters. We present experiments validating this for MLPs, CNNs, and RNNs.\n\n*This repository contains an implementation of the algorithm in our paper.*\n\n# Citation\n```\n@article{chandra2022gradient,\n    title = {Gradient Descent: The Ultimate Optimizer},\n    author = {Chandra, Kartik and Xie, Audrey and Ragan-Kelley, Jonathan and Meijer, Erik},\n    journal = {NeurIPS},\n    year = {2022},\n    url = {https://arxiv.org/abs/1909.13371}\n}\n```\n\n# Install\n```\n# install pytorch for your specific machine\n...\n\n# install our package\npip install gradient-descent-the-ultimate-optimizer\n```\n# Example\nFirst, build the MLP and initialize data loaders as you would normally in PyTorch.\n```python\nimport math\nimport torch\nimport torchvision\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nclass MNIST_FullyConnected(nn.Module):\n    \"\"\"\n    A fully-connected NN for the MNIST task. This is Optimizable but not itself\n    an optimizer.\n    \"\"\"\n    def __init__(self, num_inp, num_hid, num_out):\n        super(MNIST_FullyConnected, self).__init__()\n        self.layer1 = nn.Linear(num_inp, num_hid)\n        self.layer2 = nn.Linear(num_hid, num_out)\n\n    def initialize(self):\n        nn.init.kaiming_uniform_(self.layer1.weight, a=math.sqrt(5))\n        nn.init.kaiming_uniform_(self.layer2.weight, a=math.sqrt(5))\n\n    def forward(self, x):\n        \"\"\"Compute a prediction.\"\"\"\n        x = self.layer1(x)\n        x = torch.tanh(x)\n        x = self.layer2(x)\n        x = torch.tanh(x)\n        x = F.log_softmax(x, dim=1)\n        return x\n\nBATCH_SIZE = 256\nEPOCHS = 5\nDEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'\n\nmnist_train = torchvision.datasets.MNIST('./data', train=True, download=True, transform=torchvision.transforms.ToTensor())\nmnist_test = torchvision.datasets.MNIST('./data', train=False, download=True, transform=torchvision.transforms.ToTensor())\ndl_train = torch.utils.data.DataLoader(mnist_train, batch_size=BATCH_SIZE, shuffle=True)\ndl_test = torch.utils.data.DataLoader(mnist_test, batch_size=10000, shuffle=False)\n\nmodel = MNIST_FullyConnected(28 * 28, 128, 10).to(DEVICE)\n```\nNext, import our package and initialize a stack of hyperoptimizers. This example uses the stack `Adam/SGD`.\n```python\nfrom gradient_descent_the_ultimate_optimizer import gdtuo\n\noptim = gdtuo.Adam(optimizer=gdtuo.SGD(1e-5))\n```\n`gdtuo.ModuleWrapper` allows any `nn.Module` to be optimized by hyperoptimizers.\n```python\nmw = gdtuo.ModuleWrapper(model, optimizer=optim)\nmw.initialize()\n```\nLastly, use `mw` instead of a PyTorch optimizer to optimize the model. The train loop is nearly identical to what you would typically implement in PyTorch (differences are marked by comments).\n```python\nfor i in range(1, EPOCHS+1):\n    running_loss = 0.0\n    for j, (features_, labels_) in enumerate(dl_train):\n        mw.begin() # call this before each step, enables gradient tracking on desired params\n        features, labels = torch.reshape(features_, (-1, 28 * 28)).to(DEVICE), labels_.to(DEVICE)\n        pred = mw.forward(features)\n        loss = F.nll_loss(pred, labels)\n        mw.zero_grad()\n        loss.backward(create_graph=True) # important! use create_graph=True\n        mw.step()\n        running_loss += loss.item() * features_.size(0)\n    train_loss = running_loss / len(dl_train.dataset)\n    print(\"EPOCH: {}, TRAIN LOSS: {}\".format(i, train_loss))\n```\nNote that on the first step of the train loop PyTorch will return the following warning:\n```\nUserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak.\n```\nThis is normal and to be expected.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkach%2Fgradient-descent-the-ultimate-optimizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkach%2Fgradient-descent-the-ultimate-optimizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkach%2Fgradient-descent-the-ultimate-optimizer/lists"}