https://github.com/e-sensing/torchopt
R implementation of advanced optimizers for torch
https://github.com/e-sensing/torchopt
deep-learning numerical-optimization
Last synced: 9 days ago
JSON representation
R implementation of advanced optimizers for torch
- Host: GitHub
- URL: https://github.com/e-sensing/torchopt
- Owner: e-sensing
- Created: 2022-04-07T13:19:42.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2023-06-08T19:23:14.000Z (over 2 years ago)
- Last Synced: 2025-12-09T14:09:08.782Z (3 months ago)
- Topics: deep-learning, numerical-optimization
- Language: R
- Homepage:
- Size: 4.38 MB
- Stars: 26
- Watchers: 6
- Forks: 5
- Open Issues: 0
-
Metadata Files:
- Readme: README.Rmd
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
---
output: github_document
editor_options:
chunk_output_type: console
markdown:
wrap: 72
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
# torchopt
[](https://github.com/e-sensing/torchopt/actions)
[](https://cran.r-project.org/package=torchopt)
[](https://lifecycle.r-lib.org/articles/stages.html)
[](https://www.apache.org/licenses/LICENSE-2.0)
The `torchopt` package provides R implementation of deep learning optimizers proposed in the literature. It is intended to support the use of the torch package in R.
## Installation
Installing the CRAN (stable) version of `torchopt`:
```{r, eval = FALSE}
install.packages("torchopt")
```
Installing the development version of `torchopt` do as :
```{r, eval = FALSE}
library(devtools)
install_github("e-sensing/torchopt")
```
```{r, echo = FALSE}
library(torch)
if (!torch::torch_is_installed())
torch::install_torch()
library(torchopt)
```
## Provided optimizers
`torchopt` package provides the following R implementations of torch
optimizers:
- `optim_adamw()`: AdamW optimizer proposed by Loshchilov & Hutter
(2019). Converted from the `pytorch` code developed by Collin
Donahue-Oponski available at
- `optim_adabelief()`: Adabelief optimizer proposed by Zhuang et al
(2020). Converted from the authors' PyTorch code:
.
- `optim_adabound()`: Adabound optimizer proposed by Luo et al.(2019).
Converted from the authors' PyTorch code:
.
- `optim_adahessian()`: Adahessian optimizer proposed by Yao et al.(2021).
Converted from the authors' PyTorch code:
.
- `optim_madgrad()`: Momentumized, Adaptive, Dual Averaged Gradient
Method for Stochastic Optimization (MADGRAD) optimizer proposed by
Defazio & Jelassi (2021). The function is imported from
[madgrad](https://CRAN.R-project.org/package=madgrad) package and
the source code is available at
- `optim_nadam()`: Incorporation of Nesterov Momentum into Adam
proposed by Dozat (2016). Converted from the PyTorch site
.
- `optim_qhadam()`: Quasi-hyperbolic version of Adam proposed by Ma
and Yarats(2019). Converted from the code developed by Meta AI:
.
- `optim_radam()`: Rectified verison of Adam proposed by Liu et al.
(2019). Converted from the PyTorch code
.
- `optim_swats()`: Optimizer that switches from Adam to SGD proposed by
Keskar and Socher(2018).
Converted from the `pytorch` code developed by Patrik Purgai:
- `optim_yogi()`: Yogi optimizer proposed by Zaheer et al.(2019).
Converted from the `pytorch` code developed by Nikolay Novik:
## Optimization test functions
You can also test optimizers using optimization [test
functions](https://en.wikipedia.org/wiki/Test_functions_for_optimization)
provided by `torchopt` including `"ackley"`, `"beale"`, `"booth"`,
`"bukin_n6"`, `"easom"`, `"goldstein_price"`, `"himmelblau"`,
`"levi_n13"`, `"matyas"`, `"rastrigin"`, `"rosenbrock"`, `"sphere"`.
Optimization functions are useful to evaluate characteristics of
optimization algorithms, such as convergence rate, precision,
robustness, and performance. These functions give an idea about the
different situations that optimization algorithms can face.
In what follows, we perform tests using `"beale"` test function. To
visualize an animated GIF, we set `plot_each_step=TRUE` and capture each
step frame using [gifski](https://CRAN.R-project.org/package=gifski)
package.
### `optim_adamw()`:
```{r test_adamw, echo=TRUE, fig.show='animate', fig.height=8, fig.width=8, animation.hook='gifski', aniopts='loop', dpi=96, interval=0.1, out.height='50%', out.width='50%', cache=TRUE}
# test optim adamw
set.seed(12345)
torchopt::test_optim(
optim = torchopt::optim_adamw,
test_fn = "beale",
opt_hparams = list(lr = 0.1),
steps = 500,
plot_each_step = TRUE
)
```
### `optim_adabelief()`:
```{r test_adabelief, echo=TRUE, fig.show='animate', fig.height=8, fig.width=8, animation.hook='gifski', aniopts='loop', dpi=96, interval=0.1, out.height='50%', out.width='50%', cache=TRUE}
set.seed(42)
test_optim(
optim = optim_adabelief,
opt_hparams = list(lr = 0.5),
steps = 400,
test_fn = "beale",
plot_each_step = TRUE
)
```
### `optim_adabound()`:
```{r test_adabound, echo=TRUE, fig.show='animate', fig.height=8, fig.width=8, animation.hook='gifski', aniopts='loop', dpi=96, interval=0.1, out.height='50%', out.width='50%', cache=TRUE}
# set manual seed
set.seed(22)
test_optim(
optim = optim_adabound,
opt_hparams = list(lr = 0.5),
steps = 400,
test_fn = "beale",
plot_each_step = TRUE
)
```
### `optim_adahessian()`:
```{r test_adahessian, echo=TRUE, fig.show='animate', fig.height=8, fig.width=8, animation.hook='gifski', aniopts='loop', dpi=96, interval=0.1, out.height='50%', out.width='50%', cache=TRUE}
# set manual seed
set.seed(290356)
test_optim(
optim = optim_adahessian,
opt_hparams = list(lr = 0.2),
steps = 500,
test_fn = "beale",
plot_each_step = TRUE
)
```
### `optim_madgrad()`:
```{r test_madgrad, echo=TRUE, fig.show='animate', fig.height=8, fig.width=8, animation.hook='gifski', aniopts='loop', dpi=96, interval=0.1, out.height='50%', out.width='50%', cache=TRUE}
set.seed(256)
test_optim(
optim = optim_madgrad,
opt_hparams = list(lr = 0.05),
steps = 400,
test_fn = "beale",
plot_each_step = TRUE
)
```
### `optim_nadam()`:
```{r test_nadam, echo=TRUE, fig.show='animate', fig.height=8, fig.width=8, animation.hook='gifski', aniopts='loop', dpi=96, interval=0.1, out.height='50%', out.width='50%', cache=TRUE}
set.seed(2903)
test_optim(
optim = optim_nadam,
opt_hparams = list(lr = 0.5, weight_decay = 0),
steps = 500,
test_fn = "beale",
plot_each_step = TRUE
)
```
### `optim_qhadam()`:
```{r test_qhadam, echo=TRUE, fig.show='animate', fig.height=8, fig.width=8, animation.hook='gifski', aniopts='loop', dpi=96, interval=0.1, out.height='50%', out.width='50%', cache=TRUE}
set.seed(1024)
test_optim(
optim = optim_qhadam,
opt_hparams = list(lr = 0.1),
steps = 500,
test_fn = "beale",
plot_each_step = TRUE
)
```
### `optim_radam()`:
```{r test_radam, echo=TRUE, fig.show='animate', fig.height=8, fig.width=8, animation.hook='gifski', aniopts='loop', dpi=96, interval=0.1, out.height='50%', out.width='50%', cache=TRUE}
set.seed(1024)
test_optim(
optim = optim_radam,
opt_hparams = list(lr = 1.0),
steps = 500,
test_fn = "beale",
plot_each_step = TRUE
)
```
### `optim_swats()`:
```{r test_swats, echo=TRUE, fig.show='animate', fig.height=8, fig.width=8, animation.hook='gifski', aniopts='loop', dpi=96, interval=0.1, out.height='50%', out.width='50%', cache=TRUE}
set.seed(234)
test_optim(
optim = optim_swats,
opt_hparams = list(lr = 0.5),
steps = 500,
test_fn = "beale",
plot_each_step = TRUE
)
```
### `optim_yogi()`:
```{r test_yogi, echo=TRUE, fig.show='animate', fig.height=8, fig.width=8, animation.hook='gifski', aniopts='loop', dpi=96, interval=0.1, out.height='50%', out.width='50%', cache=TRUE}
# set manual seed
set.seed(66)
test_optim(
optim = optim_yogi,
opt_hparams = list(lr = 0.1),
steps = 500,
test_fn = "beale",
plot_each_step = TRUE
)
```
## Acknowledgements
We are thankful to Collin Donahue-Oponski ,
Amir Gholami ,
Liangchen Luo , Liyuan Liu
, Nikolay Novik , Patrik Purgai Juntang Zhuang and the PyTorch team for providing pytorch code for the optimizers implemented in this package. We also thank Daniel Falbel for providing support
for the R version of PyTorch.
## Code of Conduct
The torchopt project is released with a [Contributor
Code of Conduct](https://contributor-covenant.org/version/2/0/CODE_OF_CONDUCT.html).
By contributing to this project, you agree to abide by its terms.
## References
- ADABELIEF: Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar Tatikonda, Nicha
Dvornek, Xenophon Papademetris, James S. Duncan. "Adabelief
Optimizer: Adapting Stepsizes by the Belief in Observed Gradients",
34th Conference on Neural Information Processing Systems (NeurIPS
2020), .
- ADABOUND: Liangchen Luo, Yuanhao Xiong, Yan Liu, Xu Sun, "Adaptive Gradient
Methods with Dynamic Bound of Learning Rate", International
Conference on Learning Representations (ICLR), 2019.
.
- ADAHESSIAN: Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer,
Michael W. Mahoney. "Adahessian: An Adaptive Second Order Optimizer
for Machine Learning", AAAI Conference on Artificial Intelligence, 35(12),
10665-10673, 2021. .
- ADAMW: Ilya Loshchilov, Frank Hutter, "Decoupled Weight Decay
Regularization", International Conference on Learning
Representations (ICLR) 2019.
.
- MADGRAD: Aaron Defazio, Samy Jelassi, "Adaptivity without Compromise: A
Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic
Optimization", arXiv preprint arXiv:2101.11075, 2021.
- NADAM: Timothy Dazat, "Incorporating Nesterov Momentum into Adam",
International Conference on Learning Representations (ICLR), 2019.
- QHADAM: Jerry Ma, Denis Yarats, "Quasi-hyperbolic momentum and Adam
for deep learning".
- RADAM: Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu,
Jianfeng Gao, Jiawei Han, "On the Variance of the Adaptive Learning
Rate and Beyond", International Conference on Learning
Representations (ICLR) 2020. .
- SWATS: Nitish Keskar, Richard Socher, "Improving Generalization Performance
by Switching from Adam to SGD".
International Conference on Learning Representations (ICLR), 2018.
.
- YOGI: Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, Sanjiv
Kumar, "Adaptive Methods for Nonconvex Optimization", Advances in
Neural Information Processing Systems 31 (NeurIPS 2018).