{"id":35142302,"url":"https://github.com/inikishev/visualbench","last_synced_at":"2026-03-27T03:11:00.199Z","repository":{"id":318686448,"uuid":"1069599037","full_name":"inikishev/visualbench","owner":"inikishev","description":"Fast benchmarks for optimization algorithms with visualizations. Supports pytorch optimizers as well as solvers from other libraries such as scipy.optimize, optuna, etc","archived":false,"fork":false,"pushed_at":"2026-02-18T17:07:08.000Z","size":174258,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-18T19:10:07.168Z","etag":null,"topics":["benchmark","benchmarks","machine-learning","optimization","python","test-problems","visualization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/inikishev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-04T08:45:52.000Z","updated_at":"2026-02-18T17:12:09.000Z","dependencies_parsed_at":"2025-10-08T17:55:21.072Z","dependency_job_id":null,"html_url":"https://github.com/inikishev/visualbench","commit_stats":null,"previous_names":["inikishev/visualbench"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/inikishev/visualbench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inikishev%2Fvisualbench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inikishev%2Fvisualbench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inikishev%2Fvisualbench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inikishev%2Fvisualbench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/inikishev","download_url":"https://codeload.github.com/inikishev/visualbench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inikishev%2Fvisualbench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31013962,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-27T02:58:54.984Z","status":"ssl_error","status_checked_at":"2026-03-27T02:58:46.993Z","response_time":164,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","benchmarks","machine-learning","optimization","python","test-problems","visualization"],"created_at":"2025-12-28T12:05:12.260Z","updated_at":"2026-03-27T03:11:00.174Z","avatar_url":"https://github.com/inikishev.png","language":"Python","readme":"# \u003ch1 align='center'\u003evisualbench\u003c/h1\u003e\n\nFast benchmarks for optimization algorithms - PyTorch optimizers as well as solvers from any other libraries such as scipy.optimize, optuna, etc.\n\nMany benchmarks support visualization where you can plot or render a video to see how the solution is being optimized.\n\n### Installation\n\n```bash\npip install visualbench\n```\n\nThe dependencies are `pytorch`, `numpy`, `scipy`, `matplotlib` and `opencv-python`.\n\nFew benchmarks also use `torchvision`, `scikit-learn`, `mnist1d`, `gpytorch`.\n\n### Function descent\n\nUseful to debug optimizers:\n\n```python\nimport torch\nimport visualbench as vb\n\n# \"booth\" is a pre-defined function\n# can also pass custom one like `lambda x, y: x**2 + y**2`\nbench = vb.FunctionDescent(\"booth\")\nopt = torch.optim.Adam(bench.parameters(), 1e-1)\n\nbench.run(opt, max_steps=1000)\n```\n\nwe can now plot a visualization:\n\n```python\nbench.plot()\n```\n\n\u003cimg width=\"450\" height=\"auto\" alt=\"image\" src=\"https://github.com/user-attachments/assets/7c561126-c2ed-4476-ae5f-c1b374f0e9f3\" /\u003e\n\nor render it to MP4/GIF (I recommend MP4 because its much faster to render)\n\n```python\nbench.render(\"Adam.mp4\")\n```\n\n\u003cimg width=\"400\" height=\"auto\" alt=\"image\" src=\"https://github.com/inikishev/visualbench/blob/main/assets/readme/Adam.gif\" /\u003e\n\n### Colorization\n\nHere is GD with momentum on the colorization problem from \u003chttps://distill.pub/2017/momentum/\u003e. The objective is to minimize differences between adjacent pixels.\n\n```python\nbench = vb.Colorization().cuda()\nopt = torch.optim.SGD(bench.parameters(), lr=2e-1, momentum=0.999)\nbench.run(opt, 1000)\nbench.render(\"Colorization.mp4\")\n```\n\n\u003chttps://github.com/user-attachments/assets/37b32d75-6f80-4c6b-a360-254aea368aa8\u003e\n\n### NeuralDrawer\n\nIn this objective the goal is to train a neural network which predicts pixel color of a given image based on its two coordinates:\n\n```python\nimport heavyball\n\nbench = vb.NeuralDrawer(\n    vb.data.WEEVIL96, # path to an image file, or a numpy array/torch tensor\n    vb.models.MLP([2,16,16,16,16,16,16,16,3], bn=True), # neural net\n    expand=48 # renders 48 pixels outside of the image\n).cuda() # don't forget to move to CUDA!\n\nopt = heavyball.ForeachSOAP(bench.parameters(), lr=1e-2)\nbench.run(opt, 1000)\nbench.render(\"NeuralDrawer.mp4\", scale=2)\n```\n\n\u003chttps://github.com/user-attachments/assets/99031a4f-d2aa-4640-b940-dc87c3316fdb\u003e\n\n# \u003ch1 align='center'\u003eProblems\u003c/h1\u003e\n\n#### Linear algebra\n\nSo for example SVD decomposes A into USV*, where U and V are orthonormal unitary, S is diagonal. So in `SVD` benchmark we optimize U, S and V so that USV* approximates A, and so that U and V are orthonormal.\n\nStochastic versions usually work by using matrix-vector products with random vectors. For example in `StochasticMatrixRecovery` we optimize B to recover A by using loss = mse(Av, Bv) where v is a vector sampled randomly on each step. It also computes test loss as mse(A, B). Those are very fast to evaluate and might be good proxies for real stochastic ML tasks.\n\nAll of them (may be slightly outdated):\n\n`LDL, LU, LUP, NNMF, QR, SVD, Cholesky, Eigendecomposition, KroneckerFactorization, Polar, RankFactorization, Drazin, Inverse, MoorePenrose, StochasticInverse, StochasticRLstsq, Preconditioner, StochasticPreconditioner, LeastSquares, MatrixIdempotent, MatrixLogarithm, MatrixRoot, StochasticMatrixIdempotent, StochasticMatrixRoot, StochasticMatrixSign, StochasticMatrixRecovery, BilinearLeastSquares, TensorRankDecomposition, TensorSpectralNorm`\n\n#### Drawing\n\n- `LinesDrawer`, `CurvesDrawer` - optimize lines to draw an image.\n- `PartitionDrawer`, `VoronoiDrawer` - optimize partitions draw an image.\n- `GaborDrawer` - Optimize Gabor filters (sinusoidal gratings) to draw an image.\n- `RectanglesDrawer`, `TrianglesDrawer`, `RectanglesDrawer` - optimize shapes to draw an image\n- `NeuralDrawer` - neural net predicts pixel color based on its two coordinates\n- `LayerwiseNeuralDrawer` - same as `NeuralDrawer` but it also visualizes all intermediate layers\n\n#### 2D functions\n\nYou can pass a function like `lambda x,y: x**2 + y**2`, or string name of one of pre-defined functions of which there are many, I usually use `\"booth\"`, `\"rosen\"`, and `\"ill\"` which is a rotated ill-conditioned quadratic.\n\n- `FunctionDescent` - to see how optimizer descends a 2D function.\n- `DecisionSpaceDescent` - optimize a model to output coordinates that minimize a 2D function. This is a great way to test how much curvature an optimizer actually uses on larger models.\n- `SimultaneousFunctionDescent` - same as FunctionDescent, except the optimizer optimizes all points at the same time.\n- `MultiAgentDescent` - multiple points descend a 2D image with optional repulsion.\n- `MetaLearning` - the goal is to optimize hyperparameters of an optimizer to descend a 2D function.\n\n#### Packing / Covering\n\n`RigidBoxPacking`, `SpherePacking`\n\n#### Projected objectives\n\nThis projects the trajectory of an optimizer on some problem, like neural network training, to a 2D subspace, and shows how optimizer navigates the landscape. It's actually very hard to get a good projection that doesn't bounce around and that is at least somewhat informative. I ended up using orthogonalized subspace defined by best point so far, point 5% before and point 10% before, with smoothing. On multi-dimensional rosenbrock it looks good, but neural net is still too chaotic.\n\nAll of those are in `vb.projected` namespace e.g. `vb.projected.Rosenbrock`:\n\n`Ackley`, `BumpyBowl`, `ChebushevRosenbrock`,\n`NeuralNet`, `Rastrigin`, `Rosenbrock`, `RotatedQuadratic`,\n\n#### Datasets\n\nTraining models on various datasets. Those benchmarks are basically as fast as they can be as datasets are pre-loaded into memory and use a custom very fast dataloader for mini-batching.\n\nIn all of them it says shape of input and output in the docstring. So you need to specify any model (`torch.nn.Module`) that receives and outputs those shapes, or use something from `vb.models`.\n\nDatasets with two features (like `Moons`) support visualizing/rendering the decision boundary.\n\n- sklearn datasets (requires `scikit-learn`): `CaliforniaHousing`, `Moons`, `OlivettiFaces`, `OlivettiFacesAutoencoding`, `Covertype`, `KDDCup1999`, `Digits`, `Friedman1`, `Friedman2`, `Friedman3`\n- mnist1d (requires `mnist1d`): `Mnist1d`, `Mnist1dAutoencoding`\n- Segmentation: `SynthSeg1d` (synthetic 1d semantic segmentation dataset)\n- torchvision: `MNIST`, `FashionMNIST`, `FashionMNISTAutoencoding`, `EMNIST`, `CIFAR10`, `CIFAR100`\n- other: `WDBC`\n\n#### Other machine learning\n\n- `TSNE` - T-SNE dimensionality reduction with visualization\n- `Glimmer` - Glimmer dimensionality reduction with visualization\n- `GaussianMixtureNLL` - optimize a gaussian mixture and visualizes PCA-projected decision boundaries\n- `MFMovieLens` - matrix factorization on MovieLens dataset\n- `WavePINN` - trains PINN on wave PDE\n- `AffineRegistration`, `DeformableRegistration` - image registration (2D and 3D)\n- `StyleTransfer` - VGG style transfer\n- `GaussianProcesses` (reguires GPytorch) - reconstruct 2D function with GP\n\n#### Synthetic problems\n\n- `Sphere`, `Rosenbrock`, `ChebushevRosenbrock`, `RotatedQuadratic`, `Rastrigin`, `Ackley`.\n\n#### Uncategorized problems\n\n- `AlphaEvolveB1` - Alpha Evolve B1 problem (code from \u003chttps://github.com/damek/alpha_evolve_problem_B1/blob/main/problemb1.ipynb\u003e), with visualization\n- `MuonCoeffs` - optimize Muon coefficients, this is \u003chttps://leloykun.github.io/ponder/muon-opt-coeffs/\u003e ported to pytorch\n- `HumanHeartDipole`, `PropaneCombustion` - two real-world least squares problems from MINPACK2\n- `Colorization` - colorization problem from \u003chttps://distill.pub/2017/momentum/\u003e\n- `GraphLayout` - optimize graph layout\n- `OptimalControl` - optimize trajectory\n- `CUTEst` - wrapper for CUTEst (requires `pycustest`), with a custom torch.autograd.Function function that wraps CUTEst's gradients and hessian-vector products.\n\n\u003c!-- # \u003ch1 align='center'\u003eGallery\u003c/h1\u003e\n\nI have to make this repo public to enable github pages, so those links are temorarily empty!\n\n- [More videos](wait)\n\n- [How much curvature do second order optimizers actually use?](wait) --\u003e\n\n# \u003ch1 align='center'\u003eAdvanced\u003c/h1\u003e\n\n### Custom training loops\n\nBenchmarks have `closure` method which returns the loss and optionally computes the gradients. This way one can write a custom training loop:\n\n```py\nbench = vb.Mnist1d(\n    vb.models.MLP([40, 64, 96, 128, 256, 10], act_cls=torch.nn.ELU),\n    batch_size=32, test_batch_size=None,\n).cuda()\n\n# test every 10 forward passes\nbench.set_test_intervals(test_every_forwards=10)\n\n# disable printing\nbench.set_print_inverval(None)\n\n\nopt = torch.optim.AdamW(bench.parameters(), 3e-3)\n\nfor step in range(1000):\n    opt.zero_grad()\n    loss = bench.closure(backward=False)\n    loss.backward()\n    opt.step()\n\n\nprint(f'{loss = }')\nbench.plot()\n```\n\n### Non-pytorch optimizers\n\nSolvers from other libraries can also be benchmarked/visualized easily.\n\nMany solvers work with numpy vectors, so we can get all parameters of a benchmark concatenated to a vector like this:\n\n```python\nx0 = bench.get_x0().numpy(force=True)\n```\n\nTo evaluate benchmark at parameters given in vector `x`, use `fx = bench.loss_at(x)`, `fx` will be a float.\n\nTo evaluate loss and gradient, use `fx, gx = bench.loss_grad_at(x)`. Here `gx` is a numpy array of the same length as `x`.\n\nUsing this, we can easily run solvers from other frameworks, for example scipy.optimize:\n\n```python\nimport scipy.optimize\n\nbench = vb.NeuralDrawer(\n    vb.data.WEEVIL96,\n    vb.models.MLP([2,16,16,16,16,16,16,16,3], bn=True),\n    expand=48\n).cuda()\n\nx0 = bench.get_x0().numpy(force=True)\n\nsol = scipy.optimize.minimize(\n    fun = bench.loss_grad_at, # or `bench.loss_at` if gradient-free\n    x0 = bench.get_x0().numpy(force=True),\n    method = 'l-bfgs-b',\n    jac = True, # fun returns (fx, gx)\n    options = {\"maxiter\": 1000}\n)\n\nbench.plot()\n```\n\nHere is how to visualize optuna's TPE sampler on rosenbrock function:\n\n```python\nimport numpy as np\nimport optuna\noptuna.logging.disable_default_handler() # hides very useful information\n\nbench = vb.FunctionDescent('rosen')\n\nsampler = optuna.samplers.TPESampler(prior_weight = 2.0,)\nstudy = optuna.create_study(sampler=sampler)\n\nx0 = bench.get_x0().numpy(force=True)\n\ndef objective(trial: optuna.Trial):\n    values = [trial.suggest_float(f\"p{i}\", -3, 3) for i in range(len(x0))]\n    return bench.loss_at(np.asarray(values))\n\nstudy.optimize(objective, n_trials=1000)\n\nbench.render(\"Optuna.mp4\", line_alpha=0.1)\n```\n\n\u003chttps://github.com/user-attachments/assets/021846d8-626d-4f2a-a8cb-7d2143d28673\u003e\n\n### Algebras\n\nSome benchmarks let you choose an algebra, i.e. tropical algebra so that you can optimize tropical SVD decomposition etc. In tropical algebra plus is replaced with min, and product with plus. Whenever a benchmark has `algebra` argument, you can choose a different algebra by passing one of those strings:\n\n```py\n'elementary', 'tropical', 'tropical min', 'tropical max', 'fuzzy', 'lukasiewicz', 'viterbi', 'viterbi max', 'viterbi min', 'log', 'probabilistic', 'modulo1', 'modulo5'\n```\n\n### Adding noise\n\nIt is possible to add two kinds of noise to any benchmark by calling `benchmark.set_noise` method. First kind of noise `p` evaluates function and gradient at randomly perturbed parameters. Second kind of noise `g` is just noise added to gradients.\n\n```py\nbench = vb.FunctionDescent(\"booth\").set_noise(p=2.0, g=2.0)\nopt = torch.optim.SGD(bench.parameters(), lr=1e-2)\n\nbench.run(opt, max_steps=1000)\nbench.plot()\n```\n\n\u003cimg width=\"450\" height=\"auto\" alt=\"image\" src=\"https://github.com/user-attachments/assets/9262f339-3fda-4ec5-beb6-0777ca5a3fdb\" /\u003e\n\n### Multi-objective / Least squares optimization\n\nSome benchmarks support returning multiple objectives or residuals for least squares. By default they return a single scalar value (usually sum or sum of squares, the function has to be explicitly defined in the benchmark). So to make a benchmark return multiple values, call `benchmark.set_multiobjective(True)`. Now whenever `bench.closure` is called, it returns a vector of values.\n\n```py\nimport torchzero as tz\n\n# rosenmo is version of rosenbrock which returns two residuals.\nbench = vb.FunctionDescent(\"rosenmo\")\n\n# don't forget to enable multi-objective mode\nbench.set_multiobjective(True)\n\n#  We can use a least squares solver such as Gauss-Newton\nopt = tz.Optimizer(\n    bench.parameters(),\n    tz.m.LevenbergMarquardt(tz.m.GaussNewton())\n)\n\nbench.run(opt, max_steps=100)\nbench.plot()\n```\n\n\u003cimg width=\"450\" height=\"auto\" alt=\"image\" src=\"https://github.com/user-attachments/assets/a3c9ae79-972a-42fc-8af4-6f850a7faf80\" /\u003e\n\n### Logger\n\nBenchmark has a logger object where all the metrics reside. For example you can get a dictionary which maps step to train loss like this: `train_loss = bench.logger[\"train loss\"]`.\n\nLogger is a dictionary of dictionaries, but it also has some helpful methods. You can get all values for a metric as a numpy array by using `logger.to_numpy(metric)`. First value or last value can be accessed via `logger.first(metric)` and `logger.last(metric)`. Likewise, you can access minimum or maximum via `logger.min(metric)` and `logger.max(metric)`.\n\n### More tips\n\n- don't forget to move benchmarks to CUDA! Most are much faster on CUDA than on CPU.\n\n- whenever a benchmark accepts an image or a matrix, you can pass numpy array, torch tensor, or path to an image.\n\n- if you don't need visualization, use `benchmark.set_performance_mode(True)` to disable it which makes some benchmarks much faster by not running visualization code.\n\n- to disable the stupid printing use `benchmark.set_print_inverval(None)`.\n\n- benchmarks have a `benchmark.reset()` method, which resets the benchmark to initial state. It can be much faster than re-creating the benchmark from scratch in some cases, so it is good for hyperparameter tuning.\n\n- if you use optuna pruner, use `benchmark.set_optuna_trial(trial, metric=\"train loss\")` and it will report that metric to optuna and raise `optuna.TrialPruned()` when requested. See the hyperparameter optimization with Optuna example\n\n# Defining new benchmarks\n\n`Benchmark` is a subclass of `torch.nn.Module`.\n\nTo make a benchmark, subclass `Benchmark` and define a `get_loss` method which returns a `torch.Tensor` loss.\n\nYou can log any metrics by `self.log(value)`, and log any images by `self.log_image(image)`. The images will automatically be added to the plots and videos.\n\nHere is an example of objective where the task is to invert a matrix, which also visualizes current solution and some losses:\n\n```py\nclass MatrixInverse(vb.Benchmark):\n    def __init__(self, matrix: torch.Tensor):\n        super().__init__()\n\n        # store matrix and eye as buffers, that way benchmark.cuda() will move them to cuda\n        self.matrix = torch.nn.Buffer(matrix.float().cpu())\n        self.eye = torch.nn.Buffer(torch.eye(matrix.size(-1), dtype=torch.float32))\n\n        # this will be optimized to approximate inverse of the matrix\n        self.inverse_hat = torch.nn.Parameter(torch.randn_like(self.matrix))\n\n        # this shows the input matrix when plotting and rendering animations\n        self.add_reference_image(name=\"matrix\", image=self.matrix, to_uint8=True)\n\n    def get_loss(self):\n\n        # compute loss, B is inverse of A if AB = BA = I\n        AB = self.matrix @ self.inverse_hat\n        BA = self.inverse_hat @ self.matrix\n\n        loss1 = torch.nn.functional.mse_loss(AB, BA)\n        loss2 = torch.nn.functional.mse_loss(AB, self.eye)\n        loss3 = torch.nn.functional.mse_loss(BA, self.eye)\n\n        # log individual losses\n        # note that if metric name doesn't start with \"train \" or \"test \",\n        # that will be inserted in front of the name (this is by design)\n        self.log(\"AB BA loss\", loss1)\n        self.log(\"AB identity loss\", loss2)\n        self.log(\"BA identity loss\", loss3)\n\n        # log images\n        # make sure to skip (possibly expensive) visualization code if performance mode is enabled\n        # which sets `self._make_images=False`\n        if self._make_images:\n\n            # image can be numpy array or torch tensor in (C, H, W), (H, W, C) or (H, W).\n            # to_uint8 normalizes them to (0, 255) and converts to uint8 data type\n            # or you can do that manually if you want but then logged images must be uint8\n            self.log_image(name='inverse', image=self.inverse_hat, to_uint8=True)\n            self.log_image(name='AB', image=AB, to_uint8=True)\n            self.log_image(name='BA', image=BA, to_uint8=True)\n            self.log_image(name=\"reconstruction\", image=torch.linalg.inv(self.inverse_hat), to_uint8=True)\n\n        return loss1 + loss2 + loss3\n\n\n# Done! We can now run the benchmark\nmatrix = torch.randint(0, 3, (64, 64)).sort(dim=0)[0]\nbenchmark = MatrixInverse(matrix).cuda()\noptimizer = torch.optim.LBFGS(benchmark.parameters(), line_search_fn='strong_wolfe')\nbenchmark.run(optimizer, max_passes=1000)\n\nbenchmark.plot(yscale=\"log\") # plots everything that was logged\nbenchmark.render(\"L-BFGS inverting a matrix.mp4\") # renders a video with images that were logged\n```\n\n\u003chttps://github.com/user-attachments/assets/0c768529-2e71-4667-8908-86bbce515852\u003e\n\n# License\n\nMIT\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finikishev%2Fvisualbench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finikishev%2Fvisualbench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finikishev%2Fvisualbench/lists"}