{"id":21861609,"url":"https://github.com/elcorto/nmbx","last_synced_at":"2025-07-01T11:32:37.747Z","repository":{"id":176161536,"uuid":"607069868","full_name":"elcorto/nmbx","owner":"elcorto","description":"A box of tools that deal with numbers.","archived":false,"fork":false,"pushed_at":"2024-11-15T08:29:04.000Z","size":1547,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-26T14:53:25.553Z","etag":null,"topics":["convergence","deep-learning","early-stopping","machine-learning","neural-network"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elcorto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-27T08:47:56.000Z","updated_at":"2024-11-15T08:29:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"c1f912d5-3679-44b8-bed4-fc42cb9713e1","html_url":"https://github.com/elcorto/nmbx","commit_stats":null,"previous_names":["elcorto/nmbx"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elcorto%2Fnmbx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elcorto%2Fnmbx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elcorto%2Fnmbx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elcorto%2Fnmbx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elcorto","download_url":"https://codeload.github.com/elcorto/nmbx/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244860600,"owners_count":20522466,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["convergence","deep-learning","early-stopping","machine-learning","neural-network"],"created_at":"2024-11-28T03:12:11.158Z","updated_at":"2025-03-21T20:19:17.529Z","avatar_url":"https://github.com/elcorto.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nmbx\n\nnmbx, NuMBoX, n_umb_ox, $n_umb_ox$\n\nA box of tools that deal with numbers.\n\n# Content\n\n## Convergence tests\n\nIn `nmbx.convergence` we have tools for testing convergence in a sequence of\nnumbers, such as the loss when training machine learning models or any other\niterative process with a converging metric.\n\n`SlopeZero` detects a flat plateau (\"zero slope\"), which is a general-purpose\nmethod. `SlopeRise` detects a rise in the history after a flat plateau.\n\nThe idea is to call either in a training loop, passing a history of loss\nvalues.\n\n### Example\n\n```py\nfrom nmbx.convergence import SlopeZero\n\n# Detect convergence with wait=10 iterations of \"patience\", an absolute tolerance\n# of 0.01 and a moving average window of wlen=25 points. Start checking not before\n# 100 iterations have been performed.\nconv = SlopeZero(wlen=25, atol=0.01, wait=10, delay=100)\n\nhistory = []\nwhile True:\n    history.append(compute_loss(model, data))\n    if conv.check(history):\n        print(\"converged\")\n        break\n```\n\n### Methods\n\n`SlopeZero` implements the same logic as found in [Keras'][keras_es] or\n[Lightning's][liight_es] `EarlyStopping(mode=...)` with `mode=\"min\"` or\n`\"max\"`. In addition we provide `mode=\"abs\"` (detect convergence not assuming a\ndirection).\n\nSince we only work with a given list of numbers $y_i$ in the history, we have\n$\\Delta x=1$ in the slope $\\Delta y/\\Delta x$. Therefore, the `atol` and `rtol`\nparameters are to be understood w.r.t. $y$. Please check the doc strings for\nwhat `tol` does in each method, where `tol = atol` or `tol = rtol * abs(prev)`. In\nshort\n\n* `SlopeRise`: `last - tol \u003e prev`\n* `SlopeZero`:\n  * `mode=\"abs\": |last - prev| \u003c tol`\n  * `mode=\"min\": last + tol \u003e prev`\n  * `mode=\"max\": last - tol \u003c prev`\n\n`last` and `prev` are the mean/median/... (see `wlen_avg`) over the last and\nprevious non-overlapping windows of `wlen` points each. This means that the\nearliest convergence point can be detected after `2 * wlen` iterations. With\n`delay`, the first possible convergence point is after `2 * wlen + delay`\niterations.\n\n\n### Settings\n\nWe implement several options that can make convergence checks more robust and\nversatile than vanilla \"early stopping\".\n\n* Noise filtering (smoothing): Histories are often noisy (e.g. when using\n  stochastic optimizers). In vanilla early stopping, the only counter measure is\n  using \"patience\". We have the option to smooth the history using\n  * a Gaussian filter (set `smooth_sigma`) and/or\n  * a moving reduction of window size `wlen` (reduction = mean/median/..., see\n    `wlen_avg`). `wlen=1` means a window of one, so no noise filtering of this\n    kind. You can still use the Gaussian filter by setting `smooth_sigma`.\n\n* You may use some `delay` to make sure to run at least this many iterations\n  before checking for convergence. This can help to avoid early false positive\n  convergence detection.\n\nCan we get \"transferable\" tolerances? Well, kind of.\n\n* Absolute (`atol`) or relative (`rtol`) tolerances: If you know the unit of\n  the history and can say something like \"we call changes below 0.01\n  converged\", then use `atol`. Else, try to use a relative tolerance `rtol`, in\n  which case we use `tol = rtol * abs(prev)`.\n  * Pro: This will be invariant to scaling $y' = y s$.\n  * Con: Will not be invariant to a shift $y' = y + c$.\n\n* Standardization: You can standardize `history`, for example using a z-score\n  (set `std=\"std\"` and `std_avg=np.mean`) to zero mean and unity standard\n  deviation such that, at each iteration $i$, `atol` will be\n  in units of $\\sigma_i$. Now the convergence criterion is \"stop if changes are\n  below `atol` standard deviations\".\n  * Pro: This is helpful for histories of very different numerical scale\n    but similar \"shape\" and where you don't know or care about the unit of $y$.\n    More precisely, you can apply the same `atol` to all histories which\n    differ from $y$ by an affine transform $y' = y s + c$.\n  * Con: Since `check()` is an online method, the standardization is performed\n    for each iteration, using all history values provided so far. Therefore\n    $\\sigma_i$, and thus the unit of `atol`, will change which makes the effect\n    of standardization more difficult to interpret. There are corner cases\n    where this method doesn't work (for example a noise-free constant history\n    where $\\sigma$ is zero). Also some experimentation is needed to find good\n    `atol` value. Check `examples/convergence/visualize_std.py` and all\n    ``test_atol_std*`` tests.\n\nHere are results from a parameter study in\n`examples/convergence/param_study.py` with noise-free and noisy histories,\nwhere we explore the above parameters. Blue points are the histories. The other\npoints indicate when `check()` is True. The points marked with vertical dashed\nlines are the *first* points where the check is True, i.e. where you would\nbreak out of the training loop. If no colored points show up, then this means\nthat the corresponding parameter setting leads to no convergence detection.\n\n![](doc/pics/conv_no_noise.png)\n![](doc/pics/conv_noise.png)\n\nWe observe that `SlopeZero` is pretty robust against noise, while `SlopeRise`\nis more tricky, i.e. it is not clear what the right parameters are in this\ncase.\n\n### Recommendations for parameter settings\n\n* Use `wlen \u003e 1` (smoothing by moving reduction) and `wait \u003e 1` (\"patience\").\n  Typical values are `wlen=10...20` (but that depends very much on your data),\n  and `wait=5..10`. Using the Gaussian filter  (`smooth_sigma`) is a more\n  effective smoothing option, but still some `wlen` can help.\n* When data is noisy, it *can* help to raise `tol` to prevent too early / false\n  convergence detection. But having a good setting for `wlen` or `smooth_sigma` is\n  preferable.\n* To find a good `smooth_sigma` value, start with `smooth_sigma = wlen/3` where\n  `wlen` is the value you would use for smoothing instead (e.g. `wlen=15 -\u003e\n  smooth_sigma=3`). See `examples/convergence/find_sigma_from_wlen.py`.\n* Use `delay` if you know that you need to run at least this many iterations.\n  This also helps too avoid early false positives.\n* When using standardization (e.g. `std=\"std\"`), start with `atol=0.01`, so\n  \"stop when things fluctuate by less than 0.01 standard deviations\".\n* Take the ball park numbers used in `examples/convergence/` as starting point.\n\n### Finding good parameters\n\nOne can frame this as an optimization problem. We have an example using Optuna\nin `examples/convergence/param_opt.py`. You can use this if you have a\nrepresentative history recorded for your application and plan to use\nconvergence detection for many similar runs.\n\n### Other packages\n\n* https://github.com/JuliaAI/EarlyStopping.jl\n\n[keras_es]: https://keras.io/api/callbacks/early_stopping\n[liight_es]: https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.EarlyStopping.html#lightning.pytorch.callbacks.EarlyStopping\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felcorto%2Fnmbx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felcorto%2Fnmbx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felcorto%2Fnmbx/lists"}