{"id":17113031,"url":"https://github.com/emmt/numoptbase.jl","last_synced_at":"2026-02-10T02:31:43.073Z","repository":{"id":169364648,"uuid":"645298738","full_name":"emmt/NumOptBase.jl","owner":"emmt","description":" Basic operations on variables for multi-variate numerical optimization methods","archived":false,"fork":false,"pushed_at":"2025-04-08T18:45:14.000Z","size":234,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-17T21:27:51.393Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/emmt.png","metadata":{"files":{"readme":"README.md","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-25T10:52:26.000Z","updated_at":"2025-04-08T18:45:19.000Z","dependencies_parsed_at":"2024-05-02T17:47:27.141Z","dependency_job_id":"5d48f5ef-faf8-4215-b1c9-8b8ab31c9d3e","html_url":"https://github.com/emmt/NumOptBase.jl","commit_stats":null,"previous_names":["emmt/numoptbase.jl"],"tags_count":21,"template":false,"template_full_name":null,"purl":"pkg:github/emmt/NumOptBase.jl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emmt%2FNumOptBase.jl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emmt%2FNumOptBase.jl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emmt%2FNumOptBase.jl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emmt%2FNumOptBase.jl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/emmt","download_url":"https://codeload.github.com/emmt/NumOptBase.jl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emmt%2FNumOptBase.jl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29289468,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-09T21:57:15.303Z","status":"online","status_checked_at":"2026-02-10T02:00:07.935Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T17:02:25.317Z","updated_at":"2026-02-10T02:31:43.057Z","avatar_url":"https://github.com/emmt.png","language":"Julia","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Basic operations on variables for numerical optimization in Julia\n\n[![Build Status](https://github.com/emmt/NumOptBase.jl/actions/workflows/CI.yml/badge.svg?branch=main)](https://github.com/emmt/NumOptBase.jl/actions/workflows/CI.yml?query=branch%3Amain)\n[![Build Status](https://ci.appveyor.com/api/projects/status/github/emmt/NumOptBase.jl?svg=true)](https://ci.appveyor.com/project/emmt/NumOptBase-jl)\n[![Coverage](https://codecov.io/gh/emmt/NumOptBase.jl/branch/main/graph/badge.svg)](https://codecov.io/gh/emmt/NumOptBase.jl)\n\n`NumOptBase` implements efficient basic operations on variables for\nmulti-variate numerical optimization methods in [Julia](https://julialang.org).\nIt is similar to the `BLAS` library for linear algebra methods.\n\nBy leveraging the methods provided by `NumOptBase`, numerical optimization\nmethods can be written in a general way that is agnostic to the specific type\nof arrays used to store the variables of the problem. Package\n[`ConjugateGradient`](https://github.com/emmt/ConjugateGradient.jl) is such an\nexample. The methods of `NumOptBase` are thus intended to be extended by other\npackages to apply numerical optimization methods to their own variables (that\nis their own array types). For instance, `NumOptBase` provides optimized\nmethods for [`CUDA` arrays](https://github.com/JuliaGPU/CUDA.jl) .\n\n\n## Variables in optimization methods\n\nAn optimization problem typically writes:\n\n    minₓ f(x) s.t. x ∈ Ω\n\nwhere `f: Ω → ℝ` is the objective function, `x` are the variables, and `Ω ⊆ ℝⁿ`\nis the set of acceptable solutions with `n` the dimension of the problem.\n\nIt is assumed by this package that the variables `x` are stored in Julia\narrays. Depending on the problem, these arrays may be multidimensional but are\ntreated as real-valued *vectors* by the numerical optimization methods. In that\nrespect, complex numbers are considered as pairs of reals. For efficiency, all\nentries of the arrays storing variables shall have the same floating-point type.\n\nFor now, quantities with units (such as those provided by the\n[`Unitful`](https://github.com/PainterQubits/Unitful.jl) package) are not\nsupported. If your variables have units, you may consider using `reinterpret`\nto remove units before calling the numerical optimization routines.\n\n\n## Operations on variables\n\nSome Julia methods are already available to deal with the variables of a\nnumerical optimization problem. For example, `similar` which may be called to\ncreate a new array of variables provided the element type is some\nfloating-point real.\n\nThe `NumOptBase` package provides additional methods (described in what\nfollows) to operate on *variables* and which either require no additional\nsignificant storage or store their result in an output array provided by the\ncaller. In that way, the storage requirements can be strictly controlled. All\nthese *public* methods are exported except `NumOptBase.copy!` which exists in\nJulia base but with a slightly different semantic regarding vectors.\n\nAs said before, these methods treat the variables as vectors of reals, except\nthat methods taking multiple array arguments throw a `DimensionMismatch`\nexception if these arguments do not have the same axes.\n\n\n### Norms\n\n`norm1(x)`, `norm2(x)`, and `norminf(x)` respectively yield the ℓ₁, Euclidean,\nand infinite norm of the variables `x`. They are similar to the norms in\n`LinearAlgebra` except that they treat `x` as if it is has been flattened into\na vector of reals.\n\n\n### Inner product\n\n`inner(x, y)` yields the inner product (also known as *scalar product*) of the\nvariables `x` and `y` computed as expected by numerical optimization methods;\nthat is as if `x` and `y` are real-valued vectors and treating complex values\nas pairs of reals in that respect. In other words, if `x` and `y` are\nreal-valued arrays, their inner product is given by:\n\n    Σᵢ xᵢ⋅yᵢ\n\notherwise, if `x` and `y` are both complex-valued arrays, their inner product\nis given by:\n\n    Σᵢ (real(xᵢ)⋅real(yᵢ) + imag(xᵢ)⋅imag(yᵢ))\n\nIn the above pseudo-codes, index `i` runs over all indices of `x` and `y` which\nmay be multi-dimensional arrays but must have the same indices.\n\n`inner(w, x, y)` yields:\n\n    Σᵢ wᵢ⋅xᵢ⋅yᵢ\n\nthat is the *triple inner product* of the variables `w`, `x`, and `y` which\nmust all be real-valued arrays.\n\n\n### Scaling, updating, and combining variables\n\n`scale!(dst, α, x)` returns `dst` overwritten with `α⋅x` performed\nelement-wise. Arguments `dst` and `x` are arrays of the same size while `α` is\na scalar. If `iszero(α)` holds, `dst` is zero-filled whatever the values in\n`x`. For in-place scaling of `x` by `α`, just call `scale!(α, x)` or `scale!(x, α)`.\n\n`update!(x, β, y)` returns `x` overwritten with `x + β⋅y` performed\nelement-wise. Arguments `x` and `y` are arrays of the same size while `β` is a\nscalar. If `iszero(β)` holds, `x` is left unchanged whatever the values in `y`.\nThe `update!` method may be seen as a shortcut for `combine!(x, 1, x, β, y)`.\n\n`update!(x, β, y, z)` returns `x` overwritten with `x + β⋅y⋅z` performed\nelement-wise. Arguments `x`, `y`, and `z` are arrays of the same size while `β`\nis a scalar. If `iszero(β)` holds, `x` is left unchanged whatever the values in\n`y` and `z`.\n\n`multiply!(dst, x, y)` returns `dst` overwritten with the element-wise\nmultiplication (also known as *Hadamard product*) of `x` by `y`. Arguments\n`dst`, `x`, and `y` are arrays of the same size.\n\n`combine!(dst, α, x, β, y)` overwrites `dst` with `α⋅x + β⋅y` and returns\n`dst`. Arguments `α` and `β` are real scalars while `dst`, `x`, and `y` are\narrays of the same size. If `iszero(α)` holds, the result does not depend on\nthe values of `x`. Similarly, if `iszero(β)` holds, the result does not depend\non the values of `y`.\n\n\n### Apply mappings\n\nThe method:\n\n``` julia\napply!(dst, f, args...) -\u003e dst\n```\n\noverwrites the destination variables `dst` with the result of applying the\nmapping `f` to arguments `args...`.\n\nAs of now, `apply!` only handles a few types of mappings:\n\n- If `f` is an array, a generalized matrix-vector multiplication is applied to\n  `args...` which must be a single array of variables.\n\n- If `f` is `NumOptBase.Identity()`, the identity mapping is applied, that is\n  the values of `src` are copied into `dst` (unless they are the same object).\n  The constant `NumOptBase.Id` is the singleton object of type\n  `NumOptBase.Identity`.\n\n- If `f` is an instance of `NumOptBase.Diag`, an element-wise multiplication by\n  `diag(f)` is applied.\n\nThe `NumOptBase.apply!` method shall be specialized in other argument types to\nhandle other cases.\n\n\n### Other operations\n\n`NumOptBase.copy!(dst, src)` overwrites the destination array `dst` with the\ncontents of the source array `src` throwing an error if they do not have the\nsame axes. If checking that the arguments have the same axes is not necessary,\nthe end-user may use `copyto!(dst, src)` or `copy!(dst, src)` which are basic\nJulia methods.\n\nIt is assumed that a few standard Julia methods are implemented in an efficient\nway for the type of array storing the variables:\n\n- `similar(x) -\u003e y` to create a new array of variables `y` like `x`;\n- `copyto!(dst, src) -\u003e dst` to copy source variables `src` into destination\n  variables `dst`;\n- `fill!(x, α) -\u003e x` to set all variables in `x` to the value `α`.\n\n\n## Bound constraints\n\n`NumOptBase` provides some support for separable bound constraints on the\nvariables. With such constraints, the feasible set is defined by:\n\n```\nΩ = { x ∈ ℝⁿ | ℓ ≤ x ≤ u }\n```\n\nwith `ℓ` and `u` the lower and upper bounds and where the comparisons (`≤`) are\ntaken element-wise. To represent the feasible set for bound constrained\n`N`-dimensional variables of element type `T` in Julia is done by:\n\n``` julia\nΩ = BoundedSet{T,N}(ℓ, u)\n```\n\nwhere the lower and upper bounds, `ℓ` and `u`, may be specified as:\n- `nothing` if the bound is unlimited;\n- a scalar if the bound is the same for all variables;\n- an array with the same axes as the variables.\n\nFor simplicity and type-stability, there are a number of restrictions which may\nbe alleviated in a high level interface:\n- To avoid the complexity of managing all possibilities in the methods\n  implementing bound constraints, bounds specified as arrays *conformable* with\n  the variables are not directly supported. The caller may extend the array of\n  bound values to the same size as the variables.\n- Only `nothing` and the scalars `-∞` (for a lower bound) or `+∞` (for an upper\n  bound) are considered as unlimited bounds even though all values of a lower\n  (resp. upper) bound specified as an array may be `-∞` (resp. `+∞`).\n\nA number of standard methods are applicable to a bounded set `Ω`:\n- `isempty(Ω)` yields whether the `Ω` is empty, that is infeasible.\n- `x ∈ Ω` yields whether variables `x` belongs to `Ω`.\n\n### Projection on the feasible set\n\nFor any `x ∈ ℝⁿ`, the *projected variables* `xₚ ∈ Ω` are defined by:\n\n```\nxₚ = P(x) = argmin ‖y - x‖²   s.t.   y ∈ Ω\n```\n\nwhere `P` is the projection onto the feasible set `Ω`. In other words, `xₚ` is\nthe element of `Ω` that is the closest (in the least Euclidean distance sense)\nto `x`.\n\nThe projected variables are computed by:\n\n``` julia\nproject_variables!(xₚ, x, Ω)\n```\n\nwhich overwrites the destination `xₚ` with the projection of `x ∈ ℝⁿ` onto the\nfeasible set `Ω ⊆ ℝⁿ`.\n\n\n### Projected direction and line-search\n\nA number of numerical optimization methods proceed by iterations where, at the\n`k`-th iteration, the next iterate writes:\n\n```\nxₖ₊₁ = P(xₖ ± αₖ⋅dₖ)   with   αₖ ≈ argmin f(P(xₖ ± α⋅dₖ))   s.t.   α ≥ 0\n```\n\nwith `d ∈ ℝⁿ` a well chosen search direction and where, depending on the\nnumerical implementation, `±` is either `+` or `-` depending on whether the\nvariables vary as:\n\n``` julia\nx = P(xₖ + α⋅dₖ)\n```\n\nor:\n\n``` julia\nx = P(xₖ - α⋅dₖ)\n```\n\nalong the path `α ≥ 0`. The `NumOptBase` package provides some methods to help\nimplementing such line-search methods.\n\nFor any feasible `x ∈ Ω` and search direction `d ∈ ℝⁿ`, the *projected\ndirection* `dₚ ∈ ℝⁿ` is defined by:\n\n```\n∀ α ∈ [0,ε], P(x ± α⋅d) = x ± α⋅dₚ\n```\n\nfor some `ε \u003e 0` and where `P` is the projection onto the feasible set `Ω`\npreviously defined. In other words, `dₚ` is the effective search direction in\n`Ω` for any sufficiently small step size `α`.\n\nThe projected direction is computed by:\n\n``` julia\nproject_direction!(dₚ, x, ±, d, Ω)\n```\n\nwhich overwrites the destination `dₚ` and where `±` is either `+` or `-`.\n\nA closely related function is:\n\n``` julia\nchanging_variables!(a, x, ±, d, Ω)\n```\n\nwhich overwrites the destination `a` with ones where variables in `x ∈ Ω` will\nvary along the direction `±d` while respecting the constraints implemented by\n`Ω` and zeros elsewhere. Hence, if `±d = -∇f(x)`, with `∇f(x)` the gradient of\nthe objective function, the destination is set to zero everywhere the\nKarush-Kuhn-Tucker (K.K.T.) conditions are satisfied for the problem:\n\n```\nminₓ f(x) s.t. x ∈ Ω\n```\n\nIn other words, `all(izero, a)` holds for (exact) convergence. Note that the\nprojected direction `dₚ` and `a` are related by `dₚ = a .* d`.\n\nWhen line-searching, two specific values of the step length `α ≥ 0` are of\ninterest:\n\n- `αₘᵢₙ ≥ 0` is the greatest nonnegative step length such that:\n\n  ```\n  α ≤ αₘᵢₙ  ⟹  P(x ± α⋅d) = x ± α⋅d\n  ```\n\n- `αₘₐₓ ≥ 0` is the least nonnegative step length such that:\n\n  ```\n  α ≥ αₘₐₓ  ⟹  P(x ± α⋅d) = P(x ± αₘₐₓ⋅d)\n  ```\n\nIn other words, no bounds are overcome if `0 ≤ α ≤ αₘᵢₙ` and the projected\nvariables are all the same for any `α` such that `α ≥ αₘₐₓ`. The values of\n`αₘᵢₙ` and/or `αₘₐₓ` can be computed by one of:\n\n``` julia\nαₘᵢₙ = linesearch_stepmin(x, ±, d, Ω)\nαₘₐₓ = linesearch_stepmax(x, ±, d, Ω)\nαₘᵢₙ, αₘₐₓ = linesearch_limits(x, ±, d, Ω)\n```\n\nNote that, for efficiency, `project_direction!`, `changing_variables!`,\n`linesearch_stepmin`, `linesearch_stepmax`, and `linesearch_limits` assume\nwithout checking that the input variables `x` are feasible, that is that `x ∈\nΩ` holds.\n\n\n\n## Extension to other array types\n\nTo extend the `NumOptBase` to other array types, some understanding of the\nimplementation of this package is needed. The **public methods** which can be\ncalled by the end-users are summarized in the following table.\n\n| Public method             | Description             | Remarks                            |\n|:--------------------------|:------------------------|:-----------------------------------|\n| `similar(x)`              | Yield an array like `x` |                                    |\n| `zerofill!(dst)`          | Zero-fill `dst`         |                                    |\n| `NumOptBase.copy!(dst,x)` | Copy `x` into `dst`     | See `copyto!` and `copy!` in Julia |\n| `scale!(dst,α,x)`         | `dst = α*x`             |                                    |\n| `update!(dst,α,x)`        | `dst += α*x`            |                                    |\n| `update!(dst,α,x,y)`      | `dst += α*x.*y`         |                                    |\n| `combine!(dst,α,x,β,y)`   | `dst = α*x + β*y`       |                                    |\n| `inner(x,y)`              | Inner product           |                                    |\n| `inner(w,x,y)`            | Triple inner product    |                                    |\n| `norm1(x)`                | ℓ₁ norm                 |                                    |\n| `norm2(x)`                | Euclidean norm          |                                    |\n| `norminf(x)`              | Infinite norm           |                                    |\n\nIn the above table and hereinafter, `dst`, `w`, `x`, and `y` denote arrays\n(considered as *vectors*), `α` and `β` denote scalar reals, and all operations\nand function calls are assumed to be done element-wise.\n\nThese public methods check their arguments (for having the same axes and thus\nthe same indices) and call one of the specialized methods listed below\ndepending on the operation, on the type of the array arguments, and on the\nspecific values of the multipliers `α` and `β`.\n\n| Operation         | Specialized method                    | Remarks                  |\n|:------------------|:--------------------------------------|:-------------------------|\n| `dst = 0`         | `zerofill!(dst)`                      |                          |\n| `dst = x`         | `unsafe_copy!(dst,x)`                 |                          |\n| `dst = f(x)`      | `unsafe_map!(f,dst,x)`                |                          |\n| `dst = f(x,y)`    | `unsafe_map!(f,dst,x,y)`              |                          |\n| `dst = α*x`       | `unsafe_map!(αx(α,x),dst,x)`          | `α` is not 0, nor 1      |\n| `dst = α*x + y`   | `unsafe_map!(αxpy(α,x),dst,x,y)`      | `α` is not 0, nor 1      |\n| `dst = x + β*y`   | `unsafe_map!(αxpy(β,y),dst,y,x)`      | `β` is not 0, nor 1      |\n| `dst = α*x + β*y` | `unsafe_map!(αxpβy(α,x,β,y),dst,x,y)` | neither `α` nor `β` is 0 |\n| `dst = -x`        | `unsafe_map!(-,dst,x)`                |                          |\n| `dst = x + y`     | `unsafe_map!(+,dst,x,y)`              |                          |\n| `dst = x - y`     | `unsafe_map!(-,dst,x,y)`              |                          |\n| `dst = x * y`     | `unsafe_map!(*,dst,x,y)`              |                          |\n| `inner(x,y)`      | `unsafe_inner(x,y)`                   |                          |\n| `inner(w,x,y)`    | `unsafe_inner(w,x,y)`                 |                          |\n| `norm1(x)`        | `norm1(x))`                           |                          |\n| `norm2(x)`        | `norm2(x)`                            |                          |\n| `norminf(x)`      | `norminf(x)`                          |                          |\n\nThe prefix `unsafe_` means that the axes of arguments have been checked to be\ncompatible. Any scalar argument (`α` and `β`) shall never be zero and shall\nhave been converted to the correct floating-point type (this conversion is\nautomatically done by the constructors `αx`, `αxpy`, `αxmy`, and `αxpβy`). The\ncode of the high-level methods shall be simple enough for these methods to be\nin-lined. This may lead to some optimizations (when the multipliers have\nspecific values like 0 or ±1).\n\nRemarks:\n\n- `unsafe_copy!(dst, x)` shall not be called when `dst` and `x` are the same\n  object and amounts to calling `copyto!(dst, x)` by default but may be\n  extended.\n\n- `zerofill!(dst)` amounts to calling `fill!(dst, zero(eltype(dst)))` by\n  default but `memset(pointer(dst), 0, sizeof(dst))` may be used for dense\n  arrays.\n\n- When `α` (resp. `β`) is zero, it is assumed that expression `α*x` (resp.\n  `β*y`) is everywhere zero whatever the values of `x` (resp. `y`).\n\n- It can be seen that a great deal of cases are handled by `unsafe_map!`. To\n  avoid some overheads with closures and to allow for specialization of the\n  code, `αx`, `αxpy`, `αxmy`, and `αxpβy` build callable objects which have\n  specific types and which implement simple operation involving multipliers:\n\n  ```julia\n  f1 = αx(α,x)\n  f1(xᵢ) -\u003e α*xᵢ\n  f2 = αxpy(α,x)\n  f2(xᵢ,yᵢ) -\u003e α*xᵢ + yᵢ\n  f3 = αxmy(α,x)\n  f3(xᵢ,yᵢ) -\u003e α*xᵢ - yᵢ\n  f4 = αxpβyy(α,x,β,y)\n  f4(xᵢ,yᵢ) -\u003e α*xᵢ + β*yᵢ\n  ```\n\n  where `xᵢ` and `yᵢ` denote an entry of `x` and `y`. These constructors take\n  care of converting the multipliers `α` and `β`to the correct floating-point\n  type. This the reason to provide arrays `x` and `y` along with their\n  respective multiplier to the constructors.\n\nTo support specific array types or to optimize the operations for given array\ntypes, it is sufficient to extend the specialized methods (the ones prefixed by\n`unsafe_`) and the methods that compute the norms. Specializing the method\n`zerofill!` is not mandatory as the default version shall work for all array\ntypes.\n\nYou may have a look in the files\n[ext/NumOptBaseLoopVectorizationExt.jl](ext/NumOptBaseLoopVectorizationExt.jl)\nand [ext/NumOptBaseCUDAExt.jl](ext/NumOptBaseCUDAExt.jl) which respectively\nextend `NumOptBase` to use AVX loop vectorization and Cuda GPU arrays.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femmt%2Fnumoptbase.jl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Femmt%2Fnumoptbase.jl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femmt%2Fnumoptbase.jl/lists"}