{"id":16705052,"url":"https://github.com/mcabbott/oddarrays.jl","last_synced_at":"2026-01-28T21:34:34.987Z","repository":{"id":117871417,"uuid":"420416693","full_name":"mcabbott/OddArrays.jl","owner":"mcabbott","description":"☯︎","archived":false,"fork":false,"pushed_at":"2021-11-03T16:48:12.000Z","size":30,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-06T14:40:43.535Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mcabbott.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-23T13:13:10.000Z","updated_at":"2022-01-30T13:05:21.000Z","dependencies_parsed_at":"2023-10-16T15:15:55.591Z","dependency_job_id":null,"html_url":"https://github.com/mcabbott/OddArrays.jl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mcabbott/OddArrays.jl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcabbott%2FOddArrays.jl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcabbott%2FOddArrays.jl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcabbott%2FOddArrays.jl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcabbott%2FOddArrays.jl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mcabbott","download_url":"https://codeload.github.com/mcabbott/OddArrays.jl/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcabbott%2FOddArrays.jl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28852960,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-28T15:15:36.453Z","status":"ssl_error","status_checked_at":"2026-01-28T15:15:13.020Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T19:28:32.076Z","updated_at":"2026-01-28T21:34:34.971Z","avatar_url":"https://github.com/mcabbott.png","language":"Julia","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OddArrays.jl\n\nThis defines a few array types whose storage is quite different from their values:\n\n```julia\njulia\u003e using OddArrays\n\njulia\u003e Rotation(pi/6) * Rotation(pi/6)  # stores one angle\n2×2 Rotation{Float64}, with theta = 1.0471975511965976:\n 0.5       -0.866025\n 0.866025   0.5\n\njulia\u003e Vandermonde([0,2,10])  # coefficient vector, scale 1\n3×3 Vandermonde{Int64, Vector{Int64}}:\n 1   0    0\n 1   2    4\n 1  10  100\n\njulia\u003e Full(2pi, 1, 3)  # one number\n1×3 Full{Float64, 2}:\n 6.28319  6.28319  6.28319\n\njulia\u003e Range(1,2,3)  # start, stop, length::Int\n3-element Range{Float64}, with step = 0.5:\n 1.0\n 1.5\n 2.0\n\njulia\u003e Outer([5,7], [1,10,100])  # here two vectors, allows two matrices etc.\n2×3 Outer{Int64, Vector{Int64}, Vector{Int64}}, storing 5 numbers:\n 5  50  500\n 7  70  700\n\njulia\u003e Mask([1 missing 3], [40,50,60]')  # two arrays\n1×3 Mask{Int64, 2, Matrix{Union{Missing, Int64}}, Adjoint{Int64, Vector{Int64}}}:\n 1  50  3\n\njulia\u003e PDiagMat([1,20,300])  # stores the inverse too\n3×3 PDiagMat{Float64, Vector{Float64}}:\n 1.0   0.0    0.0\n 0.0  20.0    0.0\n 0.0   0.0  300.0\n```\n\nThey exist for the purpose checking that we know what we're doing with automatic differentiation. In particular, with reverse mode AD, there are issues of how to make sure the gradient stays inside the right subpace, and how best to represent it.\n\nOne problem we'd like to avoid is:\n\n```julia\njulia\u003e using Zygote, ChainRulesCore\n\njulia\u003e gradient(_det, Vandermonde([3,4]))  # type's special definition, accesses fields\n((coeff = [-1.0, 1.0], scale = 2.0),)\n\njulia\u003e gradient(prod, Vandermonde([3,4]))  # generic rule, makes a matrix\n([12.0 4.0; 12.0 3.0],)\n\njulia\u003e gradient(x -\u003e _det(x) / prod(x), Vandermonde([3,4]))  # without projection\nERROR: MethodError: no method matching +(::Matrix{Float64}, ::NamedTuple{(:coeff, :scale), ...})\n```\n\nThere was a similar problem accumulating gradients for `Diagonal`:\n\n```julia\njulia\u003e pullback(sum, Diagonal([3,-4]))[2](1.0)[1]\n2×2 Fill{Float64}, with entries equal to 1.0\n\njulia\u003e pullback(x -\u003e 5 * x.diag[1], Diagonal([3,-4]))[2](1)[1]\n(diag = [5.0, 0.0],)\n```\n\n... which we can solve by standardising on the \"natural\" form, i.e., converting both contributions to `dx::Diagonal`:\n\n```julia\njulia\u003e gradient(x -\u003e sum(x) + 5 * x.diag[1], Diagonal([3,-4]))  # with Zygote#1104 + CRC#446 \n2×2 Diagonal{Float64, Vector{Float64}}:\n 6.0    ⋅ \n  ⋅    1.0\n```\n\nPerhaps these arrays which are nonlinear functions of their fields should instead standardise on the \"structural\" one?\n\n## Structural gradients\n\nThe operation we want is the pullback of `collect`, which is called `uncollect` here. Some arrays have their own; the (slow but not wrong) fallback version uses Zygote's `pullback` and the array's own `getindex`.\n\n```julia\njulia\u003e uncollect([0 1; 0 0], Vandermonde([3,4]))\n[ Info: generic _uncollect\n(coeff = [1.0, 0.0], scale = 3.0)\n```\n\nThis is now called by `ProjectTo` for these arrays, which in turn is called by many generic rules, including the one for `prod`:\n\n```julia\njulia\u003e gradient(x -\u003e _det(x) / prod(x), Vandermonde([3,4]))\n┌ Info: projecting to Tangent{Vandermonde}\n└   typeof(dx) = Matrix{Float64}\n[ Info: generic _uncollect\n((coeff = [-0.1111111111111111, 0.0625], scale = -0.16666666666666666),)\n```\n\nHere `ProjectTo{OddArray}` saves the whole original array. Because in general the gradient subspace depends on the point.\n\nWhile the possible perturbations of `θ` in `Rotation(θ)` are a 1-dimensional subspace of 2x2 matrices, the particular subspace depends on `θ`, etc. This is also why we cannot implement something like `+(::Matrix, ::Tangent{Rotation})` since, by that stage, the original `θ` has been lost.\n\n```julia\njulia\u003e uncollect([0 1; 0 0], Rotation(pi/3))\n(theta = -0.5000000000000001,)\n\njulia\u003e uncollect([0 1; 0 0], Rotation(pi/4))\n(theta = -0.7071067811865476,)\n```\n\nCan this go wrong? We like \"natural\" `dx::Diagonal` so that it can flow backwards into generic rules. For this to matter, the original `x::Diagonal` must have been the output of a function which has a generic rule. Here, there are methods for multiplication of `r::Rotation`, which means one can be produced by `*` which has a generic rule. Which then fails, unless we opt out:\n\n```julia\njulia\u003e gradient(x -\u003e _getindex(x*x, 1,2), Rotation(pi/7))  # not good!\n[ Info: *(::Rotation, ::Rotation)\nERROR: MethodError: no method matching *(::Tangent{Any, NamedTuple{(:theta,), Tuple{Float64}}}, ::Adjoint{Float64, Rotation{Float64}})\n\njulia\u003e gradient(x -\u003e (_collect(Rotation(x))*_collect(Rotation(x)))[1,2], pi/7)  # desired result\n(-1.2469796037174672,)\n\njulia\u003e ChainRulesCore.@opt_out ChainRulesCore.rrule(::typeof(*), ::Rotation, ::Rotation)\n\njulia\u003e gradient(x -\u003e _getindex(x*x, 1,2), Rotation(pi/7))\n[ Info: *(::Rotation, ::Rotation)\n((theta = -1.2469796037174672,),)\n```\n\nThere is also a method `mul!(::Vector, ::Rotation, ::Vector)` which doesn't cause problems, since it never returns a `Rotation` matrix. The generic rule does make a full matrix before `uncollect` is called, and this can't be avoided by opting out:\n\n```julia\njulia\u003e gradient(x -\u003e (x * [1,0])[1], Rotation(pi/7))\n[ Info: mul!(_, ::Rotation, _)\n┌ Info: projecting to Tangent{Rotation}\n└   typeof(dx) = Matrix{Float64} (alias for Array{Float64, 2})\n┌ Info: uncollect(_, ::Rotation)\n└   theta = -0.4338837391175581\n((theta = -0.4338837391175581,),)\n\njulia\u003e ChainRulesCore.@opt_out ChainRulesCore.rrule(::typeof(*), ::Rotation, ::Vector)\n\njulia\u003e gradient(x -\u003e (x * [1,0])[1], Rotation(pi/7))\n[ Info: mul!(_, ::Rotation, _)\nERROR: Mutating arrays is not supported -- called setindex!(::Vector{Float64}, _...)\n```\n\n## Natural gradients\n\nFor some of these types, we can plausibly standardise on a \"natural\" gradient instead. Here we need other functions, mapping onto a different representation of the tangent space. \n\nThe most trivial example is probably `Diagonal` (an honorary `OddArray`). The two new functions we need in general are, `restrict` \u0026 `naturalise`, are:\n\n```julia\njulia\u003e restrict([1 2; 3 4], Diagonal)\n2×2 Diagonal{Int64, Vector{Int64}}:\n 1  ⋅\n ⋅  4\n\njulia\u003e uncollect([1 2; 3 4], Diagonal)\n(diag = [1, 4],)\n\njulia\u003e naturalise(ans, Diagonal)\n2×2 Diagonal{Int64, Vector{Int64}}:\n 1  ⋅\n ⋅  4\n```\n\nThese obey the following properties, all of them a bit trivial:\n\n```julia\nx = Diagonal(rand(3))\ndx = rand(3,3); dx2 = randn(3,3)\n\n@testset \"simple naturalise checks for x::$(typeof(x).name.name)\" begin\n  # doesn't forget things which uncollect remembers:\n  @test uncollect(naturalise(uncollect(dx, x), x), x) ≈ uncollect(dx, x)\n\n  # linearity (using that of uncollect):\n  @test naturalise(uncollect(33 * dx, x), x) ≈ 33 * naturalise(uncollect(dx, x), x)\n  @test naturalise(uncollect(dx + dx2, x), x) ≈ naturalise(uncollect(dx, x), x) + naturalise(uncollect(dx2, x), x)\n\n  # this defines restrict in terms of naturalise:\n  @test restrict(dx, x) ≈ naturalise(uncollect(dx, x), x)\nend\n```\n\nThese are also satisfied by `x = Full(2,3,3)`, if the action of `restrict` \u0026 `naturalise` is this:\n\n```julia\njulia\u003e uncollect([10 0 5], Full(pi,1,3))\n(value = 15, size = nothing)  # sum\n\njulia\u003e restrict([10 0 5], Full)  # doesn't depend on the point, just type\n1×3 Full{Float64, 2}:\n 5.0  5.0  5.0                # mean\n\njulia\u003e naturalise((value = 15, size = nothing), Full(pi,1,3))  # needs the size\n1×3 Full{Float64, 2}:\n 5.0  5.0  5.0                # value / length\n```\n\nWhat those tests don't check is that `naturalise` maps into the cotangent subspace corresponding to the submanifold defined by the type.\nThis is a bit more involved to check, but is the argument against this returning say `[15 0 0]`, which is in the same equivalence class as `[10 0 5]` and `[5 5 5]` according to `uncollect`.\nYou can try the above tests with `x = Full(2.0, 3, 3; one=true)`, but the new check is this:\n\n```julia\njulia\u003e restrict([10 0 5], Full(pi, 1, 3; one=true))\n1×3 OneElement(::Int64):\n 15  0  0\n\njulia\u003e subspacetest(Full, 2.0, 3, 3; one=true);\n┌ Warning: naturalise(dw, Full) has components in 8 directions outside T*S\n└ @ OddArrays ~/.julia/dev/OddArrays/src/OddArrays.jl:958\n\njulia\u003e subspacetest(Full, 2.0, 3, 3);\n[ Info: naturalise(dw, Full) seems to live in T*S, as it should\n```\n\nLess obviously, there is a correct projection for `Range` objects:\n\n```julia\njulia\u003e uncollect([1,1,13], Range(1,2,3))\n(start = 1.5, stop = 13.5, len = nothing)\n\njulia\u003e naturalise(ans, Range(1,2,3))\n3-element Range{Float64}, with step = 6.0:\n -1.0\n  5.0\n 11.0\n\njulia\u003e ans ≈ restrict([1,1,13], Range)\ntrue\n\njulia\u003e x = Range(0,ℯ,5); dx = rand(5); dx2 = rand(5);  # for above @testset\n\njulia\u003e subspacetest(Range, 1.2, 3.4, 5);\n[ Info: naturalise(dw, Range) seems to live in T*S, as it should\n```\n\nFor rotation matrices, an example of `restrict` which passes the above tests but fails `subspacetest` is this:\n\n```julia\njulia\u003e restrict([1 2; 3 4], Rotation(pi/3))\n2×2 AntiSymOne{Float64}:\n  0.0      3.83013\n -3.83013  0.0\n\njulia\u003e x = Rotation(randn()); dx = randn(2,2); dx2 = randn(2,2);  # for above @testset\n\njulia\u003e subspacetest(Rotation, pi/3);\n┌ Warning: naturalise(dw, Rotation) has components in 3 directions outside T*S\n└ @ OddArrays ~/.julia/dev/OddArrays/src/OddArrays.jl:958\n\n```\n\nIt's pretty that the cotangent lives in the Lie algebra, but in fact irrelevant.\nThe way to stay inside the submanifold is to use the dual part of this, which you could represent as a scaled rotation matrix:\n\n```julia\njulia\u003e using ForwardDiff: Dual\n\njulia\u003e Rotation(Dual(pi/3, 1000))\n2×2 Rotation{Dual{Nothing, Float64, 1}}, with theta = Dual{Nothing}(1.0471975511965976,1000.0):\n Dual{Nothing}(0.5,-866.025)    Dual{Nothing}(-0.866025,-500.0)\n Dual{Nothing}(0.866025,500.0)   Dual{Nothing}(0.5,-866.025)\n```\n\nThis can't be a `Rotation` struct, in fact that's obvious from the start as the cotangent representation has to be a vector space, but the sum of two rotation matrices is outside the set.\n\n\n## Over-parameterised types\n\nThese store more numbers than there are dimensions in the matrix subspace. They have unambiguous \"structural\" gradients:\n\n```julia\njulia\u003e gradient(x -\u003e x[1], UnitVector([3,0,4]))\n┌ Info: projecting to Tangent{UnitVector}\n└   typeof(dx) = OneElement{Float64, 1, Tuple{Int64}, Tuple{Base.OneTo{Int64}}}\n[ Info: generic _uncollect\n((raw = [0.128, 0.0, -0.096],),)\n\njulia\u003e gradient(x -\u003e x[1], Outer(3, [4 5; 6 7]))\n┌ Info: projecting to Tangent{Outer}\n└   typeof(dx) = Matrix{Int64} (alias for Array{Int64, 2})\n((x = 4, y = [3 0; 0 0], size = nothing),)\n\njulia\u003e gradient(x -\u003e sum(abs2, x), Mask([1,NaN,3], [40,50,60]))  # with default OddArray projection\n┌ Info: projecting to Tangent{Mask}\n└   typeof(dx) = Vector{Float64} (alias for Array{Float64, 1})\n((alpha = [2.0, 0.0, 6.0], beta = [0.0, 100.0, 0.0]),)\n```\n\nDo they have \"natural\" ones? For `Mask` can you just add `alpha + beta`:\n\n```julia\njulia\u003e restrict([1 2 3], Mask)\n1×3 Matrix{Int64}:\n 1  2  3\n\njulia\u003e naturalise((alpha = [2 0 6], beta = [0 100 0]), Mask)\n1×3 Matrix{Int64}:\n 2  100  6\n\njulia\u003e x = Mask([1,NaN,3], [40,50,60]); dx = randn(3); dx2 = randn(3);  # for above @testset\n\njulia\u003e subspacetest(Mask, [1,NaN,3], [40,50,60]);\n[ Info: naturalise(dw, Mask) seems to live in T*S, as it should\n\njulia\u003e gradient(x -\u003e x.beta[1]^2, x)  # reads a field which doesn't contribute... garbage primal?\n([80.0, 0.0, 0.0],)\n```\n\nFor `Outer`, there is more serious redundancy, `Outer([4], [9,9]) == Outer([6], [6,6]) == Outer([9], [4,4])` describe the matrix `x`. And the constructor is nonlinear.\nYou can still make a valid `naturalise`, I think, but it's not trivial and it cannot in general re-use the struct:\n\n```julia\njulia\u003e uncollect([3 0 0; 0 0 0], Outer([5,5], [7,7,7]))  # S is 4 dimensional here\n(x = [21, 0], y = [15, 0, 0], size = nothing)\n\njulia\u003e naturalise(ans,  Outer([5,5], [7,7,7]))  # this cannot be written as Outer\n2×3 Matrix{Float64}:\n 2.0   0.5   0.5\n 1.0  -0.5  -0.5\n\njulia\u003e uncollect(ans, Outer([5,5], [7,7,7]))\n(x = [21.0, -8.881784197001252e-16], y = [15.0, -8.881784197001252e-16, 0.0], size = nothing)\n\njulia\u003e subspacetest(Outer, [5,6], [7,8,9]);\n[ Info: naturalise(dw, Outer) seems to live in T*S, as it should\n```\n\nThe case `Outer(::Matrix, ::Number)` is simpler:\n\n```julia\njulia\u003e uncollect([1 10 100], Outer([4 5 6], 7))\n(x = [7 70 700], y = 654, size = nothing)\n\njulia\u003e naturalise(ans, Outer([4 5 6], 7))\n1×3 Matrix{Float64}:\n 1.0  10.0  100.0\n\njulia\u003e ans == Outer([1 10 100], 1) == Outer([2 20 200], 1/2)  # but no advantage\ntrue\n```\n\nNext, `PDiagMat` stores both the diagonal and its inverse. It specialises `*` of two such to produce a third, and opts out of the generic rule:\n\n```julia\njulia\u003e gradient(x -\u003e (x * x)[5], PDiagMat([1,2,3]))\n[ Info: *(::PDiagMat, ::PDiagMat)\n┌ Info: projecting to Tangent{PDiagMat}\n└   typeof(dx) = Matrix{Float64} (alias for Array{Float64, 2})\n((dim = nothing, diag = [0.0, 4.0, 0.0], inv_diag = nothing),)\n\njulia\u003e gradient(x -\u003e (x * _inv(x))[5], PDiagMat([1,2,3]))  # weird, uncollect could never make this\n[ Info: _inv(::PDiagMat)\n[ Info: *(::PDiagMat, ::PDiagMat)\n┌ Info: projecting to Tangent{PDiagMat}\n└   typeof(dx) = Matrix{Float64} (alias for Array{Float64, 2})\n((dim = nothing, diag = [0.0, 0.5, 0.0], inv_diag = [0.0, 2.0, 0.0]),)\n\njulia\u003e gradient(x -\u003e (PDiagMat(x) * _inv(PDiagMat(x)))[5], [1,2,3])\n[ Info: _inv(::PDiagMat)\n[ Info: *(::PDiagMat, ::PDiagMat)\n┌ Info: projecting to Tangent{PDiagMat}\n└   typeof(dx) = Matrix{Float64} (alias for Array{Float64, 2})\n([0.0, 0.0, 0.0],)\n```\n\nHaven't sorted these ones out.\n\n## Discussed elsewhere\n\nThis PR https://github.com/JuliaDiff/ChainRulesCore.jl/pull/449 contains some comparable maps. (Formatted [notes.md](https://github.com/JuliaDiff/ChainRulesCore.jl/blob/wct/writing-generic-rrules/notes.md) and [examples](https://github.com/JuliaDiff/ChainRulesCore.jl/blob/wct/writing-generic-rrules/examples.jl).)\n\n* Since `destructure == collect`, the useful map from Matrix to Tangent is called `destructure_pullback` or else `pullback_of_destructure(x)(dx)` for `uncollect(dx, x)` here.\n\n* There is also a \"Restructure\", and I think `pullback_of_restructure` is playing a role like `naturalise` here. But I am not very sure.\n\n* The `ScaledVector` example there is much like `Outer(pi, [0 1 2])` here, but `Outer` allows other things.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmcabbott%2Foddarrays.jl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmcabbott%2Foddarrays.jl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmcabbott%2Foddarrays.jl/lists"}