{"id":22416933,"url":"https://github.com/jloveric/high-order-layers-torch","last_synced_at":"2025-08-21T13:31:30.437Z","repository":{"id":37675130,"uuid":"241753889","full_name":"jloveric/high-order-layers-torch","owner":"jloveric","description":"High order and sparse layers in pytorch.  Lagrange Polynomial, Piecewise Lagrange Polynomial, Piecewise Discontinuous Lagrange Polynomial (Chebyshev nodes) and Fourier Series layers of arbitrary order.  Piecewise implementations could be thought of as a 1d grid (for each neuron) where each grid element is Lagrange polynomial.  Both full connected and convolutional layers included.","archived":false,"fork":false,"pushed_at":"2024-02-22T03:44:21.000Z","size":2973,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-04-14T07:35:20.740Z","etag":null,"topics":["chebyshev-polynomials","deeplearning","discontinuous","fluid-dynamics","fourier-series","grid","high-order-methods","hp-refinement","implicit-representions","lagrange-polynomial-interpolation","piecewise-polynomial","pytorch","pytorch-lightning","sparse-network","spline"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jloveric.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-20T00:01:58.000Z","updated_at":"2024-05-04T14:41:12.462Z","dependencies_parsed_at":"2023-12-30T16:33:03.914Z","dependency_job_id":"e9c8289e-6653-4a9f-9b82-563a26df4cc9","html_url":"https://github.com/jloveric/high-order-layers-torch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jloveric%2Fhigh-order-layers-torch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jloveric%2Fhigh-order-layers-torch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jloveric%2Fhigh-order-layers-torch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jloveric%2Fhigh-order-layers-torch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jloveric","download_url":"https://codeload.github.com/jloveric/high-order-layers-torch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230516100,"owners_count":18238346,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chebyshev-polynomials","deeplearning","discontinuous","fluid-dynamics","fourier-series","grid","high-order-methods","hp-refinement","implicit-representions","lagrange-polynomial-interpolation","piecewise-polynomial","pytorch","pytorch-lightning","sparse-network","spline"],"created_at":"2024-12-05T15:16:46.318Z","updated_at":"2024-12-19T23:39:09.489Z","avatar_url":"https://github.com/jloveric.png","language":"Python","funding_links":[],"categories":["Library"],"sub_categories":["Alternative"],"readme":"[![CI](https://github.com/jloveric/high-order-layers-torch/actions/workflows/python-app.yml/badge.svg)](https://github.com/jloveric/high-order-layers-torch/actions/workflows/python-app.yml)\n\n\n# Piecewise Polynomial Layers and Other High Order Layers in PyTorch\n\nThis is a PyTorch implementation of my tensorflow [repository](https://github.com/jloveric/high-order-layers) and is more complete due to the flexibility of PyTorch.\n\nLagrange Polynomial, Piecewise Lagrange Polynomial, Discontinuous Piecewise Lagrange Polynomial, Fourier Series, sum and product layers in PyTorch.  The sparsity of using piecewise polynomial layers means that by adding new segments the representational power of your network increases, but the time to complete a forward step remains constant. Implementation includes simple fully connected layers, convolution layers and deconvolutional layers using these models. This is a PyTorch implementation of this [Discontinuous Piecewise Polynomial Neural Networks](https://www.researchgate.net/publication/276923198_Discontinuous_Piecewise_Polynomial_Neural_Networks) which was written almost a decade before the recent interest in KAN's, including huge number of extensions including continuous, Fourier series and convolutional neural networks... and many applications with varrying degrees of success. If you\ncome from a computational physics background, this type of approach seems very obvious as ancient techniques like the finite element method are discretized\nthis way.\n\n## Collab Notebook\nUsing simple high order layers\n[Simple function approximation](https://colab.research.google.com/drive/1kew0Kz4v5GB5D59-wP1rHZuCdhYknz4s?usp=sharing)\n\nUsing simple high order MLP\n[2d function approximation](https://colab.research.google.com/drive/14wSNzBUFYk-1o6fuqiux_y33aV9VuwkF?usp=sharing)\n\n\n## Idea\n\nThe idea is extremely simple, instead of a weight at the synapse we have a function F(x) that can be arbitrarily complex. As a practical matter I implement this by using multiple weights corresponding to each link, these weight are used as parameters of the function, and to make sure there is still some GPU efficiency, these weights are just coefficients of the basis functions.  In most of this work, the n-weights describe the value of a piecewise polynomial on a regular grid (in the case of a piecewise polynomial) each of the n-weights can be updated independently. A Lagrange polynomial and Gauss Lobatto points are used to minimize oscillations of the polynomial.  The same approach can be applied to any \"functional\" synapse, and I also have Fourier series synapses in this repo as well. Because the non-linearity is applied on the link, the node is simply a summation\n\nIn the image below each \"link\" instead of being a single weight, is a function of both x and a set of weights.  These functions can consist of an orthogonal basis functions for efficient approximation.\n\n\u003cimg src=\"plots/NetworkZoom.png\" width=50% height=50% style=\"display: block; margin: 0 auto\"\u003e\n\nA small layer then looks like this, the values at the nodes are just summed.\n\n\u003cimg src=\"plots/PiecewisePolynomialLayer.svg\" width=50% height=50% style=\"display: block; margin: 0 auto\"\u003e\n\nA single neuron input output pair with a piecewise function is shown below. In the case where we use polynomials, Lagrange polynomials are being used so the values of the weights are identical to the value of the function at that point. The spacing is determined by chebyshev lobatto points, so there are always weights at the edge of each segment. In the case of discontinuous polynomial, the weights there are 2 weights for each interior segment edge.\n\n\u003cimg src=\"plots/NeuronDrawing.svg\" width=50% height=50% style=\"display: block; margin: 0 auto\"\u003e\n\nThe image below shows the function passing through the weights when using lagrange polynomials. Note that there is no derivative continuity at the boundaries.\n\n\u003cimg src=\"plots/NeuronDrawingWeights.svg\" width=50% height=50% style=\"display: block; margin: 0 auto\"\u003e\n\n\n## Why\n\nUsing higher order polynomial representations allow networks with much fewer total weights in certain cases. There is a\nwell known phenomena in numerical modeling known as exponential convergence using spectral methods when using hp refinement,\nit's possible something like that can happen in neural networks as well.\n\n## Is this a KAN?\n\nActually a single layer piecewise polynomial KAN (which is actually 2 layers) is a special case of a 2 layer\npiecewise polynomial network, which is used in this repo. Therefore, a piecewise polynomial layer is actually \"Half a KAN\" so\nit's actually simpler - Often all you need is a single polynomial layer at the input followed by a standard MLP so\nhaving the piecewise polynomial layer is important. Other names that have been used in the past Deep FLANN\n(functional link artificial neural network).\n\nLagrange polynomials are widely used in finite element analysis and have the advantage that the value of the weight\nis actually the value of the function at that point in space. By limiting the weights you are limiting the maximum\nvalue of the function (the function may be higher than the weights in between the nodes). Also, when you go beyond the\nrange of definition [-1,1] the polynomial is still defined using the last (or first) polynomial in the sequence, whether you want it defined that way at high polynomial order is another question. I mention a paper at the bottom where they do a linear\nextension beyond the range [-1,1] so values do not rise too fast - but normalization works as well.\n\n## Issues\n\nWhat about instabilities due to steep gradients? Seems like you can get around those with various approaches, polynomial\nrefinement is one (start with piecewise linear and than increase the polynomial order after it converges), the lion\noptimizer helps a lot as well, while sophia may be even better since it's second order.\n\nThe biggest issues I've experienced though are that it's slower than dense networks and certain operations can\ntake up more memory which can cause major issues with models that already push the limits of your gpu. Now that\nKANs are popular, hopefully there will be enough people to address all these issues.\n\nIn general, with enough effort, it seems I can make them \"work\" for any place the classic ReLU network works and\nin certain situations they clearly work much better. They also do a great job of overfitting, which just means,\nI need more data. For problems where your inputs are positional, x and y..., they seem to be far better.\n\nFinally, I believe these methods actually will benefit much more from (approximate) second order optimizers. I used those in\nmy original implementation. Although there are plenty of second order optimizers out there, to date, pytorch does\nnot have a standard one except LBFGS which has its own issues.\n\n## Fully Connected Layer Types\nAll polynomials are Lagrange polynomials with Chebyshev interpolation points.\n\nA helper function is provided in selecting and switching between these layers\n\n```python\nfrom high_order_layers_torch.layers import *\nlayer1 = high_order_fc_layers(\n    layer_type=layer_type,\n    n=n,\n    in_features=784,\n    out_features=100,\n    segments=segments,\n)\n```\n\nwhere `layer_type` is one of\n| layer_type          | representation\n|--------------------|-------------------------|\n|continuous         |  piecewise polynomial using sum at the neuron |\n|continuous_prod    |  piecewise polynomial using products at the neuron |\n|discontinuous      |  discontinuous piecewise polynomial with sum at the neuron|\n|discontinuous_prod | discontinous piecewise polynomial with product at the neuron|\n|polynomial | single polynomial (non piecewise) with sum at the neuron|\n|polynomial_prod | single polynomial (non piecewise) with product at the neuron|\n|product | Product |\n|fourier | fourier series with sum at the neuron |\n\n\n`n` is the number of interpolation points per segment for polynomials or the number of frequencies for fourier series, `segments` is the number of segments for piecewise polynomials, `alpha` is used in product layers and when set to 1 keeps the linear part of the product, when set to 0 it subtracts the linear part from the product.\n\n## Convolutional Layer Types\n\n```python\nconv_layer = high_order_convolution_layers(layer_type=layer_type, n=n, in_channels=3, out_channels=6, kernel_size=5, segments=segments, rescale_output=rescale_output, periodicity=periodicity)\n```\n\nAll polynomials are Lagrange polynomials with Chebyshev interpolation points.\n| layer_type   | representation       |\n|--------------|----------------------|\n|continuous(1d,2d)   | piecewise continuous polynomial\n|discontinuous(1d,2d) | piecewise discontinuous polynomial\n|polynomial(1d,2d) | single polynomial\n|fourier(1d,2d) | fourier series convolution\n\n## Initializing of layers\nThe default initialization is to initialize each link to a random constant, i.e. all weights have the same value in a link. This seems to work\npretty well, however, I also have linear random linear initialization (non constant). The implementation of the linear initialization is slower and I'm not sure it's actually better.\n\n Here is a function that does this linear initialization for non convolutional layers (it can be found in [networks.py](https://github.com/jloveric/high-order-layers-torch/blob/master/high_order_layers_torch/networks.py))\n```\ndef initialize_network_polynomial_layers(\n    network: nn.Module,\n    max_slope: float,\n    max_offset: float,\n    scale_slope: Callable[[float], float] = lambda input_size: 1,\n)\n```\n\n## h and p refinement\np refinement is taking an existing network and increasing the polynomial order of that network without changing the network output.  This allow the user to train a network at low polynomial order and then use that same network to initialize a network with higher polynomial order.  This is particularly useful since a high order polynomial network will often converge poorly without the right initialization, the lower order network provides a good initial solution.  The function for changing the order of a network is\n```\nfrom high_order_layers_torch.networks import interpolate_high_order_mlp\ninterpolate_high_order_mlp(\n    network_in: HighOrderMLP, network_out: HighOrderMLP\n```\ncurrent implementation only works with high order MLPs, not with convnets.  A similar function exists for h refinement.  h refinement is\nrefining the number of segments in a layer, and is used for similar reasoning.  Layers with lots of segments may be slow to converge\nso the user starts with a small number of segments (1 or 2) and then increases the number of segments (h) using the lower initialization.  The following function currently only works for high order MLPs, not with convnets\n```\nfrom high_order_layers_torch.network import hp_refine_high_order_mlp\nhp_refine_high_order_mlp(\n    network_in: HighOrderMLP, network_out: HighOrderMLP\n)\n```\n# Installing\n\n## Installing locally\n\nThis repo uses poetry, so run\n\n```\npoetry install\n```\n\nand then\n\n```\npoetry shell\n```\n\n## Installing from pypi\n\n```bash\npip install high-order-layers-torch\n```\n\nor\n\n```\npoetry add high-order-layers-torch\n```\n# Examples\n\n## Simple function approximation\n\nApproximating a simple function using a single input and single output (single layer) with no hidden layers\nto approximate a function using continuous and discontinuous piecewise polynomials (with 5 pieces) and simple\npolynomials and fourier series.  The standard approach using ReLU is non competitive.  To see more complex see\nthe implicit representation page [here](https://github.com/jloveric/high-order-implicit-representation).\n\n![piecewise continuous polynomial](plots/piecewise_continuous.png)\n![piecewise discontinuous polynomial](plots/piecewise_discontinuous.png)\n![polynomial](plots/polynomial.png)\n![fourier series](plots/fourier_series.png)\n\n```python\npython examples/function_example.py\n```\n\n## XOR : 0.5 for x*y \u003e 0 else -0.5\n```python\n python3 examples/xor.py\n```\n\nSimple XOR problem using the standard network structure (2 inputs 2 hidden 1 output) this will also work with no hidden layers. The function is discontinuous along the axis and we try and fit that function. Using piecewise discontinuous layers the model can match the function exactly.\n![piecewise discontinuous polynomial](plots/xor_discontinuous.png)\nWith piecewise continuous.\n![piecewise continuous polynomial](plots/xor_continuous.png)\nWith polynomial using similar number of parameters.\n![polynomial](plots/xor_polynomial.png)\nUsing 2D polynomial \"layer\" this is just a single input and single output. The polynomial 2d is a link that takes in 2 variables (a 2d vector) and in this\ncase outputs a single value. It's the cartesian product of the basis function of a single polynomial function, but in the x and y direction. Using 5 point polynomials would have 25 basis functions in 2d whereas it has only 5 basis functions in 1d.\n```\npython3 examples/xor.py layer_type=polynomial_2d optimizer.lr=0.01 optimizer=sophia epochs=100\n```\n![polynomial 2d](plots/xor_polynomial_2d.png)\n## MNIST (convolutional)\n\n```python\npython examples/mnist.py -m train_fraction=1 layer_type=polynomial2d,discontinuous2d,continuous2d n=2,3,4,5,6 segments=2 max_epochs=40 batch_size=1024 channels=[12,32] kernel_size=5 optimizer=sophia optimizer.lr=1e-3\n```\nBelow using max_abs and sophia kernel_size=5, channels=[12,32]. The output layer was a standard linear\nlayer - so only the convolutions (2 of them) were high order. Nothing particularly interesting here, except\nthat instability doesn't seem to be an issue. I only ran each test once. 40 epochs batch size 1024. The case\nof the polynomial where n=2 is actually linear, so I would expect about 0.89 score, but there is nonlinearity\nthrough the normalization layer (and max pooling). Also note that in the linear case, every link has 2 weights.\n| n   | test (polynomial)      | test (continuous) | test (discontinuous)\n|--------------|----------------------|------------------|------------------|\n|2 | 0.986 | 0.988 | 0.985\n|3 | 0.987 | 0.989 | 0.989\n|4 | 0.987 | 0.989 | 0.986\n|5 | 0.987 | 0.988 | 0.987\n|6 | 0.989 | 0.986 | 0.984\n\nreduce the size of the network, same as above but channels=[3,8]\n| n   | test (polynomial)      | test (continuous) | test (discontinuous)\n|--------------|----------------------|------------------|------------------|\n|2 | 0.969 | 0.977 | 0.980\n|3 | 0.978 | 0.983 | 0.983\n|4 | 0.984 | 0.983 | 0.984\n|5 | 0.983 | 0.984 | 0.983\n|6 | 0.981 | 0.982 | 0.982\n\nCreating a deeper network with 4 convolutions and max_abs normalization between layers, 2 segments, avg pooling after each pair\nof convolutions\n```\npython examples/mnist.py -m train_fraction=1 layer_type=discontinuous2d polynomial2d,continuous2d n=2,3,4,5,6 segments=2 max_epochs=100 batch_size=1024 channels=[16,32] output_layer_type=auto double=True kernel_size=3 optimizer=sophia optimizer.lr=1e-3\n```\n| n   | test (polynomial)      | test (continuous) | test (discontinuous)\n|--------------|----------------------|------------------|------------------|\n|2 | 0.927 | 0.990 | 0.989\n|3 | 0.989 | 0.989 | 0.988\n|4 | 0.989 | 0.989 | 0.990\n|5 | 0.990 | 0.988 | 0.989\n|6 | 0.987 | 0.985 | 0.990\n|7 | 0.989 |     |\n|8 | 0.989 |     |\n|9 | 0.988 |     |\n|10| 0.989 |     |\n\nReduced the learning rate on sophia by a factor of 10, run up to a 9th order polynomial\n```\npython examples/mnist.py -m train_fraction=1 layer_type=polynomial2d n=2,3,4,5,6,7,8,9,10 segments=2 max_epochs=100 batch_size=1024 channels=[16,32] output_layer_type=auto double=True kernel_size=3 normalization=max_abs optimizer=sophia optimizer.lr=1e-4\n```\n| n   | test (polynomial)\n|--------------|----------------------|\n|2 | 0.910 |\n|3 | 0.985 |\n|4 | 0.988 |\n|5 | 0.989 |\n|6 | 0.988 |\n|7 | 0.990 |\n|8 | 0.991 |\n|9 | 0.991 |\n|10| 0.989 |\n\nOnly Using 2 convolutions and lion optimizer\n```\npython examples/mnist.py -m train_fraction=1 layer_type=polynomial2d n=2,3,4,5,6,7,8,9,10 segments=2 max_epochs=100 batch_size=1024 channels=[16,32] output_layer_type=auto,linear double=False kernel_size=5 normalization=max_abs optimizer=lion optimizer.lr=1e-4\n```\nThe results below are the same problem, but one has a polynomial last layer (auto) and the other has a linear last layer.\nThe polynomial last layer is the same as that used in the convolution.\n\n| n   | polynomial polynomial| polynomial linear|\n|--------------|----------------------|---------------|\n|2 | 0.928 | 0.930\n|3 | 0.984 | 0.984\n|4 | 0.991 | 0.990\n|5 | 0.989 | 0.990\n|6 | 0.990 | 0.990\n|7 | 0.989 | 0.989\n|8 | 0.989 | 0.989\n|9 | 0.990 | 0.989\n|10| 0.990 | 0.990\n\n## MNIST Block 3D polynomial / block piecewise polynomial\nBasics implemented, needs to be made more efficient and model needs to change a bit. Currently uses a single NxNxN polynomial\nwith 3 inputs (grayscale color, x, y) and 10 outputs to solve mnist. Not performant, but will work on it.\n```\npython examples/block_mnist.py optimizer=sophia optimizer.lr=1e-3 batch_size=32 max_epochs=100 n=20 layer_type=polynomial_3d\n```\nwith accuracy 0.9289000034332275\nand\n```\npython examples/block_mnist.py optimizer=lion optimizer.lr=1e-3 batch_size=16 max_epochs=100 n=28 layer_type=polynomial_3d\n```\nand then for block piecewise polynomial\n```\npython examples/block_mnist.py optimizer=lion optimizer.lr=1e-3 batch_size=16 max_epochs=100 n=3 segments=10 layer_type=continuous_nd\n```\n## CIFAR100 (convolutional)\n\n```\npython examples/cifar100.py -m max_epochs=20 train_fraction=1.0 layer_type=polynomial segments=2 n=7 nonlinearity=False rescale_output=False periodicity=2.0 lr=0.001 linear_output=False\n```\n## Autoencoder (haven't finished implementing)\n\n```\npython examples/autoencoder.py -m max_epochs=300 train_fraction=1.0\n```\n\n## Variational Autoencoder\nStill a WIP.  Does work, but needs improvement.\n```\npython examples/variational_autoencoder.py -m max_epochs=300 train_fraction=1.0\n```\nrun with nevergrad for parameter tuning\n```\npython examples/variational_autoencoder.py -m\n```\n\n## Invariant MNIST (fully connected)\nSome comparisons using parameter scans maxabs normalization as default. piecewise polynomial cases use 2 segments. I only\ndid one run each.\n```\npython3 examples/invariant_mnist.py -m mlp.n=2,3,4,5,6 mlp.hidden.width=128 mlp.layer_type=polynomial optimizer=sophia\n```\nto run all the examples in the table\n```\n./scripts/invariant_mnist.sh\n```\nAll the following results are a single hidden layer of 128 units. I only ran\none test case each, but it looks like sophia is the best optimizer. Layernorm\nis best until you get to higher order and then max_abs is better, most likely\nbecause it does a better job of keeping in [-1,1]. I'll try and get the higher\norders to where I think they can be. Also, all of these memorize the training set.\n\nBelow using max_abs and sophia\n| n   | test (polynomial)      | test (continuous) | test (discontinuous)\n|--------------|----------------------|------------------|------------------|\n|2 | 0.950 | 0.974 | 0.973\n|3 | 0.979 | 0.970 | 0.963\n|4 | 0.971 | 0.970 | 0.961\n|5 | 0.965 | 0.960 | 0.964\n|6 | 0.967 | 0.963 | 0.960\n\nUsing layernorm and sophia\n| n   | test (polynomial)      | test (continuous) | test (discontinuous)\n|--------------|----------------------|------------------|------------------|\n|2 | 0.959| 0.979 | 0.972\n|3 | 0.977 | 0.974 | 0.969\n|4 | 0.971 | 0.962 | 0.965\n|5 | 0.966 | 0.954 | 0.953\n|6 | 0.951 | 0.946 | 0.947\n\nUsing layernorm and lion\n| n   | test (polynomial)      | test (continuous) | test (discontinuous)\n|--------------|----------------------|------------------|------------------|\n|2 | 0.948| 0.977 | 0.975\n|3 | 0.979 | 0.970 | 0.967\n|4 | 0.972 | 0.968 | 0.965\n|5 | 0.966 | 0.955 | 0.955\n|6 | 0.960 | 0.949 | 0.949\n\nUsing max_abs and lion\n| n   | test (polynomial)      | test (continuous) | test (discontinuous)\n|--------------|----------------------|------------------|------------------|\n|2 | 0.113 | 0.970 | 0.970\n|3 | 0.972 | 0.971 | 0.963\n|4 | 0.966 | 0.962 | 0.966\n|5 | 0.966 | 0.960 | 0.964\n|6 | 0.962 | 0.953 | 0.960\n\nWithout polynomial refinement\n```python\npython examples/invariant_mnist.py max_epochs=100 train_fraction=1 mlp.layer_type=continuous mlp.n=5 mlp.p_refine=False mlp.hidden.layers=4\n```\nwith polynomial refinement (p-refinement)\n```\npython examples/invariant_mnist.py max_epochs=100 train_fraction=1 layer_type=mlp.continuous mlp.n=2 mlp.target_n=5 mlp.p_refine=True\n```\nI've also added hp refinement, but it needs a lot of testing.\n\n## Implicit Representation\n\nAn example of implicit representation for image compression, language generation can be found [here](https://github.com/jloveric/high-order-implicit-representation).  I intend to explore generative models in natural language further [here](https://github.com/jloveric/language-interpolation)\n\n## PDEs in Fluid Dynamics\n\nAn example using implicit representation to solve hyperbolic (nonlinear) wave equations can be found [here](https://github.com/jloveric/neural-network-pdes)\n\n## Natural Language Generation\n\nExamples using these networks for natural language generation can be found\n[here](https://github.com/jloveric/language-interpolation)\n\n## Generative music\n\nNo real progress here\n[here](https://github.com/jloveric/high-order-generative-music)\n\n\n## Test and Coverage\n\nAfter installing and running\n```\npoetry shell\n```\nrun\n```\npytest\n```\nfor coverage, run\n```\ncoverage run -m pytest\n```\nand then\n```\ncoverage report\n```\n## A note on the product unit (I rarely use anymore)\nThe layers used here do not require additional activation functions and use a simple sum or product in place of the activation.\nI almost always use sum units, but product units are performed in this manner\n\n$$ product=-1+\\prod_{i}(1 + f_{i})+(1-\\alpha)\\sum_{i}f_{i} $$\n\nThe 1 is added to each function output to as each of the sub products is also computed.  The linear part is controlled by\nthe alpha parameter.\n\n## Notes on optimizer\nThe Lion or Sophia optimizers seems to be the best choice since it performs better than Adam in general, but seems to work especially well\nfor the case of polynomials.\n\n## Notes on normalization\nAlthough you can use batchnorm, layernorm etc... work better, I've found that you can actually just use the infinity norm (\"max_abs\" norm) which has no parameters\nfor this formulation (same approach seems not to work very well for standard relu networks - but need to investigate this further).\nThe max_abs normalization is defined this way\n```\nnormalized_x = x/(max(abs(x))+eps)\n```\nwhere the normalization is done per sample (as opposed to per batch).  The way the layers are formulated, we don't want the neuron\nvalues to extend beyond [-1, 1] as the polynomial values grow rapidly beyond that range.  You can also use mirror periodicity to keep the\nvalues within from growing rapidly. We want the values to cover the entire range [-1, 1] of the polynomials as the weights\nare packed towards the edges of each segment (though using even number of segments means you'll have a lot of weights near the origin).\n\n\n## Reference\nYou can refer to this repo here\n```\n@misc{Loverich2020,\n  author = {Loverich, John},\n  title = {High Order Layers Torch},\n  year = {2020},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/jloveric/high-order-layers-torch}},\n}\n```\nand there is an old paper which was originally written in c++ and doesn't cover\nnearly as much as this repo and all the dependent repos, which you can also refer\nto\n```\n@article{loverich2015discontinuous,\n  title={Discontinuous Piecewise Polynomial Neural Networks},\n  author={Loverich, John},\n  journal={arXiv preprint arXiv:1505.04211},\n  year={2015}\n}\n```\n## Notes\n\n[KAN: Kolmogorov–Arnold Networks 2024](https://arxiv.org/pdf/2404.19756) was published (9 years after the original implementation of the technique in this repo), where B-splines were used on the grid. Looking at that repo, the real difference seems to be B-splines vs lagrange\npolynomials.\n\n[Variations on the Chebyshev-Lagrange Activation Function](https://arxiv.org/abs/1906.10064) implements a linear extension\nto the values beyond [-1,1] which would solve the problem of polynomial growth outside that range.\n\n[KAN: Kolmogorov–Arnold Networks: A review 2024](https://vikasdhiman.info/reviews/KAN_a_review.pdf) A review of KANs with respect to other types of networks, especially spline networks\n\n[Linear spline networks 2020](https://arxiv.org/pdf/2001.06263)\n\n[Learning Activation Functions in Deep (Spline) Neural Networks 2020](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9264754) using B splines\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjloveric%2Fhigh-order-layers-torch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjloveric%2Fhigh-order-layers-torch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjloveric%2Fhigh-order-layers-torch/lists"}