{"id":23280840,"url":"https://github.com/jloveric/language-interpolation","last_synced_at":"2026-02-18T02:35:24.446Z","repository":{"id":44425313,"uuid":"376592550","full_name":"jloveric/language-interpolation","owner":"jloveric","description":"Experiments in language interpolation with high order sparse neural networks","archived":false,"fork":false,"pushed_at":"2024-10-10T03:39:09.000Z","size":1626,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-12-19T23:39:04.869Z","etag":null,"topics":["deep-learning","gutenberg","high-order-methods","kan","lion","mamba","natural-language-processing","piecewise-polynomial","pytorch-lightning","sparse-mlp","sparse-network"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jloveric.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-13T16:39:59.000Z","updated_at":"2024-10-10T03:39:13.000Z","dependencies_parsed_at":"2023-11-27T01:25:53.611Z","dependency_job_id":"dd7f1637-3fcb-4fb4-b9e5-07144fe1279e","html_url":"https://github.com/jloveric/language-interpolation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jloveric%2Flanguage-interpolation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jloveric%2Flanguage-interpolation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jloveric%2Flanguage-interpolation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jloveric%2Flanguage-interpolation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jloveric","download_url":"https://codeload.github.com/jloveric/language-interpolation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238540799,"owners_count":19489476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","gutenberg","high-order-methods","kan","lion","mamba","natural-language-processing","piecewise-polynomial","pytorch-lightning","sparse-mlp","sparse-network"],"created_at":"2024-12-19T23:39:11.026Z","updated_at":"2025-10-18T22:06:59.122Z","avatar_url":"https://github.com/jloveric.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CI](https://github.com/jloveric/language-interpolation/actions/workflows/python-app.yml/badge.svg)](https://github.com/jloveric/language-interpolation/actions/workflows/python-app.yml)\n\n# Natural Language Generation with Sparse High Order Layers (MLP, Mamba)\nHigh order and piecewise networks for natural language generation (see [here](https://github.com/jloveric/high-order-layers-torch) for a description of High order layers being used).  The typical high order network design with piecewise polynomial\nlayers here is a fully connected network where each link has multiple segments.  Only one segment in\na link is active for each input so the network sparsity is determined by the number of segments. Although it looks like a standard MLP, the structure is more complicated and is a form of routing network with piecewise polynomials.\n\n![image](images/language-interpolation-drawing.png)\n\nI'm interested in creating larger language models from an ensemble of smaller models.  This would give better flexibility in adding or removing specific sources.\n\nWorking models for High Order MLPs, and High Order Mamba (SSM) as well\nas a few others.\n\n# Dataset\n\nData from project Gutenberg are used, either single or multiple books.  Training is done on the character level.  A pytorch lightning data module for project Gutenberg has been implemented for data loading.\n\n# Language interpolation of Books\nRun single case (data appears in outputs)\n```\npython examples/high_order_interpolation.py data.type=sequence\n```\nwith nevergrad (data appears in multirun)\n```\npython examples/high_order_interpolation.py -m data.type=sequence\n```\n# Decent parameters\nA few notes. For normalization, layer norm is the best, followed by maxabs and then no normalization . The only advantage\nto maxabs is that there are no trainable parameters. The best optimizer is Lion, it seems by a long shot. I think for\nthese type of networks that have potentially steep gradients due to the polynomials, this is especially the case (since it uses the sign of the gradient). So far,\nkaiming initialization seems to be performing better than linear initialization, but I need to investigate this further.\n\n### sparse mlp\nA few networks which are large enough to memorize \"The Dunwich Horror\" which is fairly short (120KB). Using Lion optimizer + learning rate scheduler.\n\n#### Piecewise constant\nPiecewise constant (requires discontinuous). Only the first layer can actually be optimized since derivatives beyond that are zero\n```\npython examples/high_order_interpolation.py data.type=sequence net=large_single_layer net.hidden.width=200 max_epochs=100 optimizer.lr=1e-3 batch_size=1000 net.layer_type=discontinuous net.n=1 net.segments=150 net.hidden.layers=3\n```\n#### Piecewise linear or better\n\n1 hidden layer 2 segments per link\n```\npython examples/high_order_interpolation.py data.type=sequence net=large_single_layer net.hidden.width=200 max_epochs=100 optimizer.lr=1e-4 batch_size=1000\n```\n2 hidden layers 2 segments per link\n```\npython examples/high_order_interpolation.py data.type=sequence net=large_double_layer max_epochs=100 net.hidden.width=250 optimizer.lr=1e-5\n``` \n1 hidden layer 2 segments per link\n```\npython examples/high_order_interpolation.py data.type=sequence net=large_single_layer net.hidden.width=250 max_epochs=100 net.n=3 optimizer.lr=1e-4\n```\n#### Higher order version\n\n3 hidden layers quadratic (n=3) 2 segments per link\n```\npython examples/high_order_interpolation.py data.type=sequence net=small net.hidden.width=250 max_epochs=100 net.n=3 net.hidden.layers=3 optimizer.lr=1e-5\n```\n3 hidden layers cubic (n=4) elements 2 segments per link\n```\npython examples/high_order_interpolation.py data.type=sequence net=small net.hidden.width=250 max_epochs=100 net.n=4 net.hidden.layers=3 optimizer.lr=1e-5\n```\n1 hidden layer with quartic (n=5) elements 2 segments per link\n```\npython examples/high_order_interpolation.py data.type=sequence net=large_single_layer net.hidden.width=200 max_epochs=100 optimizer.lr=1e-4 batch_size=1000 net.n=5\n```\n#### Only using high order for the input\nStandard ReLU network, however, the input layer is piecewise linear so that it can bin the characters into each segment.  The rest of the network\nlook like a standard net.\n```\npython examples/high_order_interpolation.py data.type=sequence net=large_standard net.hidden.width=1000 max_epochs=100 optimizer.lr=1e-4\n```\n#### Without any hidden layers\ncan you memorize with just input and output layers with no hidden layers?  In this example we get to about 95% accuracy.  Discontinuous works\nbetter than continuous as it has double the parameters for piecewise linear.  Increase the order of accuracy does nothing since the inputs are\ndiscrete and not continuous - in this case we should have a piecewise constant option, but then the gradients would be 0.\n```\npython examples/high_order_interpolation.py data.type=sequence net=large_single_layer net.hidden.layers=0 max_epochs=100 optimizer.lr=1e-4 batch_size=1000 net.layer_type=discontinuous\n```\n### High order Mamba (and low order)\nThe following haven't yet been optimized, but they run.\nFor low order (standard), specify layer_type=linear as below\n```\n python examples/high_order_interpolation.py data.type=sequence net=mamba optimizer.lr=1e-4 data.max_features=16 batch_size=1024 net.layer_type=linear\n ```\n for high order specify layer_type as 'continuous' or 'discontinuous' which converts the linear layers into piecewise polynomial layers. If hidden layers \u003e 0 it becomes\n a high order network instead of a linear layer.\n ```\n python examples/high_order_interpolation.py data.type=sequence net=mamba optimizer.lr=1e-4 data.max_features=16 batch_size=16 net.n_layer=2 net.n=3 net.segments=2 net.hidden_layers=0\n ```\n\n### High order transformers\nUsing high order transformer blocks. These are in development and not as good as the MLPs above.\n```\npython examples/high_order_interpolation.py data.type=sequence net=transformer max_epochs=100 optimizer.lr=1e-3 batch_size=512 net.layer_type=continuous data.repeats=5 net.n=2 data.max_features=10 optimizer.patience=20 initialize.type=linear\n```\nUsing only high order input, the rest being standard ReLU\n```\npython examples/high_order_interpolation.py data.type=sequence net=transformer max_epochs=100 optimizer.lr=1e-4 batch_size=16 net.layer_type=continuous data.repeats=1 net.n=3 data.max_features=20 optimizer.patience=20 initialize.type=linear accumulate_grad_batches=16 net.segments=2 net.model_type=high_order_input_transformer optimizer=lion\n```\n### sparse convolutional network\nUsing conv layers (not done too much here, see below for a possibly better network)\n```\npython examples/high_order_interpolation.py data.type=sequence net=conv max_epochs=100 optimizer.lr=1e-4 batch_size=1000 data.add_channel_dimension=true\n```\n### tail focus network\nUsing tail focus network you can handle much much longer sequences, however the accuracy needs to be much higher to not get garbage (random ascii characters that don't look like any language) for a given input\n```\npython examples/high_order_interpolation.py data.type=sequence net=tail_focus max_epochs=100 optimizer.lr=1e-3 batch_size=8000 data.add_channel_dimension=true\n```\nto run a prediction\n```\npython examples/high_order_interpolation.py train=False checkpoint=\\\"outputs/2022-07-12/12-26-21/lightning_logs/version_0/checkpoints/'epoch=74-step=237075.ckpt'\\\" topk=1 num_predict=10000 prompts=[\"Who are you?\"] data.add_channel_dimension=true net=tail_focus\n```\nand the result is jibberish sentences with a context of 256 characters and a single book and choosing the next best character.  I'll try this again at some point with a much larger context window and a much larger dataset.\n```\nWho are you?hhhIhr II hham hahe _I wa hhhar hit wohe _ower_ him hI whosow, hhough I wan og .hI. ht. __aday. Out hi sg them Cacler --a stabke wist his _Treag art unt.. The worn in the boumd di no ce. The hracises. Onte canheremhis counded teghing the hacling che conders. so collel, and as thing the eot to sheed an the wan or the lliused to the grom- of corning in. He who pout timetime, cu\\' to e onled. The opar nor ly the notike.. The that ,uen\\' forss will, liff us of that dert, the st bouthis spon the rills abec sire gors.  Then\\'t alite alline the scomery dowped distured the _ anda stipy rouse pre. The comch bor, the tale hotives to frows in the cagane of cearsite to giss it a mameverise on the ping, as if withh doed crown onligriat ffmled afisht to crothin sing it aningstib catedep with tilled, and tather it nowms a sraned the eNid that hef on  follitines reas of in the lights he at the listent frog the and arenguy the consus dis, and it that himm--thoold y\\'t heous. Ilikepad to was it ans on the tole of the shey. They mongite folker    sorece\\'s abon the loud sote mathers verite to Corgass. Thele. Octereried to enttones of the vision inse frabity.\n```\n## Low order mlp\nThis one uses an embedding as the first layer and averages to pass to the\nstandard MLP. The averaging means this is like \"bag of letters\" so don't expect\nsuper high accuracy. Still need to create a text generator for evaluation with this one\n```\npython examples/high_order_interpolation.py data.type=sequence net=large_standard net.hidden.width=1000 max_epochs=100 optimizer.lr=1e-4 net.model_type=low_order_mlp\n```\n## Dual Convolution\nThe idea is to repeately apply the same high order 1d convolution to reduce the input sequence to a single remaining vector. The update is dynamic and the number of times the convolution is applied depends on the length of the sequence. Remarkable this actually sort of works but is insanely slow. Maybe I can get faster convergence somehow.\n```\npython examples/high_order_interpolation.py data.type=sequence net=dual_convolution max_epochs=100 optimizer.lr=1e-5 batch_size=32 net.layer_type=discontinuous data.repeats=1 net.n=3 net.segments=4 data.max_features=10 optimizer.patience=20  net.embedding_dimension=128 net.hidden_width=1024 net.normalize=maxabs initialize.type=linear\n```\nAnd this one was a bit better. Heated my office for a while. Performance did not quit get to 50% which means it could be only the predictions one layer deep are accurate and it ignores everything else.\n```\npython examples/high_order_interpolation.py data.type=sequence net=dual_convolution max_epochs=100 optimizer.lr=1e-5 batch_size=32 net.layer_type=continuous data.repeats=1 net.n=3 net.segments=16 data.max_features=10 optimizer.patience=20  net.embedding_dimension=128 net.hidden_width=1024 net.normalize=maxabs initialize.type=linear\n```\n## Notes\nI use input layer (continuous or discontinuous) with 128 segments, one for each ASCII character.  You can bump this down to 64, but the convergence doesn't seem quite as good - presumably it still works because most books don't use all the ascii characters anyway.\n\n## Apply a model using sequence model\n```\npython examples/high_order_interpolation.py train=False checkpoint=\\\"outputs/2022-06-15/16-13-08/lightning_logs/version_0/checkpoints/epoch=32-step=104577.ckpt\\\" topk=2 num_predict=1000 prompts=[\"Who are you?\"]\n```\nexample output (model trained to predict the next character given the preceeding 16) using a single hidden layer. The prompt is not in the\ndataset, however, the data eventually evolves into something that is in the dataset.\n```\nprompt: Who are you? \nresult: Who are you? I the set it wall night, and the whippoorwills in the glen, Selina Frye tottered to the telephone and spread what news she could of the second phase of the horror.  The next day all the countryside. Trees, grass, and underbrush were whipped into a fury; and the frightened crowd at the mountain\\'s base huddled still closer, and winced as if in expectation of a blow.  \"_Ygnaiih ... ygnaiih ... thflthkh\\'ngha ... Yog-Sothoth...._\" They trailed off into nothingness as the whippoorwills in the glen, Selina Frye tottered to the telephone and spread what news she could of the second phase of the horror.  The next day all the countryside. Trees, grass, and underbrush were whipped into a fury; and the frightened crowd at the mountain\\'s base huddled still closer, and winced as if in expectation of a blow.  \"_Ygnaiih ... ygnaiih ... thflthkh\\'ngha ... Yog-Sothoth...._\" They trailed off into nothingness as the whippoorwills in the glen, Selina Frye tottered to the telephone and spread what news she\n```\n# Using lightgbm\nEarly experiments with lightgbm.\n\nTraining\n```\npython examples/language_interpolation_gbm.py \n```\nUsing\n```\npython examples/language_interpolation_gbm.py train=false checkpoint=outputs/2022-07-17/08-16-29/model.txt text=\"What did they say?\"\n```\nsample output\n```\nprompt: What did they say? response: y.oon. These,ins onchased,by,sesionic subelaborishor, wondse trentel opowed my fould a midd, attrow \n```\n\n\n# Other Stuff\n## Run with centered model (language cellular automaton)\n```\npython examples/language_cellular_automata.py net.features=11 data.type=centered\n```\n\n## As cellular automaton\nA centered model can be repeatedly applied to the same text as a moving window and will work with arbitrary length sentences.  This\napproach is similar to stencils used in solving partial differential equations.\n```\npython examples/language_cellular_automata.py net.features=11 data.type=centered train=False checkpoint=\\\"outputs/2021-08-21/20-14-40/lightning_logs/version_0/checkpoints/epoch=20-step=35909.ckpt\\\" topk=2 num_predict=200 text=\"The monster awakes\" topk=1 data.reapply=10\n```\n\n# Interesting papers related to sparse mlps for language\n\n[Efficient Language Modeling with Sparse all-MLP](https://arxiv.org/pdf/2203.06850.pdf)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjloveric%2Flanguage-interpolation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjloveric%2Flanguage-interpolation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjloveric%2Flanguage-interpolation/lists"}