{"id":43698,"url":"https://github.com/koayon/awesome-adaptive-computation","name":"awesome-adaptive-computation","description":"A curated reading list of research in Adaptive Computation, Inference-Time Computation \u0026 Mixture of Experts (MoE).","projects_count":173,"last_synced_at":"2026-10-19T11:00:22.485Z","repository":{"id":188912117,"uuid":"679673318","full_name":"koayon/awesome-adaptive-computation","owner":"koayon","description":"A curated reading list of research in Adaptive Computation, Inference-Time Computation \u0026 Mixture of Experts (MoE).","archived":false,"fork":false,"pushed_at":"2025-01-01T13:49:09.000Z","size":339,"stargazers_count":165,"open_issues_count":1,"forks_count":9,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-09-29T22:18:12.386Z","etag":null,"topics":["adaptive-computation","computer-vision","machine-learning","mixture-of-experts","nlp","pytorch","tensorflow","transformers"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/koayon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-08-17T11:11:32.000Z","updated_at":"2026-09-02T11:17:25.000Z","dependencies_parsed_at":"2024-01-16T23:27:23.090Z","dependency_job_id":"65a5a297-41ce-4a32-a253-ffc03f341b60","html_url":"https://github.com/koayon/awesome-adaptive-computation","commit_stats":null,"previous_names":["koayon/awesome-adaptive-computation"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/koayon/awesome-adaptive-computation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koayon%2Fawesome-adaptive-computation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koayon%2Fawesome-adaptive-computation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koayon%2Fawesome-adaptive-computation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koayon%2Fawesome-adaptive-computation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/koayon","download_url":"https://codeload.github.com/koayon/awesome-adaptive-computation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koayon%2Fawesome-adaptive-computation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":344872684,"owners_count":38355170,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-10-03T21:59:58.778Z","status":"online","status_checked_at":"2026-10-19T02:00:06.414Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2024-01-13T21:18:43.566Z","updated_at":"2026-10-19T11:00:22.485Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["Tools \u0026 Agents","Early Exit: End-to-End Adaptive Computation","Mixture of Experts (Sparse MoE)","Continual Learning","Adaptive Computation for Black-box models","Games","About","Scaling Laws","Other Modular Architectures","Pre-cursors to Adaptive Computation","Open Source Libraries","AI Safety","Other","More Compute Per Output Token"],"sub_categories":[],"readme":"# Awesome Adaptive Computation\n\n[![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\n[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\nAwesome Adaptive Computation is a curated list of Adaptive Computation papers,\nmodels, explainers and libraries for Machine Learning.\n\n## Contents\n\n- [Contents](#contents)\n- [About](#about)\n- [Mixture of Experts (Sparse MoE)](#mixture-of-experts-sparse-moe)\n- [Other Modular Architectures](#other-modular-architectures)\n- [Early Exit: End-to-End Adaptive Computation](#early-exit-end-to-end-adaptive-computation)\n- [More Compute Per Output Token](#more-compute-per-output-token)\n- [Adaptive Computation for Black-box models](#adaptive-computation-for-black-box-models)\n- [Continual Learning](#continual-learning)\n- [Tools \\\u0026 Agents](#tools--agents)\n- [Games](#games)\n- [Pre-cursors to Adaptive Computation](#pre-cursors-to-adaptive-computation)\n- [Open Source Libraries](#open-source-libraries)\n- [AI Safety](#ai-safety)\n- [Scaling Laws](#scaling-laws)\n- [Other](#other)\n\n## About\n\n`Adaptive Computation` (sometimes called `Dynamic Compute`) is the ability of a machine learning system to adjust its\n`function` and `compute budget` for each example.\n\n\u003c!-- We can think of this as giving models a [System 2](https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow) mode. --\u003e\n\nAdaptive Computation techniques include **Mixture of Experts** (decoupling model\ncapacity and model compute), **Early Exiting** (saving compute on easy\ninputs) and **Inference-Time Computation** (search and verification at inference time) as well as sampling techniques.\n\n\u003c!-- [The Bitter Lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) states that the two general methods to utilise large amounts of compute are `Learning` and `Search`. Whilst large pre-trained models focus on `learning` at _train time_, Adaptive Computation is about spending more compute at inference time with mechanisms similar to `Search`. --\u003e\n\n---\n\nIn this repo, links are organised by topic and have explanations so you can\ndecide what you would like to read. Especially recommended links are starred 🌟\n\nStar this repository to see the latest developments in this research field.\n\nWe accept contributions! We strongly encourage researchers \u0026 practitioners to\nmake pull requests with papers, approaches and explanations that they feel\nothers in the community would benefit from 🤗\n\n\u003c!-- Ordered by topic, then date published --\u003e\n\n## Mixture of Experts (Sparse MoE)\n\nThe Mixture of Experts (MoE) paradigm uses a routing layer to choose a limited number of parameters\nto apply to a given input rather than using all the available parameters.\n\nThis conditional computation allows us model capacity to increase without also\nscaling the compute required for each forward pass. This is useful because\nbigger models are more sample efficient and more compute efficient to train.\n\nMoE models are also useful for compartmentalising knowledge and avoiding\nnegative interference from irrelevant computation.\n[Mixtral-8x7B](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) is an\nopen-weights MoE model which is comparable to much larger models. Google DeepMind similarly show that their [Gemini 1.5 Pro](https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf) based on an MoE architecture is competitive with their much larger Gemini 1 Ultra. Databricks/Mosaic [DBRX](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm) is another powerful MoE model and it seems that MoE is now the go-to architecture for large models.\n\n[JetMoE](https://research.myshell.ai/jetmoe), based on the [ModuleFormer](https://arxiv.org/pdf/2306.04640.pdf) MoE conception, shows that MoEs can also be effective at smaller scales.\n\n**D2DMoE: Dense to Dynamic-k Mixture-of-Experts Conversion**, Szatkowski et al. (2024), [pdf](https://arxiv.org/pdf/2310.04361) [code](https://github.com/bartwojcik/D2DMoE)\n\n\u003e While MoE models are mostly used for scaling up the parameter count, recently [MoEfication](https://arxiv.org/abs/2110.01786) has shown that static dense models can be converted to MoEs to improve execution time. D2DMoE makes further progress in improving the efficiency of these dense-to-MoE converted models by: 1) showing that the efficiency of the resulting model can be significantly enhanced by enforcement of activation sparsity in the base model; 2) proposing Expert Contribution Routing, a novel objective for the training of the gating networks, which are now tasked to predict the output norm of each expert for the given input, enabling approximation of each expert’s relative contribution; 3) introducing dynamic-k gating, which allows the model to appropriately distribute its computational budget between easy and hard inputs; 4) extending the proposed conversion scheme to any linear layers such as multi-head attention projections. \n\n**Skywork-MoE, Skywork (2024)**\n[pdf](https://github.com/SkyworkAI/Skywork-MoE/blob/main/skywork-moe-tech-report.pdf)\n\n\u003e An open-source MoE in the style of Switch Transformer. They detail two training tricks for getting better MoE performance. Firstly they normalise the routing logits before it goes through the softmax in order to reduce the entropy in the router and make the router more decisive. Secondly they have a different auxiliary loss coefficient for each layer and this is tuned during training depending on how many tokens were dropped at that layer. This helps to reduce impact of the auxiliary loss as the router becomes more balanced and confident.\n\n**DynMoE, CUHK: Guo et al (2024)**\n[pdf](https://arxiv.org/pdf/2405.14297)\n[code](https://arxiv.org/pdf/2405.14297)\n\n\u003e One problem with the token choice MoE approach is that every token is allocated the same number of experts. Ideally for true adaptive computation this would be variable depending on how difficult the token is. The authors introduce a routing mechanism which allows for a variable number of experts per token as well as a procedure for dynamically changing the number of experts during training. This allows for better performance with less hyperparameter sweeping.\n\u003e\n\u003e Also see [Dynamic Routing in MoEs](https://arxiv.org/pdf/2403.07652)\n\n**DeepSeek-v2, DeepSeek (2024)**\n[pdf](https://arxiv.org/pdf/2405.04434)\n\n\u003e A large (236B), performant and open MoE with lots of details about training and checkpoints available. Useful for understanding a modern MoE recipe.\n\n**MoEs for Deep-RL, Google DeepMind: Obando-Ceron et al (2024)**\n[pdf](https://arxiv.org/pdf/2402.08609.pdf)\n\n\u003e The authors show that MoEs can be used to improve the sample efficiency of popular RL systems such as DQN and Rainbow. The authors show that using MoEs (in particular the SoftMoE variant) improves the ultimate performance of the RL systems. Previously, scaling up the underlying models in RL systems was often wasteful in parameters, but the authors show using MoEs they can get predictable performance improvements with scale. This suggests that scaling laws for Deep RL systems could be possible.\n\n**MoE Design Choices, EPFL: Fan et al (2024)**\n[pdf](https://arxiv.org/pdf/2402.13089.pdf)\n\n\u003e The authors ablate some decision decisions for MoEs and shows the benefits of compared to vanilla transformers. They unfortunately only study very small models and so similar analysis for larger models could likely be useful for the community.\n\n🌟 **Routers in Vision MoEs, DeepMind: Liu et al (2024)**\n[pdf](https://arxiv.org/pdf/2401.15969.pdf)\n\n\u003e Compares the performance of different routing mechanisms in MoEs trained on Vision tasks. They show that Language Model routers can adapt well to Vision and that for Vision tasks (where the task isn't autoregressive), Soft MoE outperforms. They also reframe some previous routing methods mathematically to more clearly detail the differences. Worth a read for anyone deciding which MoE approach to choose for their application.\n\n**MoE-LLaVA, Peking University: Lin et al (2024)**\n[pdf](https://arxiv.org/pdf/2401.15947.pdf)\n[code](https://github.com/PKU-YuanGroup/MoE-LLaVA)\n\n\u003e Whilst MoEs have had much success in ViTs and LLMs, the authors also show that they can be effective in LVLMs (Large Vision Language Models). By exploiting the sparsity and increased parameter count of MoEs whilst maintaining FLOPs, we get the expected boost in both performance and hallucination avoidance.\n\n**Mixtral of Experts, Mixtral (2024)**\n[pdf](https://arxiv.org/pdf/2401.04088.pdf)\n[official code](https://github.com/mistralai/mistral-src)\n\n\u003e The paper describing Mixtral's State of Art LLM based on the MoE paradigm.\n\n**BlackMamba (MoE-Mamba), Zyphra: Anthony et al (2024)**\n[pdf](https://arxiv.org/pdf/2402.01771.pdf)\n[code](https://github.com/Zyphra/BlackMamba)\n[models](https://huggingface.co/Zyphra)\n\n\u003e The authors combine the MoE paradigm with a recent SSM-based architecture\n\u003e Mamba. Mamba provides Transformer-like performance and scaling properties\n\u003e whilst using a sub-quadratic attention variant allowing for much larger\n\u003e sequence lengths. Here we see that this architecture can additionally be\n\u003e combined with MoEs to increase performance, similarly to MoEs for transformers\n\u003e or RNNs previously. For a general explainer on Mamba see [here](https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html).\n\n🌟 **Offloading for Fast MoE Inference, Moscow: Eliseev \u0026 Mazur (2023)**\n[pdf](https://arxiv.org/pdf/2312.17238.pdf)\n[official code](https://github.com/dvmazur/mixtral-offloading)\n\n\u003e Despite some open-weights MoEs being available, they are not the most popular\n\u003e models used for inference due to the large memory footprint required at\n\u003e inference time. To address this the authors propose architecture-aware\n\u003e quantisation, an LRU cache for experts (to exploit the fact that experts are\n\u003e more likely than chance to repeat for two adjacent tokens) and a speculative\n\u003e expert loading algorithm. Since the inputs (x) to each layer only differ\n\u003e iteratively by a small amount (due to the residual stream carrying information\n\u003e from layer to layer), they note that by applying the routing function for the\n\u003e subsequent layer at the current layer, you can get a good guess for which\n\u003e experts to load. The upshot of this is that MoE models, like Mixtral, can be\n\u003e run on consumer grade hardware with much increased generation speed. This is a\n\u003e huge win for inference efficiency in the memory constrained and single batch\n\u003e regime. [MoE-Infinity](https://arxiv.org/pdf/2401.14361.pdf) is a similar offloading paradigm with code [here](https://github.com/TorchMoE/MoE-Infinity)\n\n🌟 **QMoE, ISTA: Frantar \u0026 Alistarh (2023)**\n[pdf](https://arxiv.org/pdf/2310.16795.pdf)\n[code](https://github.com/IST-DASLab/qmoe)\n\n\u003e Generally MoEs require larger footprints but fewer FLOPs compared to a dense\n\u003e model of which achieves similar performance. In the quest to reduce the memory\n\u003e footprint, we might seek to perform quantisation. The authors present a\n\u003e compression method which takes advantage of the inherent sparsity to compress\n\u003e the model at a 20x compression rate whilst retaining most performance. This\n\u003e compresses each fp16 weight to the equivalent of less than one bit. For the\n\u003e first time it's possible to run a trillion parameter model on consumer\n\u003e hardware.\n\n**SparseMixer - Sparse Backpropagation for MoE Training, Microsoft: Liu et al\n(2023)**\n[pdf](https://arxiv.org/pdf/2310.00811.pdf)\n\n\u003e One of the most important parts of an MoE is the router which allows the\n\u003e experts to specialise well. Unfortunately, typical MoE training gives\n\u003e suboptimal routers as suggested by Hash routers performing almost as well as\n\u003e more principled routing mechanisms. This paper suggests the reason is due to\n\u003e MoE training ignoring parts of the gradient and suggests a midpoint-rule based\n\u003e gradient approximation which substantially improves training.\n\n**MoV/MoLoRA, Cohere For AI: Zadouri et al (2023)**\n[pdf](https://arxiv.org/pdf/2309.05444.pdf),\n[official Jax code](https://github.com/for-ai/parameter-efficient-moe)\n\n\u003e Introduces parameter efficient MoE models where instead of routing between\n\u003e entire FFN layers, we route between adapters such as LoRAs or $(IA)^3$ with\n\u003e the same base model. This allows for much of the benefits of the (Soft) MoE\n\u003e paradigm but without the huge memory footprint (particularly compared to\n\u003e previous upscaling methods). The\n\u003e [HydraMoE](https://github.com/SkunkworksAI/hydra-moe) project also take a\n\u003e similar approach.\n\n**Soft Merging of Experts (SMEAR), UNC: Zadouri et al (2023)**\n[pdf](https://arxiv.org/pdf/2306.03745.pdf)\n\n\u003e Takes the opposite approach to Soft-MoE and averages the Expert weights rather\n\u003e than the tokens. This is interesting given that model merging approaches which\n\u003e show that linear combinations of models can perform well on tasks that either\n\u003e model was trained for. Note here the averaging operation can become\n\u003e prohibitively expensive if we different experts for each token (similar FLOPs\n\u003e to forward passes on all experts and ensembling). Hence the method relies on a\n\u003e Task-MoE approach of picking the same expert configuration on a per-example\n\u003e rather than a per-token basis.\n\n**AutoMoE, UBC/Microsoft: Jawahar et al (2023)**\n[pdf](https://arxiv.org/pdf/2210.07535.pdf),\n[official PyTorch code](https://github.com/microsoft/AutoMoE)\n\n\u003e One of the promises of MoE is being able to apply different amounts of compute\n\u003e to each token. Generally, this has been achieved by different tokens being\n\u003e processed and dropped by different numbers of experts per layer. AutoMoE also\n\u003e uses differently sized experts to achieve more heterogeneity. They perform an\n\u003e Architectural search for optimal architectures given computational\n\u003e constraints.\n\n🌟 **Expert Choice MoEs, Google: Zhou et al (2022)**\n[pdf](https://arxiv.org/pdf/2202.09368.pdf),\n[blog](https://ai.googleblog.com/2022/11/mixture-of-experts-with-expert-choice.html),\n[PyTorch code](https://github.com/koayon/ml-replications/blob/main/mixture_of_experts/expert_choice_layer.py)\n\n\u003e Introduces a principled, truly compute-adaptive MoE model. In traditional MoE\n\u003e models the tokens select the top experts that they would most like to be\n\u003e processed by. In Expert Choice routing however, the experts choose the top\n\u003e tokens that they would like to process. Hence multiple experts can pick the\n\u003e same token and give it lots of compute, and similarly all experts can ignore a\n\u003e token so it is skipped for that layer. As well as improving training\n\u003e efficiency, this approach also has the benefits that it helps with load\n\u003e balancing and eliminates the need for auxiliary loss functions.\n\n🌟 **Task Level MoEs, Various (2022)**\n[DeMix pdf](https://arxiv.org/pdf/2108.05036.pdf),\n[Task-MoE pdf](https://arxiv.org/pdf/2110.03742.pdf)\n[c-BTM code](https://github.com/kernelmachine/cbtm)\n\n\u003e Instead of routing each token separately these approaches use the same Expert\n\u003e for entire documents based on the task (which is supplied to the network).\n\u003e Instead of learning the routing, we supply the routing based on what we know\n\u003e about the tasks, inducing our own inductive bias. Also note that this offers\n\u003e memory footprint benefits at inference time - if inference is for a limited\n\u003e set of tasks, we only need these enough GPU memory for these experts.\n\u003e [ELMForest - Branch, Train, Merge (BTM)](https://arxiv.org/pdf/2208.03306.pdf%7D),\n\u003e [c-BTM](https://arxiv.org/pdf/2303.14177.pdf) and\n\u003e [Branch, Train, Mix (BTX)](https://arxiv.org/pdf/2403.07816.pdf)\n\u003e are follow-ups which use\n\u003e ensembling approaches from multiple LMs trained independently in a continual\n\u003e learning approach.\n\n\u003c!-- There are possibly additional benefits to combining task and token level experts. We could input a task and use this to decide which routers that we want to use in our MoE layers see [Multi-gate](https://dl.acm.org/doi/pdf/10.1145/3219819.3220007).\nAlternatively we could concatenate task information to the input of the router network so it can use it if it wants.\nIt's not clear whether having inductive bias would be better.  --\u003e\n\n**No Language Left Behind, Meta (2022)**\n[pdf](https://arxiv.org/abs/2207.04672),\n[official PyTorch code](https://github.com/facebookresearch/fairseq/tree/nllb)\n\n\u003e Translation is a natural setting for MoEs - some but not all of the parameters\n\u003e for English to Chinese translation be relevant in English to French\n\u003e translation as well. But using all of the English to Chinese knowledge might\n\u003e confuse the model. MoE therefore has useful inductive biases to allow this\n\u003e model to use only the relevant parts. Here the researchers show that the MoE\n\u003e approach scales even for extremely low-resource languages. Translation may be\n\u003e a natural environment for task/document-level rather than token-level routing.\n\n**Hash Routing, Meta: Roller et al (2021)**\n[pdf](https://browse.arxiv.org/pdf/2106.04426.pdf)\n\n\u003e Uses a static rather than fixed routing per input token and shows similar\n\u003e results to more principled routing methods in some regimes. Suggests that\n\u003e previous routing methods may be somewhat under-optimised.\n\n**Switch Transformers, Google: Fedus et al (2021)**\n[pdf](https://arxiv.org/pdf/2101.03961.pdf),\n[review paper](https://arxiv.org/pdf/2209.01667.pdf),\n[PyTorch code](https://nn.labml.ai/transformers/switch/index.html),\n[model](https://huggingface.co/docs/transformers/model_doc/switch_transformers)\n\n\u003e Simplifies the MoE routing algorithm with top-1 routing. Shows that we can\n\u003e exploit the scaling laws with parameters as well as simply compute and\n\u003e develops distributed systems approach to MoE\n\n**WideNet (Go Wider Instead of Deeper), NUS: Xue et al (2021)**\n[pdf](https://arxiv.org/pdf/2107.11817.pdf)\n\n\u003e Suggests a parameter sharing approach using a single layer of multiple MoEs\n\u003e repeated multiple times as transformer blocks (similar to the Universal\n\u003e Transformer but with MoEs). This results in a deep model which has\n\u003e O(expert_num) instead of O(layer_depth) parameters. They achieve SoTA results\n\u003e with fewer parameters than previous models. More recently, Apple's\n\u003e [One Wide Feedforward paper](https://arxiv.org/pdf/2309.01826.pdf) details the\n\u003e amount of redundancy across layers. This suggests this approach is\n\u003e increasingly fruitful for on-device models.\n\n🌟 **Outrageously Large Neural Networks (aka The Sparse MoE Layer), Google:\nShazeer et al (2017)** [pdf](https://arxiv.org/pdf/1701.06538.pdf)\n\n\u003e Introduces Mixture of Expert models in their modern form using Sparsely Gated\n\u003e MoE layer and a trainable gating network. They use RNNs as this is pre\n\u003e \"Transformers Eating The World\".\n\n\u003c!-- A Noam Shazeer, Geoff Hinton collab - two true legends of Deep Learning --\u003e\n\n## Other Modular Architectures\n\n**Stylus Diffusion Adapter Selection, UC Berkeley: Luo et al (2024)**\n[pdf](https://arxiv.org/pdf/2404.18928),\n[repo](https://github.com/stylus-diffusion/stylus)\n\n\u003e Diffusion model users often use adapters rather than full finetunes to achieve models which perform well on a particular style. The authors here automatically select and compose relevant adapters for the prompt using a model routing approach. A nice application of inference-time routing which we might expect to become more commonplace in the future.\n\n**MoDE - CLIP Data Experts via Clustering, Meta: Ma et al (2024)**\n[pdf](https://arxiv.org/pdf/2404.16030),\n[pytorch code](https://github.com/facebookresearch/MetaCLIP/tree/main/mode)\n\n\u003e The authors apply a Task-specific MoE based on training parallel CLIP models on restricted domains and ensembling these together. They show increased performance with less compute and note the ability to add in new \"experts\" for new domains asynchronously and after initial training as a Continual Learning play.\n\n🌟 **Mixture of Depths, DeepMind: Raposo et al (2023)**\n[pdf](https://arxiv.org/pdf/2404.02258.pdf),\n\n\u003e The traditional Early-Exit formulation allows \"easier\" tokens to not go\n\u003e through the whole network, reducing compute but it has a couple of problems.\n\u003e Firstly, it's not clear that the layers that you want to skip are necessarily\n\u003e the final ones (perhaps an easy token should skip some middle layers instead)\n\u003e and secondly, Early Exit might not be in practise that much faster on GPUs due\n\u003e to the variable compute graph. In an attempt to align Early-Exit work with the\n\u003e Hardware Lottery, the authors suggest enforcing a compute budget but fixing the\n\u003e computational graph allowing dynamic allocation of FLOPs across tokens in the\n\u003e sequence, optimising the allocation along the sequence for different layers\n\u003e across the model depth. Because they opt for an expert-choice routing mechanism,\n\u003e they also introduce novel sampling methods to ensure validity for autoregressive\n\u003e generation which seem broadly applicable to other MoE models.\n\n\u003e This builds on the [LayerDrop](https://arxiv.org/pdf/1909.11556.pdf) approach to structured dropout.\n\n**Fast FeedForward (FFF), ETH Zurich: Belcak et al (2023)**\n[pdf1](https://arxiv.org/pdf/2308.14711.pdf),\n[pdf2](https://arxiv.org/pdf/2311.10770.pdf),\n[official pytorch code1](https://github.com/pbelcak/fastfeedforward),\n[official pytorch code2](https://github.com/pbelcak/UltraFastBERT),\n[pytorch code](https://github.com/sap-ient-ai/FFF)\n\n\u003e Instead of the usual FeedForward Network, the authors propose a balanced tree\n\u003e structure where depending on your path through the tree, a different function\n\u003e is applied to the input. Inputs go either left or right through the tree\n\u003e depending on the result of a dot product with a learned discriminating vector.\n\u003e This approach results in encouraging performance with somewhat limited\n\u003e inference FLOPs but high training FLOPs and unstructured inference sparsity\n\u003e which must be applied sequentially which falls foul of the\n\u003e [Hardware Lottery](https://arxiv.org/pdf/2009.06489.pdf) and doesn't\n\u003e parallelise nicely on GPUs.\n\n🌟 **Soft-MoE, Google DeepMind: Puigcerver et al (2023)**\n[pdf](https://arxiv.org/pdf/2308.00951.pdf),\n[pytorch code](https://github.com/fkodom/soft-mixture-of-experts/blob/main/soft_mixture_of_experts/soft_moe.py)\n\n\u003e Instead of Sparse MoE models, they allow each expert to using its router to\n\u003e select weights for a weighted average of input tokens that it wants to\n\u003e process. They show SoTA results on Image recognition tasks. Note since the\n\u003e approach relies on Expert Choice, it doesn't yet generalise to autoregressive\n\u003e generation.\n\n## Early Exit: End-to-End Adaptive Computation\n\nEarly Exit approaches ask if we get the output of a neural network without going\nthrough all the layers, particularly if faced with an easier example. This is\ntypically done by learning an exit probability at each layer.\n\n**EE-LLM, Alibaba: Chen et al (2024)**\n[pdf](https://arxiv.org/pdf/2312.04916)\n[pytorch code](https://github.com/pan-x-c/EE-LLM)\n\n\u003e The authors extend Megatron into a library which natively supports the Early Exit paradigm taking full advantage of 3D parallelism. Other contributions include methods to efficiently facilitate backprop even when some layers may be unused and methods to handle the fact that using naive early exit would result in missing KV-caches for some tokens. They find that limiting the Early Exit layers to a few intermediate layers substantially improves performance.\n\n**Sparse Universal Transformer (SUT), MILA: Tan et al (2023)**\n[pdf](https://arxiv.org/pdf/2310.07096.pdf)\n\n\u003e Combines the Universal Transformer approach (RNN with transformer blocks) with\n\u003e the Mixture of Experts paradigm (multiple experts instead of a single FFN\n\u003e layer). They also use a new stick-breaking-based dynamic halting mechanism.\n\u003e This brings all the benefits of Sparse MoEs (such as less inference compute\n\u003e whilst having a lot of parameters) and the benefits of Universal Transformer\n\u003e (parameter efficiency, Turing-completeness and generalization ability)\n\u003e together.\n\n**AdaTape, Google: Xue et al (2023)**\n[pdf](https://arxiv.org/pdf/2301.13195.pdf),\n[blog](https://ai.googleblog.com/2023/08/adatape-foundation-model-with-adaptive.html),\n[official jax code](https://github.com/google-research/scenic/blob/main/scenic/projects/adatape/adatape_vit/adatape_vit.py)\n\n\u003e Extends the ACT method by giving the model a \"tape\" which contains some\n\u003e additional inputs which may be useful for encoding. For each token the model\n\u003e may append a variable number of tape tokens to the input, which allows it to\n\u003e regulate how much additional compute we add. The paper shows impressive\n\u003e performs on image classification tasks and the 'parity' task on long\n\u003e sequences.\n\n**Dataset Pruning Using Early Exit Networks, Görmez et al (2023)**\n[pdf](https://openreview.net/pdf?id=Kh114370zL)\n\n\u003e Early Exit Networks naturally learn which input examples are \"easy\" (can be\n\u003e exited early) or \"difficult\" (require all the layers of the network). The\n\u003e authors use this property to prune datasets to use for training and\n\u003e finetuning. The algorithm EEPrune achieves SOTA performance for dataset\n\u003e pruning in some regimes.\n\n🌟 **PonderNet, DeepMind: Banino et al (2021)**\n[pdf](https://arxiv.org/pdf/2107.05407.pdf),\n[PyTorch code](https://github.com/koayon/ml-replications/tree/main/ponder)\n\n\u003e Allows the model to exit after each transformer layer if it's confident in the\n\u003e answer. It introduces a stable probabilistic policy for halting which provides\n\u003e low-variance unbiased gradient updates. This can also be combined with the\n\u003e [SkipNet](\u003c[pdf](https://arxiv.org/pdf/1711.09485)\u003e) paradigm where we instead\n\u003e of exiting directly, skip to the final few layers to allow our universal\n\u003e computation (applied to all inputs) to be at the end as well as the start of\n\u003e the network.\n\n**PaBEE, DeepMind: Zhou et al (2020)**\n[pdf](https://arxiv.org/pdf/2006.04152.pdf),\n[official PyTorch code](https://github.com/huggingface/transformers/tree/main/examples/research_projects/bert-loses-patience)\n\n\u003e Introduces Patient Early Stopping. Whilst ACT has a learned exit probability,\n\u003e PABEE instead looks at the output class if it _were_ to exit. We exit if the\n\u003e intermediate outputs are the same over multiple layers. Interestingly they\n\u003e suggest that the reason for this isn't just speed; they suggest that early\n\u003e stopping will _improve_ performance due to lower risk of \"overthinking\"\n\u003e (analogously to stopping training earlier to prevent overfitting).\n\u003e [F-PaBEE](https://arxiv.org/pdf/2305.11916.pdf) prevents a slightly more\n\u003e flexible approach based on similarity scores.\n\n**Universal Transformer, Google: Dehghani et al (2019)**\n[pdf](https://arxiv.org/pdf/1807.03819.pdf)\n\n\u003e Reuses the transformer block recurrently across multiple layers with an\n\u003e ACT-like halting mechanism. RNNs can be better than transformers at length\n\u003e extrapolation but here we get the best of the Transformer (training\n\u003e parallelizability) and the best of the RNN (recurrent inductive bias).\n\u003e Universal Transformers can also be shown to be Turing-complete.\n\n**Adaptive Computation Time (ACT) for RNNs, Google: Graves (2016)**\n[pdf](https://arxiv.org/pdf/1603.08983.pdf)\n\n\u003e Introduces the ACT approach for models to learn how many computational steps\n\u003e they should take before returning an output. This approach is built on and\n\u003e refined in many later papers such as PonderNet.\n\n## More Compute Per Output Token\n\n**Masked Diffusion Language Models, Cornell: Sahoo et al (2024)**\n[pdf](https://arxiv.org/pdf/2406.07524),\n[pytorch code](https://github.com/kuleshov-group/mdlm)\n[video](https://www.youtube.com/watch?v=WjAUX23vgfg)\n\n\u003e Autoregressive models sample a single token at a time, regardless of how difficult this token is to predict. Diffusion models however have the benefit that the number of steps from the noised input to the final output can be varied which acts as a knob controlling the amount of compute applied. The authors introduce a simplified method for Diffusion Language models based on BERT which achieves better perplexity than previous Diffusion Language models (though not at autoregressive model levels). This avenue provides a different approach to varying compute per output token.\n\n🌟 **Quiet-STaR, Stanford: Zelikman et al (2024)**\n[pdf](https://arxiv.org/pdf/2403.09629.pdf),\n\n\u003e One of the core motivations of Adaptive Computation is noting that for\n\u003e difficult tokens we should spend more compute. There have been prompting\n\u003e ways to do this (e.g. Chain of Thought) and recurrent ways to do this (e.g.\n\u003e Universal Transformers) but ideally we'd want the LLM to just start writing\n\u003e down more tokens, using all faculties on difficult tokens, without being told\n\u003e when to apply this technique and in a natural next-token prediction way that\n\u003e takes advantage of its pretraining. Quiet-STaR is exactly that. The model\n\u003e generates hidden `rationale` tokens which it can use to reason but don't get\n\u003e shown to the user (or loss function). This generalises the previous `STaR` work\n\u003e by the same authors and the `Pause Token` results in a way that is much more\n\u003e generally effective.\n\n\u003e This is the real deal folks! I'm extremely excited about this approach.\n\u003e And it's a cracked team too, they start the paper with a quote from\n\u003e Danish philosopher Søren Kierkegaard. An excellent formulation and one of\n\u003e those papers that makes you realise why you got into this field.\n\n## Adaptive Computation for Black-box models\n\nFor black box pre-trained models, perhaps those behind an API, there are some\ntechniques for using Adaptive Computation. These are promising techniques for\nthose with limited compute budgets.\n\nPrompting techniques such as\n[Reflexion](https://browse.arxiv.org/pdf/2303.11366.pdf),\n[Debate](https://browse.arxiv.org/pdf/2305.14325.pdf),\n[Chain of Thought](https://browse.arxiv.org/pdf/2201.11903.pdf),\n[Tree of Thought](https://browse.arxiv.org/pdf/2305.10601.pdf) and\n[Chain of Verification](https://browse.arxiv.org/pdf/2309.11495.pdf) can also be\nused to improve performance for Black-box models.\n\n**Online Speculative Decoding, Berkeley: Liu et al (2024)**\n[pdf](https://arxiv.org/pdf/2310.07177),\n[pytorch code](https://github.com/LiuXiaoxuanPKU/OSD)\n\n\u003e The authors propose an Active Learning approach to choosing draft models for speculative decoding. In downtime when the GPUs are not maxed out for inference, they use the capacity to instead finetune a draft model on the recent outputs from the large teacher model. In this way the system can respond to distribution shift and still stay performant by accepting more tokens from the draft model. Depending on the distribution shift this can result in a latency reductions going from 1.22x (naive speculative decoding under distribution shift) to 3.06x (their method).\n\n🌟 **Scaling LLM Test-Time Compute, Deepmind: Snell et al (2024)**\n[pdf](https://arxiv.org/pdf/2408.03314),\n\n\u003e They test two strategies for using test-time compute: (1) searching against dense, process-based verifier reward models in a tree-like fashion and (2) utilising Dynamic Evaluation-style updating of the model’s distribution at test time given the prompt. They find that using these strategies they're able to achieve a 4x improvement over using best-of-N with the same compute budget. Even more strikingly, they find a 14x improvement over FLOP-matching with using a larger model. This is a huge win for Adaptive Compute-style approaches. On a different note the paper's motivating setup and styling are well executed which makes the paper a nice read.\n\n🌟 **Jacobi Consistency Large Language Models (CLLMs), SJTU: Kou et al (2024)**\n[pdf](https://arxiv.org/pdf/2403.00835),\n[blog](https://hao-ai-lab.github.io/blogs/cllm/)\n\n\u003e An interesting approach to the multi-token prediction problem. They give a language model a prompt\n\u003e and a random k-tokens to come next. They then run a forward pass: this will give the \"correct\"\n\u003e next token (i.e. directly after the prompt) but there's some chance it also updates one of the following\n\u003e tokens to being correct as well. They use a consistency loss (similar to in Diffusion models) to improve the trajectory\n\u003e from [k random tokens] --\u003e [k correct tokens] very similar to a diffusion approach. We notes that this can take at most k\n\u003e forward passes but ideally can be done fewer forward passes.\n\n\u003e The aim of the game is to repeatedly put the k tokens through the model until they reach a fixed point (i.e. a forward pass doesn't change them). (Also note that on this final forward pass we also have the k+1th token returned). This is an interesting approach that changes fundamentally the language modelling objective from autoregressive prediction to predicting trajectories for multi-token sequences.\n\n🌟 **Multi-Token Prediction, Meta: Gloeckle et al (2024)**\n[pdf](https://arxiv.org/pdf/2404.19737)\n\n\u003e Traditionally LLMs predict one token at a time. This is somewhat inhuman and\n\u003e inefficient because often once the start of a word/phrase is predicted, the end\n\u003e is trivial. The authors here treat subsequent token prediction as an auxiliary task\n\u003e and train additional heads to predict further tokens. The real benefit of this approach\n\u003e isn't in inference though but in training. Pre-training with this auxiliary task is more\n\u003e sample efficient and forces the LM to learn better representations for medium-term dependencies.\n\u003e In generative tasks such as coding, these models much outperform traditional LLMs with lower latency.\n\n**Many-Shot In-Context Learning, Google DeepMind: Agarwal et al (2024)**\n[pdf](https://arxiv.org/pdf/2404.11018.pdf)\n\n\u003e It has been long observed that language models can learn how to do a new\n\u003e task from examples of inputs, reasoning chains and outputs. This is known\n\u003e as few-shot Chain of Thought (CoT).\n\u003e Historically, the number of examples has been limited by the context window\n\u003e though. In this work, the authors suggest that using hundreds or thousands of\n\u003e examples (typically model generated) can aid performance and out-of-distribution\n\u003e robustness via the In-Context Learning mechanism. In other words, they formalise\n\u003e another way to turn inference time compute into better performance.\n\n\u003e Can be added to your DSPy program for a long-context models.\n\n🌟 **Martian LLM Router, Martian: Hu et al (2024)**\n[pdf](https://arxiv.org/pdf/2403.12031.pdf),\n[blog](https://withmartian.com/)\n\n\u003e Martian provide the first LLM router, which dynamically routes queries\n\u003e to the best LLM in real-time, to achieve higher performance and lower cost\n\u003e than any individual API. They're able to choose models which might be better\n\u003e at a single task and to route away from powerful expensive models when a cheaper\n\u003e one will suffice. In order to choose which model to use, they use a new interpretability\n\u003e technique known as [model mapping](https://blog.withmartian.com/post/mission#:~:text=Understanding%20Models%20Through%20Model%20Mapping).\n\u003e Worth paying attention to.\n\n\u003e Chip Huyen discusses model routing approaches [here](https://huyenchip.com/2024/02/28/predictive-human-preference.html)\n\n**EAGLE, Peking/Microsft: Li et al (2024)**\n[pdf](https://arxiv.org/pdf/2401.15077.pdf),\n[PyTorch code](https://github.com/SafeAILab/EAGLE)\n\n\u003e An improvement to speculative decoding which uses the fact that upper\n\u003e layers in the model have good features for multiple tokens ahead to predict\n\u003e future tokens from the current one without using all the layers.\n\u003e This approach is typically 50% faster than previous single-model speculative\n\u003e decoding efforts and 3x faster than vanilla decoding.\n\n🌟 **Contrastive Decoding, Stanford: Li et al (2023)**\n[pdf](https://arxiv.org/pdf/2210.15097.pdf),\n[pdf2](https://arxiv.org/pdf/2309.09117.pdf)\n\n\u003e A small helper model generates tokens alongside the main model. Tokens are\n\u003e up-weighted if the large model finds them proportionally much more plausible\n\u003e than the small model. This approach improves the quality of open-ended\n\u003e generations and reasoning ability. To extend this method towards additionally\n\u003e adaptive computation, smaller contrastive models could be applied\n\u003e conditionally depending on the input.\n\n🌟 **Speculative Sampling, DeepMind: Chen et al (2023)**\n[pdf](https://arxiv.org/pdf/2302.01318.pdf),\n[pdf2](https://arxiv.org/pdf/2211.17192.pdf),\n[blog](https://jaykmody.com/blog/speculative-sampling/),\n[PyTorch code](https://github.com/jaymody/speculative-sampling)\n[PyTorch blog](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/)\n\n\u003e A smaller model generates multiple tokens autoregressively and then a larger\n\u003e model checks the smaller model against what it would have generated (all in\n\u003e one go). We accept only the tokens where the two models agree (by some\n\u003e acceptance criteria) and then the larger model's next token. This gives\n\u003e exactly the same output as the larger model would have but with significantly\n\u003e reduced sampling time. This takes advantage of the fact that we can\n\u003e parallelise evaluation whilst generation happens token by token. Additionally\n\u003e [Online Speculative Decoding](https://arxiv.org/pdf/2310.07177.pdf) suggests\n\u003e we can use any excess compute (at inference time) to retrain the small model\n\u003e online on the query distribution with teacher-student distillation.\n\u003e Note that the small model need not be a transformer:\n\u003e [Recurrent Drafter](https://arxiv.org/pdf/2403.09919.pdf) from Apple suggest\n\u003e using a fast RNN for speculative decoding and [large n-gram models](https://arxiv.org/pdf/2401.17377.pdf)\n\u003e could also be used as a non-parametric approach. Indeed [REST](https://www.semanticscholar.org/paper/REST%3A-Retrieval-Based-Speculative-Decoding-He-Zhong/532c2c7a247d9e97d20abec1b2f4612984fdab93) suggest retrieving follow-on tokens from the web for the speculative decoding head.\n\u003e See also [Accelerated Speculative Sampling (ASpS) with Tree Monte Carlo](https://openreview.net/pdf?id=stMhi1Sn2G) (or [video](https://www.youtube.com/watch?v=53VqZFmOSB8)) for further improvements to this method.\n\n\u003c!-- The general principle here is that it's easier to evaluate than to generate. --\u003e\n\n**FrugalGPT, Stanford: Chen et al (2023)**\n[pdf](https://arxiv.org/pdf/2305.05176.pdf)\n\n\u003e Details various approaches for fully black box adaptive computation (i.e. from\n\u003e an API where you don't even get logits). They use an LLM Cascade strategy\n\u003e where given a prompt they select n models to try sampling with, in order of\n\u003e increasing parameter count. The first model samples and we check the\n\u003e generation with a scoring function. If the generation is rejected then we\n\u003e generate with a more capable model. We continue this process until we accept\n\u003e the generation or are using the largest model. Interestingly this approach\n\u003e provides some shielding against\n\u003e [inverse scaling](https://arxiv.org/pdf/2306.09479.pdf) problems. They also\n\u003e use completion caching.\n\n**Beam Search, Google: Sutskever et al (2014)**\n[pdf](https://browse.arxiv.org/pdf/1409.3215.pdf)\n\n\u003e Beam search allows LMs to see the probability of choosing a few tokens at a\n\u003e time before selecting one by building out a tree. Increasing the number of\n\u003e beams increases the number of options explored downstream and hence the amount\n\u003e of compute per token.\n\n\u003c!-- Test-Time Compute is the System 2. We want to convert time to think into better outputs. We want to do structured reasoning over trees of possibilities. We want to convert time into better accuracy so when you think for longer, you get monotonically more accurate outputs.  --\u003e\n\n## Continual Learning\n\n🌟 **Lifelong-MoE, Google DeepMind: Chen et al (2023)**\n[pdf](https://arxiv.org/pdf/2305.12281.pdf)\n\n\u003e Trains a language model for multiple tasks by training for one task, freezing\n\u003e these weights and then adding some additional layers which can help to train\n\u003e the next task (in combination with the frozen layers) This treats pretrained\n\u003e weights more like an API (which you can use but not edit) when training a\n\u003e model to do a new task. This helps to eliminate the catastrophic forgetting\n\u003e that can happen with naive finetuning.\n\n**Sparse Upcycling, Google Research: Komatsuzaki et al (2023)**\n[pdf](https://arxiv.org/pdf/2212.05055.pdf)\n\n\u003e Shows that you can use an pre-trained dense model checkpoints as an\n\u003e initialisation for training sparse MoEs. This reduces the overall compute\n\u003e budget needed and reduces the sunk costs for already trained models. Sparse\n\u003e upcycling can be viewed as an efficient form of finetuning which converts a\n\u003e pretrained dense model to a sparse model for inference.\n\n**MuNet, Google: Gesmundo et al (2022-23)**\n[pdf](https://arxiv.org/pdf/2205.10937.pdf),\n[pdf2](https://arxiv.org/pdf/2205.12755.pdf),\n[pdf3](https://arxiv.org/pdf/2209.14745.pdf),\n[pdf4](https://arxiv.org/pdf/2302.02721.pdf),\n[official jax code](https://github.com/google-research/google-research/tree/master/muNet)\n\n\u003e Defines an evolutionary algorithm which adds different tasks onto an existing\n\u003e base model by (1) inserting adapter layers, (2) changing hyperparameters, (3)\n\u003e freezing layers and (4) copying layers to retrain. An interesting sketch of\n\u003e what Adaptive Computation could look like in the future.\n\n## Tools \u0026 Agents\n\nOne way of varying compute is on some tokens calling out to an external API for\nparts of completions.\n\n**SWE-Agent: Princeton, Yang et al (2024)**\n[code](https://github.com/princeton-nlp/SWE-agent)\n[demo](https://swe-agent.com/)\n\n\u003e An AI Software Engineer (à la Devin) which takes a GitHub issue and autonomously tries to fix it. Operates fast (couple of minutes) and performs well on the SWE-bench benchmark. One of the first AI agents to actually work in the real world and it's open-source.\n\n🌟 **LLM-Powered Autonomous Agents, OpenAI: Lilian Weng (2023)**\n[blog](https://lilianweng.github.io/posts/2023-06-23-agent/)\n\n\u003e An overview of agents as general problem-solvers powered by LLMs such as\n\u003e [AutoGPT](https://github.com/Significant-Gravitas/Auto-GPT) and\n\u003e [GPT-Engineer](https://github.com/AntonOsika/gpt-engineer) Agents typically\n\u003e can act within the world are are augmented with the ability to do explicit\n\u003e long term planning (via decomposing goals into sub-goals and learning from its\n\u003e mistakes), long-term memory (via a vector database) and tool use (calling\n\u003e external APIs).\n\n**ChatGPT Plugins, OpenAI (2023)**\n[blog](https://openai.com/blog/chatgpt-plugins),\n[demo](https://chat.openai.com/?model=gpt-4)\n\n\u003e GPT-4 has access to plugins for tasks where it would be better suited to call\n\u003e an API. Examples include Code Interpreter, web browser and Wolfram Alpha.\n\n\u003c!-- RETRO, DeepMind:\n\u003e k-Nearest Neighbour approaches\n --\u003e\n\n**Toolformer, Meta: Schick et al (2023)**\n[pdf](https://arxiv.org/pdf/2302.04761.pdf),\n[pdf2](https://arxiv.org/pdf/2305.17126.pdf)\n\n\u003e Trained models to decide which APIs to call, when to call them, what arguments\n\u003e to pass, and how to best incorporate the results into future token prediction\n\u003e Effectively the LMs teach themselves how to use tools. In the limit case of\n\u003e this we simply require LMs/agents to be able to ask the right questions, know\n\u003e where to ask them and possibly be able to interpret the answers they receive.\n\u003e In other words, we offload the actual computation to external APIs (which may\n\u003e themselves be ML models) and use much smaller base models.\n\n## Games\n\n🌟 **Libratus: heads-up no-limit poker, Meta: Brown and Sandholm (2017)**\n[pdf](https://www.science.org/doi/epdf/10.1126/science.aao1733),\n[pdf2](https://arxiv.org/pdf/1705.02955.pdf),\n[video](https://www.youtube.com/watch?v=2dX0lwaQRX0)\n\n\u003e The first AI to beat humans at Texas Hold Em Poker (heads up). An important\n\u003e part of the approach was in computing real-time responses to opponent moves,\n\u003e spending more compute on less obvious moves.\n\n**AlphaGo/AlphaZero, DeepMind: Silver et al (2016)**\n[pdf](https://storage.googleapis.com/deepmind-media/alphago/AlphaGoNaturePaper.pdf),\n[pdf2](https://www.nature.com/articles/nature24270.epdf),\n[film](https://www.youtube.com/watch?v=WXuK6gekU1Y),\n[blog](https://www.deepmind.com/research/highlighted-research/alphago)\n\n\u003e This result needs no introduction. In terms of Adaptive Computation, they the\n\u003e depth of the Monte Carlo Tree Search (MCTS) was allowed to be variable.\n\n## Pre-cursors to Adaptive Computation\n\n**Dynamic Evaluation**\n[blog](https://gwern.net/doc/ai/nn/dynamic-evaluation/index)\n\n\u003e Dynamic evaluation is an inference-time finetuning approach which which allows for online learning to increase performance on a given task. This was popular for RNN approaches but has fallen out of favour due to wanting simple-to-deploy models over an API and the rise of In-Context Learning. Similar approaches have seen some success on the [ARC challenge](https://arcprize.org/arc). See also [Jack Cole interview](https://lab42.global/community-interview-jack-cole/)\n\n**Attention and The Transformer, Vaswani et al (2017)**\n[pdf](https://arxiv.org/pdf/1706.03762.pdf)\n[pdf2](https://arxiv.org/pdf/1409.0473.pdf)\n\n\u003e Although we don't normally think of it this way, attention can be viewed as a\n\u003e conditional computation mechanism. The matrix which is applied to the input is\n\u003e dependant on the incoming data.\n\n**Conditional Computation, Bengio et al. (2016)**\n[pdf](https://arxiv.org/pdf/1511.06297.pdf)\n\n\u003e They use Reinforcement Learning to train a policy gradient to decide which\n\u003e parts of the network to activate, in effect learning a dropout policy for\n\u003e sparsity.\n\n**Adaptive Mixtures of Local Experts, Jacobs et al (1991)**\n[pdf](https://www.cs.toronto.edu/~hinton/absps/jjnh91.pdf)\n\n\u003e Collaborative, learned Mixture of Experts approaches to handle subsets of the\n\u003e training set are proposed. It's remarkable how close current approaches are to\n\u003e the original gating network. They also show intuitive expert specialisation on\n\u003e the task of vowel discrimination.\n\n## Open Source Libraries\n\n🌟 **DSPy, Stanford: Kandpal et al (2023)**\n[pdf](https://arxiv.org/pdf/2310.03714.pdf),\n[code](https://github.com/stanfordnlp/dspy)\n\n\u003e A framework which allows AI engineers to build LLM pipelines in code. Here we can also\n\u003e algorithmically optimize LM prompts and weights using their compilation tools. Within\n\u003e this framework pipelines are written like PyTorch code and engineers can write control\n\u003e flows to allow for Adaptive Computation.\n\n**git_theta, UNC: Kandpal et al (2023)**\n[pdf](https://arxiv.org/pdf/2306.04529.pdf),\n[official framework-agnostic code](https://github.com/r-three/git-theta/)\n\n\u003e A git extension which allows tracking and merging changes to model checkpoints\n\u003e like git does with code. With git_theta you can see diffs in parameter groups\n\u003e and merge model finetuning branches with merging approaches. It's also\n\u003e efficient with low-rank changes to parameter groups.\n\n🌟 **MegaBlocks, Stanford/Databricks (2022)**\n[pdf](https://arxiv.org/pdf/2211.15841.pdf)\n[pytorch code](https://github.com/databricks/megablocks)\n\n\n\u003e A lightweight library for training MoE models which is well integrated with\n\u003e MegaTron-LM. Maintained by Databricks and used by Mistral, it's becoming the\n\u003e standard in MoE training. The core of the library is implementing \"dropless-MoE\"\n\u003e efficiently.\n\n🌟 **DeepSpeed-MoE, Microsoft: Rajbhandari et al (2022)**\n[blog](https://www.microsoft.com/en-us/research/blog/deepspeed-advancing-moe-inference-and-training-to-power-next-generation-ai-scale/),\n[pdf](https://arxiv.org/pdf/2201.05596.pdf),\n[official PyTorch code](https://github.com/microsoft/DeepSpeed/tree/master/deepspeed/moe)\n\n\u003e Training and inference solution for distributed MoE models. They also present\n\u003e a new MoE architecture PR-MoE which has more experts in higher layers and a\n\u003e method for distilling expert models into dense 'student models'.\n\n\u003c!-- Sten (2022) [pdf](https://arxiv.org/pdf/2304.07613.pdf)\n\u003e PyTorch implementation of efficient, unstructured sparsity linear algebra operations with gradients.\n\n--\u003e\n\n## AI Safety\n\nWith adaptive computation, models can choose to use more compute on harder\nproblems.\n\nFor problems where we're concerned about systems failing by not being able to do\nsufficient computation then Adaptive Computation is very positive for Alignment.\nWe should expect fewer mistakes from a model utilising Adaptive Computation,\neven on more difficult problems. Additionally,\n[Adaptive Computation based systems are less susceptible to Adversarial Attacks](https://arxiv.org/pdf/2210.10253.pdf).\nThat is to say Adaptive Computation makes models more `robust`.\n\nHowever, for problems where we're concerned about systems being deceptive or\nmesa-optimising increasing the amount of inference-time compute increases their\nability to do so. Here the failure is not a \"mistake\" but entirely intentional\nfrom the system's perspective. Inference-time search is one way that a model\ncould implement [deceptive alignment](https://arxiv.org/pdf/1906.01820.pdf) for\nexample.\n\n## Scaling Laws\n\n\n**Toward Inference-Optimal MoEs, UCSD: Yun et al (2024)**\n[pdf](https://arxiv.org/pdf/2404.02852)\n\n\u003e The Chinchilla scaling laws focused on how to allocate compute to get the best model for a given amount of training compute. Since then LLama and others have focused on optimising for inference-compute as well as training compute. For MoEs there are additional considerations here - how many experts should you assign for the parameter count given that at inference time cost depends on the active parameters? The authors find that fewer experts are more efficient at inference time but more experts are more efficient at training time.\n\n**Knowledge Capacity Scaling Laws, Meta: Allen-Zhu \u0026 Li (2024)**\n[pdf](https://arxiv.org/pdf/2404.05405.pdf)\n\n\u003e The authors examine the Physics of Language Models and how much\n\u003e data they can store per parameter. They find that typically models can store\n\u003e around 2 bits per parameter, and this doesn't reduce too much with MoE models.\n\u003e This confirms (since MoEs are typically much larger in parameter count) that\n\u003e these models can store a lot more information than traditional models.\n\u003e It also suggests that knowledge capacity is relatively independent of\n\u003e forward pass compute giving a natural (if imprecise) cleaving of intelligence\n\u003e represented in compute applied and knowledge represented in the parameters.\n\n**Sparse Scaling Laws, DeepMind: Frantar et al (2023)**\n[pdf](https://arxiv.org/pdf/2309.08520.pdf)\n\n\u003e Scaling laws paper in the style of the\n\u003e [Chinchilla paper](https://arxiv.org/pdf/2203.15556.pdf). Details the optimal\n\u003e sparsity for a model given the inference FLOPs and training budget. They\n\u003e suggest that sparsity is especially important for larger models when seeing\n\u003e diminishing returns past Chinchilla optimality. See also\n\u003e [Unified Scaling Laws](https://browse.arxiv.org/pdf/2202.01169.pdf).\n\u003e\n\u003e Further [Scaling Laws For Fine-Grained MoEs](https://arxiv.org/pdf/2402.07871.pdf)\n\u003e suggest ways to optimally select the trade-off between the number and size of experts.\n\n**Scaling Scaling Laws with Board Games, Andy Jones (2021)**\n[pdf](https://arxiv.org/pdf/2104.03113.pdf)\n\n\u003e [The Bitter Lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html)\n\u003e suggests that there are two general techniques that work well in Machine\n\u003e Learning - search and learning. This paper suggests that these can be traded\n\u003e off against one another - that is, instead of additional learning you could\n\u003e add capable search to achieve similar performance. We can trade off train-time\n\u003e and test-time compute depending on our requirements.\n\n## Other\n\n**Blending Is All You Need, Cambridge: Lu et al (2024)**\n[pdf](https://arxiv.org/pdf/2401.02994)\n\n\u003e Mark this under \"bizarre\". They have a multi-modal setup where they completely randomly and uniformly select a model to answer each query in a conversation. The authors report higher user engagement and retention metrics using this approach over each individual model. One hypothesis is that each model can influence models that answer afterwards through the conditioning on previous tokens and there might be some implicit benefits to a model seeing tokens which are slightly off-distribution for what it would have produced. Perhaps they also have different refusal policies too. It's not clear why this should work but this provides a floor for more sophisticated model routing procedures.\n\n**Buffer Overflow in MoEs, DeepMind: Hayes et al (2024)**\n[pdf](https://arxiv.org/pdf/2402.05526.pdf)\n\n\u003e Typically with token choice routing methods in MoEs, there are implicitly cross-batch dependencies (i.e. the same token could be routed to different experts if in it's batch most of the other tokens also wanted to go to its preferred expert). The authors show that this batch dependency can be used as an attack surface. They present a few solutions - mostly this shouldn't be a problem if batch sizes are very large (as in inference in a big AI lab) but it's an interesting one to watch out for. We might expect ML systems security to be an increasingly large field of research.\n\n**FLOPs are all you need, Emin Orhan (2023)**\n[blog](https://severelytheoretical.wordpress.com/2023/08/14/flops-are-all-you-need-a-conjecture-about-what-really-makes-deep-learning-work/)\n\n\u003e Short post detailing how the success of deep learning models is very\n\u003e correlated with the amount of compute that they use per parameter efficiently\n\u003e and how they share parameters.\n\n**Review Paper: Dynamic Neural Networks Survey, Han et al (2022)**\n[pdf](https://arxiv.org/pdf/2102.04906.pdf)\n\n\u003e A review of Adaptive Computation approaches.\n\n\u003c!--\n\nLottery Tickets: if we prune we really do get sparsity but the problem is that the sparsity is not useful to us on modern hardware. We need block sparsity to take advantage of this. In the future it might be possible to use less structured sparsity and then this will become very relevant again.\n\n--\u003e\n\n\u003c!--\n## Benchmarks\n\nAll the usual benchmarks apply but ones that are especially suited to Adaptive Computation methods include:\n\nParity\n\nComplex logic questions\n\nContextQA location dataset\n\nARB (DuckAI benchmark)\n\nAgents benchmarks\n\nSparsity May Cry (SMC)\n\n--\u003e\n\n\u003c!-- ## Approaches We're Excited To See Explored More\n\n- When we have early exiting we essentially have to train classifiers for each layer in addition to the main model so we have additional overhead for training which is going to save us compute at inference tine. Are there better, more principled ways of early exiting at train time as well so that we don't have to learn very much from easy tokens?\n\n- Current approaches to sparsity are mainly transformer with some sparsity added on the margin. Transformers have worked so well and people are generally leaving them alone and messing with everything else around them - we're interested in paradigm shift approaches which are completely sparse and move further away from the transformer.\n--\u003e\n\n\u003cbr\u003e\n\n---\n\n\u003cbr\u003e\n\nThanks for reading, if you have any suggestions or corrections please submit a\npull request! And please hit the star button to show your appreciation.\n\n## Citing This Post\n\nIf you'd like to cite this article, please use:\n```\n@misc{ayonrinde_2023_awesome_adaptive_computation,\n  author = \"Kola Ayonrinde\",\n  title = \"Awesome Adaptive Computations\",\n  year = 2023,\n  publisher = \"GitHub\",\n  url = \"https://github.com/koayon/awesome-adaptive-computation/\"\n}\n```\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/koayon%2Fawesome-adaptive-computation/projects"}