awesome-adaptive-computation
A curated reading list of research in Adaptive Computation, Inference-Time Computation & Mixture of Experts (MoE).
https://github.com/koayon/awesome-adaptive-computation
Last synced: 9 days ago
JSON representation
About
- System 2 - ->
-
The Bitter Lesson
- trained models focus on
learningat train time, Adaptive Computation is about spending more compute at inference time with mechanisms similar toSearch. -->
Adaptive Computation for Black-box models
- Reflexion
- Debate
- Chain of Thought
- Tree of Thought
- Chain of Verification
- pdf2
- pdf2
- blog
- Online Speculative Decoding
- inverse scaling
- blog
- model mapping
- here
- Recurrent Drafter
- large n-gram models
- REST - on tokens from the web for the speculative decoding head.
- blog
- PyTorch blog
- inverse scaling
- PyTorch code
- pdf2
- PyTorch code
- Online Speculative Decoding
- Recurrent Drafter
- large n-gram models
- pytorch code
- Accelerated Speculative Sampling (ASpS) with Tree Monte Carlo
- Reflexion
- Debate
- Chain of Thought
- Tree of Thought
- Chain of Verification
- blog
- pdf2
AI Safety
Continual Learning
Early Exit: End-to-End Adaptive Computation
Games
- pdf2
- video
- pdf2
- film
- blog
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
- pdf2
Mixture of Experts (Sparse MoE)
- Mixtral-8x7B
- PyTorch code
- DeMix pdf
- Task-MoE pdf
- c-BTM
- official PyTorch code
- review paper
- PyTorch code
- model
- One Wide Feedforward paper
- Gemini 1.5 Pro - dbrx-new-state-art-open-llm) is another powerful MoE model and it seems that MoE is now the go-to architecture for large models.
- here
- models
- MoE-Infinity - Infinity)
- JetMoE
- Branch, Train, Mix (BTX)
- code
- Dynamic Routing in MoEs