https://github.com/aryamanarora/bayesian-laws-icl
Bayesian scaling laws for in-context learning.
https://github.com/aryamanarora/bayesian-laws-icl
Last synced: 3 months ago
JSON representation
Bayesian scaling laws for in-context learning.
- Host: GitHub
- URL: https://github.com/aryamanarora/bayesian-laws-icl
- Owner: aryamanarora
- Created: 2024-04-26T01:24:35.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-12-25T01:52:03.000Z (4 months ago)
- Last Synced: 2024-12-25T02:35:36.756Z (4 months ago)
- Language: Python
- Homepage:
- Size: 14.9 MB
- Stars: 10
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Bayesian scaling laws for in-context learning
**Bayesian scaling laws for in-context learning**\
(Arora et al., 2024)
This repo implements theoretically-motivated Bayesian scaling laws for in-context learning. We include a code for pretraining and finetuning (SFT or DPO) small LMs on GINC in order to collect their ICL curves (adapted from [p-lambda/incontext-learning](https://github.com/p-lambda/incontext-learning)), running inference on real-world LLMs locally or through Together API in order to collect their ICL curves, and code for fitting and analysing our Bayesian scaling laws as well as some baselines. We also include a script for replicating all figures and tables in the paper.
## Instructions
TBD