https://github.com/aryamanarora/bayesian-laws-icl

Bayesian scaling laws for in-context learning.
https://github.com/aryamanarora/bayesian-laws-icl

Last synced: about 7 hours ago
JSON representation

Bayesian scaling laws for in-context learning.

• Host: GitHub
• URL: https://github.com/aryamanarora/bayesian-laws-icl
• Owner: aryamanarora
• Created: 2024-04-26T01:24:35.000Z (7 months ago)
• Default Branch: main
• Last Pushed: 2024-10-10T23:16:51.000Z (about 1 month ago)
• Last Synced: 2024-10-26T04:16:24.565Z (20 days ago)
• Language: Python
• Homepage:
• Size: 14.8 MB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        


# Bayesian scaling laws for in-context learning

**Bayesian scaling laws for in-context learning**\

(Arora et al., 2024)



This repo implements theoretically-motivated Bayesian scaling laws for in-context learning. We include a code for pretraining and finetuning (SFT or DPO) small LMs on GINC in order to collect their ICL curves (adapted from [p-lambda/incontext-learning](https://github.com/p-lambda/incontext-learning)), running inference on real-world LLMs locally or through Together API in order to collect their ICL curves, and code for fitting and analysing our Bayesian scaling laws as well as some baselines. We also include a script for replicating all figures and tables in the paper.

## Instructions

TBD