{"id":13992595,"url":"https://github.com/huggingface/datablations","last_synced_at":"2025-10-14T15:33:04.271Z","repository":{"id":61164242,"uuid":"546757879","full_name":"huggingface/datablations","owner":"huggingface","description":"Scaling Data-Constrained Language Models","archived":false,"fork":false,"pushed_at":"2025-06-28T00:02:14.000Z","size":47993,"stargazers_count":338,"open_issues_count":5,"forks_count":18,"subscribers_count":30,"default_branch":"main","last_synced_at":"2025-07-18T07:45:28.334Z","etag":null,"topics":["gpt","high-performance-computing","language-models","large-language-models","llms","scaling-laws"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2305.16264","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-06T15:41:07.000Z","updated_at":"2025-07-16T20:43:29.000Z","dependencies_parsed_at":"2024-09-17T02:47:21.751Z","dependency_job_id":"8ef47ad6-3483-439d-8a2a-03f9972c3dfe","html_url":"https://github.com/huggingface/datablations","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/huggingface/datablations","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdatablations","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdatablations/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdatablations/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdatablations/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/datablations/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdatablations/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266530857,"owners_count":23943991,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpt","high-performance-computing","language-models","large-language-models","llms","scaling-laws"],"created_at":"2024-08-09T14:02:03.275Z","updated_at":"2025-10-14T15:32:59.233Z","avatar_url":"https://github.com/huggingface.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"readme":"# Scaling Data-Constrained Language Models\n\n![](plotstables/return_alloc.png)\n\nThis repository provides an overview of all components from the paper [Scaling Data-Constrained Language Models](https://arxiv.org/abs/2305.16264). Talks on the paper:\n\n- [65min talk](https://www.youtube.com/watch?v=TK0-sitkCMw\u0026t=2746s\u0026pp=ygUoc2NhbGluZyBkYXRhLWNvbnN0cmFpbmVkIGxhbmd1YWdlIG1vZGVscw%3D%3D) by Niklas Muennighoff\n- [60min talk](https://www.youtube.com/watch?v=lLV-g-rGPhk\u0026t=1s\u0026pp=ygUoc2NhbGluZyBkYXRhLWNvbnN0cmFpbmVkIGxhbmd1YWdlIG1vZGVscw%3D%3D) by Niklas Muennighoff\n- [63min talk](https://www.youtube.com/watch?v=Kp5R6GZh8O0\u0026pp=ygUoc2NhbGluZyBkYXRhLWNvbnN0cmFpbmVkIGxhbmd1YWdlIG1vZGVscw%3D%3D) by Alexander M. Rush\n- [25min talk](https://www.youtube.com/watch?v=FQk2YSgYLmU\u0026pp=ygUoc2NhbGluZyBkYXRhLWNvbnN0cmFpbmVkIGxhbmd1YWdlIG1vZGVscw%3D%3D) by Alexander M. Rush\n- [60min slide deck](https://docs.google.com/presentation/d/1DvOytecCyqJyNnii8tVWCGUXHH4-hPqKJ5axxnNqxBQ/edit?usp=sharing\u0026resourcekey=0-hr2Olxzz-mcMbAU7ytCZcw) used by Niklas Muennighoff\n\nWe investigate scaling language models in data-constrained regimes. We run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. Based on our runs we propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. We also experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data, perplexity-filtering and deduplication. Models and datasets from our 400 training runs are available via this repository.\n\n\u003c!-- TOC --\u003e\n\n- [Data](#data)\n    - [Repeating](#repeating)\n    - [Code](#code)\n    - [Filtering](#filtering)\n        - [Perplexity](#perplexity)\n        - [Deduplication](#deduplication)\n- [Models](#models)\n    - [Download](#download)\n    - [Training](#training)\n        - [Regular models](#regular-models)\n        - [muP](#mup)\n- [Parametric Fit](#parametric-fit)\n- [Downstream Evaluation](#downstream-evaluation)\n    - [Rank Eval / Accuracy](#rank-eval--accuracy)\n    - [Generative / Rouge](#generative--rouge)\n    - [bAbI / Exact match](#babi--exact-match)\n- [Plots \u0026 Tables](#plots--tables)\n    - [Plots](#plots)\n    - [Tables](#tables)\n- [License](#license)\n- [Citation](#citation)\n\n\u003c!-- /TOC --\u003e\n\n## Data\n\n### Repeating\n\nWe experiment with repeating data on [C4](https://huggingface.co/datasets/c4) and the non-deduplicated English split of [OSCAR](https://huggingface.co/datasets/oscar). For each dataset, we download the data and turn it into a single jsonl file, `c4.jsonl` and `oscar_en.jsonl` respectively.\n\nThen we decide on the amount of unique tokens and the respective number of samples we need from the dataset. Note that C4 has `478.625834583` tokens per sample and OSCAR has `1312.0951072` with the GPT2Tokenizer. This was calculated by tokenizing the entire dataset and dividing the number of tokens by the number of samples. We use these numbers to calculate the needed samples.\n\nFor example, for 1.9B unique tokens, we need `1.9B / 478.625834583 = 3969697.96178` samples for C4 and `1.9B / 1312.0951072 = 1448065.76107` samples for OSCAR. To tokenize the data, we first need to clone the [Megatron-DeepSpeed](https://github.com/TurkuNLP/Megatron-DeepSpeed) repository and follow its setup guide. We then select these samples and tokenize them as follows:\n\nC4:\n```bash\nhead -n 3969698 c4.jsonl \u003e c4_1b9.jsonl\npython Megatron-DeepSpeed/tools/preprocess_data_many_cores.py \\\n--input c4_1b9.jsonl \\\n--output-prefix gpt2tok_c4_en_1B9 \\\n--dataset-impl mmap \\\n--tokenizer-type PretrainedFromHF \\\n--tokenizer-name-or-path gpt2 \\\n--append-eod \\\n--workers 64\n```\n\nOSCAR:\n```bash\nhead -n  1448066 oscar_en.jsonl \u003e oscar_1b9.jsonl\npython Megatron-DeepSpeed/tools/preprocess_data_many_cores.py \\\n--input oscar_1b9.jsonl \\\n--output-prefix gpt2tok_oscar_en_1B9 \\\n--dataset-impl mmap \\\n--tokenizer-type PretrainedFromHF \\\n--tokenizer-name-or-path gpt2 \\\n--append-eod \\\n--workers 64\n```\n\nwhere `gpt2` points to a folder containing all files from https://huggingface.co/gpt2/tree/main. By using `head` we make sure that different subsets will have overlapping samples to reduce randomness.\n\n\nFor evaluation during training and the final evaluation, we use the validation set for C4:\n\n```python\nfrom datasets import load_dataset\nload_dataset(\"c4\", \"en\", split=\"validation\").to_json(\"c4-en-validation.json\")\n```\n```bash\npython Megatron-DeepSpeed/tools/preprocess_data_many_cores.py \\\n--input c4-en-validation.jsonl \\\n--output-prefix gpt2tok_c4validation_rerun \\\n--dataset-impl mmap \\\n--tokenizer-type PretrainedFromHF \\\n--tokenizer-name-or-path gpt2 \\\n--append-eod \\\n--workers 2\n```\n\nFor OSCAR which has no official validation set we take a part of the training set by doing `tail -364608 oscar_en.jsonl \u003e oscarvalidation.jsonl` and then tokenize it as follows:\n\n```bash\npython Megatron-DeepSpeed/tools/preprocess_data_many_cores.py --input oscarvalidation.jsonl --output-prefix gpt2tok_oscarvalidation --dataset-impl mmap --tokenizer-type PretrainedFromHF --tokenizer-name-or-path gpt2 --append-eod --workers 2\n```\n\nWe have uploaded several preprocessed subsets for usage with megatron:\n- C4: https://huggingface.co/datasets/datablations/c4-subsets\n- OSCAR: https://huggingface.co/datasets/datablations/oscar-subsets\n\nSome bin files were too large for git and thus split using e.g. `split --number=l/40 gpt2tok_c4_en_1B9.bin gpt2tok_c4_en_1B9.bin.` and `split --number=l/40 gpt2tok_oscar_en_1B9.bin gpt2tok_oscar_en_1B9.bin.`. To use them for training you need to cat them together again using `cat gpt2tok_c4_en_1B9.bin.* \u003e gpt2tok_c4_en_1B9.bin` and `cat gpt2tok_oscar_en_1B9.bin.* \u003e gpt2tok_oscar_en_1B9.bin`.\n\n### Code\n\nWe experiment with mixing code with the natural language data using the Python split from the [the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup). We download the data, turn it into a single jsonl file and preprocess it using the same approach as outlined above.\n\nWe have uploaded the preprocessed version for usage with megatron here: https://huggingface.co/datasets/datablations/python-megatron. We have split the bin file using `split --number=l/40 gpt2tok_python_content_document.bin gpt2tok_python_content_document.bin.`, so you need to cat them together again using `cat gpt2tok_python_content_document.bin.* \u003e gpt2tok_python_content_document.bin` for training.\n\n### Filtering\n\nWe create versions of C4 and OSCAR with perplexity and deduplication-related filtering metadata:\n- C4: https://huggingface.co/datasets/datablations/c4-filter\n- OSCAR: https://huggingface.co/datasets/datablations/oscar-filter\n\nTo recreate these metadata datasets there are instructions at `filtering/README.md`.\n\nWe provide the tokenized versions that can be used for training with Megatron at:\n- C4: https://huggingface.co/datasets/datablations/c4-filter-megatron\n- OSCAR: https://huggingface.co/datasets/datablations/oscar-filter-megatron\n\n`.bin` files were split using something like `split --number=l/10 gpt2tok_oscar_en_perplexity_25_text_document.bin gpt2tok_oscar_en_perplexity_25_text_document.bin.`, so you need to concatenate them back together via `cat gpt2tok_oscar_en_perplexity_25_text_document.bin. \u003e gpt2tok_oscar_en_perplexity_25_text_document.bin`.\n\nTo recreate the tokenized versions given the metadata dataset,\n- OSCAR:\n    - Deduplication: See `filtering/deduplication/filter_oscar_jsonl.py`\n    - Perplexity: See below.\n- C4:\n    - Deduplication: See below.\n    - Perplexity: See below.\n\n#### Perplexity\n\nTo create the perplexity percentiles, follow the below instructions.\n\nC4:\n\n```python\nfrom datasets import load_dataset\nimport numpy as np\nds = load_dataset(\"datablations/c4-filter\", streaming=False, num_proc=128)\n\np_25 = np.percentile(ds[\"train\"][\"perplexity\"], 25)\np_50 = np.percentile(ds[\"train\"][\"perplexity\"], 50)\np_75 = np.percentile(ds[\"train\"][\"perplexity\"], 75)\n\n# 25 - 75th percentile\nds[\"train\"].filter(lambda x: p_25 \u003c x[\"perplexity\"] \u003c p_75, num_proc=128).to_json(\"c4_perplexty2575.jsonl\", num_proc=128, force_ascii=False)\n# 25th percentile\nds[\"train\"].filter(lambda x: x[\"perplexity\"] \u003c p_25, num_proc=128).to_json(\"c4_perplexty25.jsonl\", num_proc=128, force_ascii=False)\n# 50th percentile\nds[\"train\"].filter(lambda x: x[\"perplexity\"] \u003c p_50, num_proc=128).to_json(\"c4_perplexty50.jsonl\", num_proc=128, force_ascii=False)\n```\n\nOSCAR:\n\n```python\nfrom datasets import load_dataset\nimport numpy as np\nds = load_dataset(\"datablations/oscar-filter\", use_auth_token=True, streaming=False, num_proc=128)\n\np_25 = np.percentile(ds[\"train\"][\"perplexity_score\"], 25)\np_50 = np.percentile(ds[\"train\"][\"perplexity_score\"], 50)\n\n# 25th percentile\nds[\"train\"].filter(lambda x: x[\"perplexity_score\"] \u003c p_25, num_proc=128).remove_columns(['meta', 'perplexity_score', 'text_length', 'url', 'domain', 'dup_ratio', 'pairs', 'repetitions', 'included_in_dedup', 'cluster', 'id']).to_json(\"oscar_perplexity25.jsonl\", num_proc=128, force_ascii=False)\n# 50th percentile\nds[\"train\"].filter(lambda x: x[\"perplexity_score\"] \u003c p_50, num_proc=128).remove_columns(['meta', 'perplexity_score', 'text_length', 'url', 'domain', 'dup_ratio', 'pairs', 'repetitions', 'included_in_dedup', 'cluster', 'id']).to_json(\"oscar_perplexity50.jsonl\", num_proc=128, force_ascii=False)\n```\n\nYou can then tokenize the resulting jsonl files for training with Megatron as described in the [Repeating](#repeating) section.\n\n#### Deduplication\n\n\nC4:\nFor C4 you just need to remove all samples where the `repetitions` field is populated, via e.g.\n\n```python\nfrom datasets import load_dataset\nimport numpy as np\nds = load_dataset(\"datablations/c4-dedup\", use_auth_token=True, streaming=False, num_proc=128)\nds.filter(lambda x: not(x[\"repetitions\"]).to_json('c4_dedup.jsonl', num_proc=128, force_ascii=False)\n```\n\nOSCAR:\nFor OSCAR we provide a script at `filtering/filter_oscar_jsonl.py` to create the deduplicated dataset given the dataset with filtering metadata.\n\n\nYou can then tokenize the resulting jsonl files for training with Megatron as described in the [Repeating](#repeating) section.\n\n## Models\n\n### Download\n\nAll models can be downloaded at https://huggingface.co/datablations. \n\nModels are generally named as follows: `lm1-{parameters}-{tokens}-{unique_tokens}`, specifically individual models in the folders are named as: `{parameters}{tokens}{unique_tokens}{optional specifier}`, for example `1b12b8100m` would be 1.1 billion params, 2.8 billion tokens, 100 million unique tokens. The `xby` (`1b1`, `2b8` etc.) convention introduces some ambiguity whether numbers belong to parameters or tokens, but you can always check the sbatch script in the respective folder to see the exact parameters / tokens / unique tokens. If you want to convert models that have not yet been converted to `huggingface/transformers`, you can follow the instructions in [Training](#regular-models).\n\nThe easiest way to download a single model is e.g.:\n```bash\nGIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datablations/lm1-misc\ncd lm1-misc; git lfs pull --include 146m14b400m/global_step21553\n```\n\nIf this takes too long, you can also use `wget` to directly download individual files from the folder, e.g.:\n```bash\nwget https://huggingface.co/datablations/lm1-misc/resolve/main/146m14b400m/global_step21553/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt\n```\n\nFor models corresponding to the experiments in the paper, consult the following repositories:\n- IsoLoss experiments \u0026 double descent data from the Appendix:\n    - https://huggingface.co/datablations/lm1-misc\n    - `lm1-misc/*dedup*` for deduplication comparison on 100M unique tokens in the appendix  \n- IsoFLOP experiments:\n    - C4:\n        - https://huggingface.co/datablations/lm1-8b7-178b-c4-repetitions\n        - https://huggingface.co/datablations/lm1-4b2-84b-c4-repetitions\n        - https://huggingface.co/datablations/lm1-4b2-84b-c4seeds\n        - https://huggingface.co/datablations/lm1-2b8-55b-c4-repetitions\n        - https://huggingface.co/datablations/lm1-2b8-55b-c4seeds\n    - OSCAR:\n        - https://huggingface.co/datablations/lm1-8b7-178b-oscar-repetitions\n        - https://huggingface.co/datablations/lm1-4b2-84b-oscar-repetitions\n        - https://huggingface.co/datablations/lm1-4b2-84b-oscarseeds\n        - https://huggingface.co/datablations/lm1-2b8-55b-oscar-repetitions\n        - https://huggingface.co/datablations/lm1-2b8-55b-oscarseeds  \n- Code augmentation:\n    - C4:\n        - https://huggingface.co/datablations/lm1-4b2-84b-c4py\n        - https://huggingface.co/datablations/lm1-4b2-84b-c4pyseeds\n        - https://huggingface.co/datablations/lm1-2b8-55b-c4py\n    - OSCAR:\n        - https://huggingface.co/datablations/lm1-2b8-55b-oscarpy\n- Perplexity-filtering:\n    - C4:\n        - https://huggingface.co/datablations/lm1-4b2-84b-c4-perplexity\n        - https://huggingface.co/datablations/lm1-2b8-55b-c4-perplexity\n        - https://huggingface.co/datablations/lm1-2b8-84b-c4-perplexity\n            - These are not referenced anywhere in the paper, but could be used\n    - OSCAR:\n        - https://huggingface.co/datablations/lm1-4b2-84b-oscar-perplexity\n        - https://huggingface.co/datablations/lm1-2b8-55b-oscar-perplexity\n- Deduplication:\n    - C4:\n        - https://huggingface.co/datablations/lm1-4b2-84b-c4-dedup\n        - https://huggingface.co/datablations/lm1-2b8-55b-c4-dedup\n    - OSCAR:\n        - https://huggingface.co/datablations/lm1-4b2-84b-oscar-dedup\n        - https://huggingface.co/datablations/lm1-2b8-55b-oscar-dedup\n- ROOTS filtering\n    - https://huggingface.co/datablations/lm1-4b2-84b-oscarroots\n    - https://huggingface.co/datablations/lm1-2b8-55b-oscarroots\n- muP (Appendix):\n    - https://huggingface.co/datasets/datablations/mup\n\nOther models not analysed in the paper:\n- UL2: https://huggingface.co/datablations/lm5-2b8-55b-c4\n- All checkpoints of the c4 8b7 model trained for 1 epoch: https://huggingface.co/datablations/lm1-8b7-176b-c4-ckpts\n- Adam beta2 ablation: https://huggingface.co/datablations/lm1-8b7-12b-beta\n- Warmup ablation: https://huggingface.co/datablations/lm1-83m-20b-nowarmup\n- An experiment pre-training only on instruction datasets: https://huggingface.co/datablations/lm1-2b8-55b-realtasky\n- Misc runs on The Pile: https://huggingface.co/datablations/lm1-misc-pile\n- Misc runs on OSCAR: https://huggingface.co/datablations/lm1-misc-oscar\n- IsoFLOP for 1.1B on C4: https://huggingface.co/datablations/lm1-1b1-21b-c4 \u0026 https://huggingface.co/datablations/lm1-1b1-21b-c4-repetitions \u0026 https://huggingface.co/datablations/lm1-1b1-21b-c4seeds\n\n### Training\n\n#### Regular models\n\nWe train models with our fork of Megatron-DeepSpeed that works with AMD GPUs (via ROCm): https://github.com/TurkuNLP/Megatron-DeepSpeed\nIf you would like to use NVIDIA GPUs (via cuda), you can use the original library: https://github.com/bigscience-workshop/Megatron-DeepSpeed\n\nYou need to follow the setup instructions of either repository to create your environment (Our setup specific to LUMI is detailed in `training/megdssetup.md`). \n\nEach model folder contains an sbatch script that was used to train the model. You can use these as a reference to train your own models adapting the necessary environment variables. The sbatch scripts reference some additional files:\n- `*txt` files that specify the data paths. You can find them at `utils/datapaths/*`, however, you will likely need to adapt the path to point to your dataset.\n- `model_params.sh`, which is at `utils/model_params.sh` and contains architecture presets.\n- `launch.sh` that you can find at `training/launch.sh`. It contains commands specific to our setup, which you may want to remove. \n\nAfter training you can convert your model to transformers with e.g. `python Megatron-DeepSpeed/tools/convert_checkpoint/deepspeed_to_transformers.py --input_folder global_step52452 --output_folder transformers --target_tp 1 --target_pp 1`.\n\nFor repeat models, we also upload their tensorboards after training using e.g. `tensorboard dev upload --logdir tensorboard_8b7178b88boscar --name \"tensorboard_8b7178b88boscar\"`, which makes them easy to use for visualization in the paper.\n\n#### muP\n\nFor the muP ablation in the Appendix we use the script at `training_scripts/mup.py`. It contains setup instructions.\n\n## Parametric Fit\n\nYou can use our formula to compute the expected loss given parameters, data and unique tokens as follows:\n\n```python\nimport numpy as np\nfunc = r\"$L(N,D,R_N,R_D)=E + \\frac{A}{(U_N + U_N * R_N^* * (1 - e^{(-1*R_N/(R_N^*))}))^\\alpha} + \\frac{B}{(U_D + U_D * R_D^* * (1 - e^{(-1*R_D/(R_D^*))}))^\\beta}$\"\na, b, e, alpha, beta, rd_star, rn_star = [6.255414, 7.3049974, 0.6254804, 0.3526596, 0.3526596, 15.387756, 5.309743]\n\nA = np.exp(a)\nB = np.exp(b)\nE = np.exp(e)\nG = ((alpha*A)/(beta*B))**(1/(alpha+beta))\n\ndef D_to_N(D):\n    return (D * G)**(beta/alpha) * G\n\ndef scaling_law(N, D, U):\n    \"\"\"\n    N: number of parameters\n    D: number of total training tokens\n    U: number of unique training tokens\n    \"\"\"\n    assert U \u003c= D, \"Cannot have more unique tokens than total tokens\"\n\n    RD = np.maximum((D / U) - 1, 0)    \n    UN = np.minimum(N, D_to_N(U))\n    RN = np.maximum((N / UN ) - 1, 0)\n\n    L = E + A/(UN + UN*rn_star*(1-np.exp(-1*RN/rn_star)))**alpha + B / (U + U * rd_star * (1 - np.exp(-1*RD/(rd_star))))**beta\n    return L\n\n# Models in Figure 1 (right):\nprint(scaling_law(6.34e9, 242e9, 25e9)) # 2.2256440889984477 # \u003c- This one is better\nprint(scaling_law(8.67e9, 178e9, 25e9)) # 2.2269634075087867\n```\n\nNote that the actual loss value is unlikely to be useful, but rather the trend of the loss as e.g. the number of parameters increases or to compare two models like in the example above. To compute the optimal allocation, you can use a simple grid search:\n\n```python\ndef chinchilla_optimal_N(C):\n    a = (beta)/(alpha+beta)\n    N_opt = G*(C/6)**a\n    return N_opt\n\ndef chinchilla_optimal_D(C):\n    b = (alpha)/(alpha+beta)\n    D_opt = (1/G)*(C/6)**b\n    return D_opt\n\ndef optimal_allocation(C, U_BASE):\n    \"\"\"Compute optimal number of parameters and tokens to train for given a compute \u0026 unique data budget\"\"\"\n    N_BASE = chinchilla_optimal_N(C)\n    D_BASE = chinchilla_optimal_D(C)\n    min_l = float(\"inf\")\n    for i in np.linspace(1.0001, 3, 500):\n        D =  D_BASE*i\n        U = min(U_BASE, D)\n        N = N_BASE/i\n        new_l = scaling_law(N, D, U)\n        if new_l \u003c min_l:\n            min_l, min_t, min_s = new_l, D, N\n        D =  D_BASE/i\n        U = min(U_BASE, D)\n        N = N_BASE*i\n        new_l = scaling_law(N, D, U)\n        if new_l \u003c min_l:\n            min_l, min_t, min_s = new_l, D, N\n    return min_l, min_t, min_s\n\n_, min_t, min_s = optimal_allocation(10**22, 25e9)\nprint(f\"Optimal configuration: {min_t} tokens, {min_t/25e9} epochs, {min_s} parameters\")\n# -\u003e 237336955477.55075 tokens, 9.49347821910203 epochs, 7022364735.879969 parameters\n# We went more extreme in Figure 1 to really put our prediction of \"many epochs, fewer params\" to the test\n```\n\nIf you derive a closed-form expression for the optimal allocation instead of the above grid search, please let us know :) We fit data-constrained scaling laws \u0026 the C4 scaling coefficients using the code at `utils/parametric_fit.ipynb` equivalent to [this colab](https://colab.research.google.com/drive/1tYIfsmOMoz4dZ_JiVp998vZiMhRqSQrf?usp=sharing).\n\n## Downstream Evaluation\n\n### Rank Eval / Accuracy\n\n1. Follow the instructions in the `Training` \u003e `Regular models` section to setup a training environment.\n2. Install the evaluation harness: `pip install git+https://github.com/EleutherAI/lm-evaluation-harness.git`. We used version 0.2.0, but newer versions should work as well.\n3. Run `sbatch utils/eval_rank.sh` modifying the necessary variables in the script first\n4. After running, we convert each file to csv using `python Megatron-DeepSpeed/tasks/eval_harness/report-to-csv.py outfile.json`\n\n### Generative / Rouge\n\n1. Clone the `addtasks` branch of the evaluation harness: `git clone -b addtasks https://github.com/Muennighoff/lm-evaluation-harness.git`\n2. Setup an environment with `cd lm-evaluation-harness; pip install -e \".[dev]\"; pip uninstall -y promptsource; pip install git+https://github.com/Muennighoff/promptsource.git@tr13` i.e. all requirements except promptsource, which is installed from a fork with the correct prompts\n3. Make sure your checkpoint path is a transformers checkpoint path\n4. Run `sbatch utils/eval_generative.sh` modifying the necessary variables in the script first\n5. After running, we merge the generation files using `python utils/merge_generative.py` and then convert them to csv with `python utils/csv_generative.py merged.json`\n\n### bAbI / Exact match\n\n1. Clone the `babi` branch of the evaluation harness: `git clone -b babi https://github.com/Muennighoff/lm-evaluation-harness.git` (Note that this branch is not compatible with the `addtasks` branch for generative tasks as it stems from `EleutherAI/lm-evaluation-harness`, while `addtasks` is based on `bigscience/lm-evaluation-harness`)\n2. Setup an environment with `cd lm-evaluation-harness; pip install -e \".[dev]\"`\n3. Make sure your checkpoint path is a transformers checkpoint with tokenizer files (If you trained a gpt2 model like all models in this work, it's just the files from here: https://huggingface.co/gpt2)\n4. Run `sbatch utils/eval_babi.sh`  modifying the necessary variables in the script first\n\n## Plots \u0026 Tables\n\n### Plots\n\n- Figure 1: `plotstables/return_alloc.pdf`, `plotstables/return_alloc.ipynb`, [colab](https://colab.research.google.com/drive/1wW3SjEoG6zPSsI7JI8TyY5uwYhVHvcLD?usp=sharing)\n- Figure 2: `plotstables/dataset_setup.pdf`, `plotstables/dataset_setup.ipynb`, [colab](https://colab.research.google.com/drive/1AqqoNduhgzOZs73geDlLBmmR8_q4frDf?usp=sharing)\n- Figure 3: `plotstables/contours.pdf`, `plotstables/contours.ipynb`, [colab](https://colab.research.google.com/drive/17eH3k6-Nh4NNTsjAMecossCghbL8Xe47?usp=sharing)\n- Figure 4: `plotstables/isoflops_training.pdf`, `plotstables/isoflops_training.ipynb`, [colab](https://colab.research.google.com/drive/1i00FUdVp0Oj-Qw40ITLSGXwzuxpqBjmw?usp=sharing)\n- Figure 5: `plotstables/return.pdf`, `plotstables/return.ipynb`, [colab](https://colab.research.google.com/drive/11L5AC2noZqlixQWjvSjCvGB_WnOWyuv7?usp=sharing)\n- Figure 6 (Left): `plotstables/strategies.pdf`, `plotstables/strategies.drawio`\n- Figure 6 (Right): `plotstables/beyond.pdf`, `plotstables/beyond.ipynb`, [colab](https://colab.research.google.com/drive/1SqB-4WxFm0WXL7rGqfwXjakjIb1yy6EA?usp=sharing)\n- Figure 7: `plotstables/cartoon.pdf`, `plotstables/cartoon.pptx`\n- Figure 8: `plotstables/isoloss_400m1b5.pdf` \u0026 same colab as Figure 3\n- Figure 9 - 11: `plotstables/mup.pdf`, `plotstables/dd.pdf`, `plotstables/dedup.pdf`, `plotstables/mup_dd_dd.ipynb`, [colab](https://colab.research.google.com/drive/1ghSdJMrGDaK_KM5vUgHYXkptS3sqKs8k?usp=sharing)\n- Figure 12: `plotstables/isoloss_alphabeta_100m.pdf` \u0026 same colab as Figure 3\n- Figure 13: `plotstables/galactica.pdf`, `plotstables/galactica.ipynb`, [colab](https://colab.research.google.com/drive/146lFaI6fhIdXaOAgtaqV4X3-rNGrH92C?usp=sharing)\n- Figure 14 - 17: `training_c4.pdf`, `validation_c4oscar.pdf`, `training_oscar.pdf`, `validation_epochs_c4oscar.pdf` \u0026 same colab as Figure 4\n- Figure 18: `plotstables/perplexity_histogram.pdf`, `plotstables/perplexity_histogram.ipynb`\n- Figure 19 - 20: `plotstabls/validation_c4py.pdf`, `plotstables/training_validation_filter.pdf`, `plotstables/beyond_losses.ipynb` \u0026 [colab](https://colab.research.google.com/drive/1bIJD0Gasv1GD1-27mrj6nURcQm6j-aEe?usp=sharing)\n- Figure 21 - 39: Manual\n\n### Tables\n\n- Table 1 - 3: Manual; The experiments for Table 1 are in `utils/parametric_fit.ipynb` equivalent to [this colab](https://colab.research.google.com/drive/1tYIfsmOMoz4dZ_JiVp998vZiMhRqSQrf?usp=sharing).\n- Table 4 - 9: `plotstables/repetition.ipynb` \u0026 [colab](https://colab.research.google.com/drive/19KW4ZiZ9iKzUY89A3IvRFzceGRpkyUK1?usp=sharing)\n- Table 10 - 11: `plotstables/python.ipynb` \u0026 [colab](https://colab.research.google.com/drive/1gFd83xDZ9JTTYkpaM2xCEgFcSKEpuTPH?usp=sharing)\n- Table 12: Manual\n- Table 13 - 14: `plotstables/filtering.ipynb` \u0026 [colab](https://colab.research.google.com/drive/1OKkrGBgRI8Uu_dq6RF8q_OIFGP-Saeah?usp=sharing)\n- Table 15: Manual\n\n## License\n\nAll models \u0026 code are licensed under Apache 2.0. Filtered datasets are released with the same license as the datasets they stem from.\n\n## Citation\n\n```bibtex\n@article{muennighoff2023scaling,\n  title={Scaling Data-Constrained Language Models},\n  author={Muennighoff, Niklas and Rush, Alexander M and Barak, Boaz and Scao, Teven Le and Piktus, Aleksandra and Tazi, Nouamane and Pyysalo, Sampo and Wolf, Thomas and Raffel, Colin},\n  journal={arXiv preprint arXiv:2305.16264},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fdatablations","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Fdatablations","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fdatablations/lists"}