{"id":19428861,"url":"https://github.com/eleutherai/the-pile","last_synced_at":"2025-10-06T05:04:09.974Z","repository":{"id":39413196,"uuid":"290621741","full_name":"EleutherAI/the-pile","owner":"EleutherAI","description":null,"archived":false,"fork":false,"pushed_at":"2023-04-27T21:00:48.000Z","size":265,"stargazers_count":1571,"open_issues_count":27,"forks_count":140,"subscribers_count":28,"default_branch":"master","last_synced_at":"2025-05-23T01:07:37.650Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EleutherAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-08-26T22:54:52.000Z","updated_at":"2025-05-22T11:23:01.000Z","dependencies_parsed_at":"2024-01-14T07:02:13.628Z","dependency_job_id":"3675ae30-6b2b-4b51-b941-9e7d020ed670","html_url":"https://github.com/EleutherAI/the-pile","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/EleutherAI/the-pile","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EleutherAI%2Fthe-pile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EleutherAI%2Fthe-pile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EleutherAI%2Fthe-pile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EleutherAI%2Fthe-pile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EleutherAI","download_url":"https://codeload.github.com/EleutherAI/the-pile/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EleutherAI%2Fthe-pile/sbom","scorecard":{"id":45080,"data":{"date":"2025-08-11","repo":{"name":"github.com/EleutherAI/the-pile","commit":"df97f8651ae3da658b19659b3ceaa6a34b0fc014"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Code-Review","score":5,"reason":"Found 5/10 approved changesets -- score normalized to 5","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":2,"reason":"8 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2024-4 / GHSA-2mqj-m65w-jghx","Warn: Project is vulnerable to: PYSEC-2023-165 / GHSA-cwvm-v4w8-q58c","Warn: Project is vulnerable to: PYSEC-2022-42992 / GHSA-hcpj-qp55-gfph","Warn: Project is vulnerable to: PYSEC-2023-137 / GHSA-pr76-5cm5-w9cj","Warn: Project is vulnerable to: PYSEC-2023-161 / GHSA-wfm5-v35h-vwf4","Warn: Project is vulnerable to: PYSEC-2017-74","Warn: Project is vulnerable to: PYSEC-2011-23 / GHSA-3jhc-wjqf-5f2c","Warn: Project is vulnerable to: PYSEC-2024-187 / GHSA-rqc4-2hc7-8c8v"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 27 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-14T22:33:09.941Z","repository_id":39413196,"created_at":"2025-08-14T22:33:09.941Z","updated_at":"2025-08-14T22:33:09.941Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278561279,"owners_count":26006954,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-06T02:00:05.630Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T14:16:56.514Z","updated_at":"2025-10-06T05:04:09.918Z","avatar_url":"https://github.com/EleutherAI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The Pile Replication Code\n\n## The official website for the the Pile is [here](http://pile.eleuther.ai/).\n\nThe Pile is a large, diverse, open source language modelling data set that consists of many smaller datasets combined together. The objective is to obtain text from as many modalities as possible to ensure that models trained using The Pile will have much broader generalization abilities.\n\n\n**This repository is for replicating or making variants of the Pile. IF YOU ARE HERE TO USE THE PILE DATASET, THIS REPO IS PROBABLY NOT WHAT YOU ARE LOOKING FOR. A copy of the Pile can be downloaded [here](https://the-eye.eu/public/AI/pile/).**\n\n|    Component    | Raw Size |Weight|Epochs|Effective Size|Mean Document Size|\n|-----------------|----------|------|-----:|--------------|------------------|\n|[Pile-CC](https://github.com/leogao2/commoncrawl_downloader)      |227.12 GiB|18.11%|   1.0|227.12 GiB    |4.33 KiB          |\n|[PubMed Central](https://github.com/EleutherAI/pile-pubmedcentral)   |90.27 GiB |14.40%|   2.0|180.55 GiB    |30.55 KiB         |\n|[Books3](https://twitter.com/theshawwn/status/1320282149329784833)        |100.96 GiB|12.07%|   1.5|151.44 GiB    |538.36 KiB        |\n|[OpenWebText2](https://github.com/EleutherAI/openwebtext2)     |62.77 GiB |10.01%|   2.0|125.54 GiB    |3.85 KiB          |\n|[ArXiv](https://gist.github.com/leogao2/e09b64eae3b987925ccf3b86401624c6)            |56.21 GiB |8.96% |   2.0|112.42 GiB    |46.61 KiB         |\n|[Github](https://github.com/EleutherAI/github-downloader)           |95.16 GiB |7.59% |   1.0|95.16 GiB     |5.25 KiB          |\n|[FreeLaw](https://github.com/thoppe/The-Pile-FreeLaw)          |51.15 GiB |6.12% |   1.5|76.73 GiB     |15.06 KiB         |\n|[StackExchange](https://github.com/EleutherAI/stackexchange-dataset)    |32.20 GiB |5.13% |   2.0|64.39 GiB     |2.16 KiB          |\n|[USPTO Backgrounds](https://github.com/EleutherAI/pile-uspto)            |22.90 GiB |3.65% |   2.0|45.81 GiB     |4.08 KiB          |\n|[PubMed Abstracts](https://github.com/thoppe/The-Pile-PubMed) |19.26 GiB |3.07% |   2.0|38.53 GiB     |1.30 KiB          |\n|[Gutenberg (PG-19)](https://github.com/deepmind/pg19)|10.88 GiB |2.17% |   2.5|27.19 GiB     |398.73 KiB        |\n|[OpenSubtitles](https://github.com/sdtblck/Opensubtitles_dataset)    |12.98 GiB |1.55% |   1.5|19.47 GiB     |30.48 KiB         |\n|[Wikipedia (en)](https://github.com/noanabeshima/wikipedia-downloader)   |6.38 GiB  |1.53% |   3.0|19.13 GiB     |1.11 KiB          |\n|[DM Mathematics](https://github.com/deepmind/mathematics_dataset)   |7.75 GiB  |1.24% |   2.0|15.49 GiB     |8.00 KiB          |\n|[Ubuntu IRC](https://github.com/EleutherAI/pile-ubuntu-irc)       |5.52 GiB  |0.88% |   2.0|11.03 GiB     |545.48 KiB        |\n|[BookCorpus2](https://github.com/shawwn/scrap/blob/master/epub2txt-all)       |6.30 GiB  |0.75% |   1.5|9.45 GiB      |369.87 KiB        |\n|[EuroParl](https://github.com/thoppe/The-Pile-EuroParl)         |4.59 GiB  |0.73% |   2.0|9.17 GiB      |68.87 KiB         |\n|[HackerNews](https://github.com/EleutherAI/hn-scraper)       |3.90 GiB  |0.62% |   2.0|7.80 GiB      |4.92 KiB          |\n|[YoutubeSubtitles](https://github.com/sdtblck/youtube_subtitle_dataset) |3.73 GiB  |0.60% |   2.0|7.47 GiB      |22.55 KiB         |\n|[PhilPapers](https://github.com/thoppe/The-Pile-PhilPapers)       |2.38 GiB  |0.38% |   2.0|4.76 GiB      |73.37 KiB         |\n|[NIH ExPorter](https://github.com/thoppe/The-Pile-NIH-ExPORTER)     |1.89 GiB  |0.30% |   2.0|3.79 GiB      |2.11 KiB          |\n|[Enron Emails](https://github.com/EleutherAI/pile-enron-emails)     |0.88 GiB  |0.14% |   2.0|1.76 GiB      |1.78 KiB          |\n|**Total**        |          |      |      |1254.20 GiB   |5.91 KiB          |\n\n\n(Epochs refers to the number of epochs elapsed after 1.2TB)\n\n\n## Usage\n\n\nInstall:\n\n```\npip install -e .\n```\n\n### To replicate pile\n\n```\npython the_pile/pile.py --interleave_output 30 --using pile_reprod\n```\n\nUse the pass 2 script [here](https://github.com/EleutherAI/The-Pile/tree/master/processing_scripts) to complete shuffling.\n\n\n### Other\n\nTo force download all data:\n```\npython the_pile/pile.py --force_download\n```\n\nTo generate fasttext training data for CC filtering (OWT2 only):\n```\nsudo apt install build-essential\npython the_pile/pile.py --using owt2 --make_fasttext \n```\n\n## Manual Download Components\n\nThe following components need manual downloading. Either download them or comment out from `pile.py`. \n\n - **Bibliotik**: `books3.tar.gz` needs to be in the current directory. Download temporarily unavailable.\n\n## Workflow\n\nTo propose a new dataset be added to the Pile, [open an issue](https://github.com/EleutherAI/The-Pile/issues/new). Your issue should include a description of the dataset, its size, what language(s) it is in, a link to the data, and any other relevant information. If a project manger approves your proposal, they will change its label to [![Datasets](https://img.shields.io/github/labels/EleutherAI/The-Pile/Dataset)](https://github.com/EleutherAI/The-Pile/labels/Dataset) and add it to [![Project: Datasets](https://img.shields.io/badge/Project-Datasets-lightgrey)](https://github.com/EleutherAI/The-Pile/projects/2). Datasets that we elect to not include in the current version of the Pile will receive a [![Deferred](https://img.shields.io/github/labels/EleutherAI/The-Pile/Deferred%20to%20v2)](https://github.com/EleutherAI/The-Pile/labels/Deferred%20to%20v2) or [![Declined](https://img.shields.io/github/labels/EleutherAI/The-Pile/Declined)](https://github.com/EleutherAI/The-Pile/labels/Declined) label. While we welcome multilingual  datasets and plan on including non-English datasets in the future, the initial release of the Pile will be English-only and all submissions of non-English datasets will be deferred.\n\nTo claim responsibility for implementing an unclaimed dataset, leave a comment on one of our unassigned issues. Once an dataset has been assigned to you, make the necessary changes to `datsets.py` and `pile.py` in a fork and submit a pull request. If you require, you can also submit a script for processing the data as shown [here](https://github.com/EleutherAI/pile_enron_emails).\n\nTo raise an issue that is not proposing a new dataset, open an issue with the tag [![Feature Request](https://img.shields.io/github/labels/EleutherAI/The-Pile/Feature%20Request)](https://github.com/EleutherAI/The-Pile/labels/Feature%20Request) or [![Bug](https://img.shields.io/github/labels/EleutherAI/The-Pile/Bug)](https://github.com/EleutherAI/The-Pile/labels/Bug) as appropriate.\n\nData ready for final implementation should meet the following criteria:\n\n- The data must be in [lm_dataformat](https://github.com/leogao2/lm_dataformat/) format.\n- The data must be shuffled.\n\n**In preparation for the initial release, we are no longer accepting additions to the *master* branch. If you would like to contribute a dataset, please submit the pull request to the *Version2* branch.**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feleutherai%2Fthe-pile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feleutherai%2Fthe-pile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feleutherai%2Fthe-pile/lists"}