https://github.com/bigcode-project/bigcode-dataset
https://github.com/bigcode-project/bigcode-dataset
Last synced: 6 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/bigcode-project/bigcode-dataset
- Owner: bigcode-project
- License: apache-2.0
- Created: 2022-10-31T10:01:57.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-08-15T13:42:30.000Z (over 1 year ago)
- Last Synced: 2025-07-18T10:42:49.612Z (6 months ago)
- Language: Jupyter Notebook
- Size: 3.8 MB
- Stars: 465
- Watchers: 10
- Forks: 76
- Open Issues: 16
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# BigCode Dataset
This repository gathers all the code used to build the BigCode datasets such as [The Stack](https://huggingface.co/datasets/bigcode/the-stack) as well as the preprocessing
necessary used for model training.
## Contents
- `language_selection`: notebooks and file with language to file extensions mapping used to build the Stack v1.1.
- `pii`: code for running PII detection and anonymization on code datasets.
- `decontamination`: script to remove files that match test-samples from code generation benchmarks.
- `preprocessing`: code for filtering code datasets based on:
- line length and percentage of alphanumeric characters (basic filter)
- number of stars, comments to code ratio, tokenizer fertility
- Additionnal filters used for StarCoder Training:
- basic-filter with parameters that depend on the file's extension.
- filter to remove XML files
- filter for HTML based on displayed-text VS code ratio
- filter to remove small and large files (for json and yaml)
- code to generate full-content with meta (repo-name, filename, num stars) for training
- Filters for GitHub Issues
- Filters for Git Commits
- Code to convert Jupyter notebooks to scripts
- Code to convert Jupyter notebooks to structured markdown-code-output triplets