{"id":13589685,"url":"https://github.com/FSoft-AI4Code/TheVault","last_synced_at":"2025-04-08T09:33:49.641Z","repository":{"id":62928250,"uuid":"558296892","full_name":"FSoft-AI4Code/TheVault","owner":"FSoft-AI4Code","description":"[EMNLP 2023] The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation","archived":false,"fork":false,"pushed_at":"2024-08-21T18:48:49.000Z","size":9900,"stargazers_count":81,"open_issues_count":1,"forks_count":8,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-10-01T23:18:19.440Z","etag":null,"topics":["ai4code","dataset","multilingual-nlp"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FSoft-AI4Code.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-10-27T09:12:22.000Z","updated_at":"2024-08-29T02:03:43.000Z","dependencies_parsed_at":"2023-12-19T08:33:27.302Z","dependency_job_id":"803cc9b4-1838-4e0f-8dc1-cb15f183e260","html_url":"https://github.com/FSoft-AI4Code/TheVault","commit_stats":null,"previous_names":["ai4code-research/codetext-data"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FSoft-AI4Code%2FTheVault","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FSoft-AI4Code%2FTheVault/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FSoft-AI4Code%2FTheVault/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FSoft-AI4Code%2FTheVault/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FSoft-AI4Code","download_url":"https://codeload.github.com/FSoft-AI4Code/TheVault/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223314285,"owners_count":17125045,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai4code","dataset","multilingual-nlp"],"created_at":"2024-08-01T16:00:33.011Z","updated_at":"2024-11-06T09:31:20.544Z","avatar_url":"https://github.com/FSoft-AI4Code.png","language":"Jupyter Notebook","funding_links":[],"categories":["Datasets"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./assets/the-vault-4-logo-png.png\" width=\"300px\" alt=\"logo\"\u003e\n\u003c/p\u003e\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT) [![Python 3.8](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/release/python-380/) [![arXiv](https://img.shields.io/badge/arXiv-2305.06156-b31b1b.svg)](https://arxiv.org/abs/2305.06156) [![The Vault on HuggingFace datasets](https://img.shields.io/badge/%F0%9F%A4%97%20Datasets-The%20Vault-yellow?style=flat)](https://huggingface.co/datasets/Fsoft-AIC/the-vault-function) \n\n# The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation\n\u003c/div\u003e\n\n## Table of content\n- [The Vault Dataset](#the-vault-dataset)\n  - [Data Summary](#data-summary)\n  - [Data Structure](#data-structure)\n    - [Data Instance](#data-instances)\n    - [Data Fields](#data-fields)\n    - [Data Near Deduplication](#data-near-deduplication)\n    - [Splitting Train/Eval/Test](#splitting-trainevaltest)\n    - [Splitting Train set](#splitting-trainset-into-multiple-subsets)\n  - [Download Dataset](#download-dataset)\n    - [Load dataset on Huggingface](#load-dataset-on-huggingface)\n    - [Download via link](#download-via-link)\n- [The Vault toolkit](#the-vault-toolkit)\n  - [Getting Started](#getting-started)\n  - [Processing Pipeline](#processing-pipeline)\n    - [Extracting raw code](#extracting-raw-code)\n    - [Filtering extracted code snippet](#filtering-extracted-code-snippet)\n    - [Processing Custom Dataset](#processing-custom-dataset)\n- [Citing The Vault](#citing-the-vault)\n- [Contact Us](#contact-us)\n- [License](#license)\n\n___________\n\n\n# The Vault Dataset\n## Data Summary\nThe Vault dataset is a comprehensive, large-scale, multilingual parallel dataset that features high-quality code-text pairs derived from The Stack, the largest permissively-licensed source code dataset.\n\nWe provide The Vault which contains code snippets from 10 popular programming languages such as Java, JavaScript, Python, Ruby, Rust, Golang, C#, C++, C, and PHP. This dataset provides multiple code-snippet levels, metadata, and 11 docstring styles for enhanced usability and versatility.\n\n![Something something](./assets/TheVault-Poster.png)\n## Data Structure\n### Data Instances\nEvery sample of The Vault are stored in form of a json object and compressed into a large json line file. Each sample corresponds to one raw code file. The content of the file are used to extracting function, class and inline set, other information (repository name, licenses, etc) are collected from source dataset (The Stack).\n\n### Data Fields\nSee detail of data fields and example for each type of set [Here](./data/README.md)\n\n### Data Near-Deduplication\nWe applied deduplication for internal and external.\n\n- **Internal**: Apply exact deduplicate in full dataset.\n- **External**: Apply near deduplicate with the test sets of CodeSearchNet, HumanEval and APPS.\n\n*[Near-deduplication](https://chenghaomou.github.io/posts/20230220150602) use MinHash LSH to clustering sample based on their code. Those sample are close to each other (or even modified version) can be detected.\n\n### Splitting train/eval/test\nWe have divided the complete dataset into three distinct sets: a training set, an evaluation set, and a test set, to maintain consistency throughout the experiment.\n\nTo avoid data leakage, we allocated all samples from the same repository to a singular set. We then subdivided these sets using code tokens as splitting factors. As a result, these subsets mirror the distribution of the full dataset.\n\n\n### Splitting trainset into multiple subsets\nGiven the substantial size of our dataset, we found it beneficial to further divide the training set into two smaller subsets for ease of experimentation:\n\n- A small training set, which contains 5% of the total data.\n- A medium training set, comprising 20% of the full dataset.\n- (And) the full training set.\n\n|            | Small set   | Medium set   | Train set  | Validation | Test   | Total         |\n|:-----------|------------:|-------------:|-----------:|-----------:|-------:|--------------:|\n|Python      |   370,657   |  1,952,110   | 7,772,647  | 30,992     | 21,652 | 7,825,291     |\n|Java        |   351,213   |  1,612,366   | 6,629,193  | 22,677     | 15,552 | 6,667,422     |\n|JavaScript  |    82,931   |    404,729   | 1,640,416  | 22,044     | 21,108 | 1,683,568     |\n|PHP         |   236,638   |  1,155,476   | 4,656,371  | 21,375     | 19,010 | 4,696,756     |\n|C           |   105,978   |    381,207   | 1,639,319  | 27,525     | 19,122 | 1,685,966     |\n|C#          |   141,090   |    783,166   | 3,305,891  | 24,787     | 19,638 | 3,350,316     |\n|C++         |    87,420   |    410,907   | 1,671,268  | 20,011     | 18,169 | 1,709,448     |\n|Go          |   267,535   |  1,319,547   | 5,109,020  | 19,102     | 25,314 | 5,153,436     |\n|Ruby        |    23,921   |    112,574   |   424,339  | 17,338     | 19,908 |   461,585     |\n|Rust        |    35,367   |    224,015   |   825,130  | 16,716     | 23,141 |   864,987     |\n|TOTAL       | 1,702,750   |  8,356,097   |33,673,594  |222,567     |202,614 |**34,098,775** |\n\n\n## Download dataset\n### Load dataset on Huggingface\n\nWe publish The Vault ([function](https://huggingface.co/datasets/Fsoft-AIC/the-vault-function)/[inline](https://huggingface.co/datasets/Fsoft-AIC/the-vault-inline)/[class](https://huggingface.co/datasets/Fsoft-AIC/the-vault-class)) on Huggingface dataset hub.\n\n```python\nfrom datasets import load_dataset\n\n# Load full function/class/inline level dataset\ndataset = load_dataset(\"Fsoft-AIC/the-vault-{function/class/inline}\")\n\n# Load function level train/validation/test set\ndataset = load_dataset(\"Fsoft-AIC/the-vault-function\", split_set=[\"train\"])\n\n# Load \"small\" (or \"medium\", \"full\") function level training set\ndataset = load_dataset(\"Fsoft-AIC/the-vault-function\", split_set=[\"train/small\"])\n\n# specific language (e.g. Python) \ndataset = load_dataset(\"Fsoft-AIC/the-vault-function\", split_set=[\"train\"], languages=['Python'])\n\n# dataset streaming\ndata = load_dataset(\"Fsoft-AIC/the-vault-function\", split_set= [\"train\"], streaming= True)\nfor sample in iter(data['train']): \n    print(sample)\n```\n### Download via link\n\nOr download the Vault directly from Azure blob storage via download link. Here are the link pattern for specific download option:\n\u003e https://ai4code.blob.core.windows.net/thevault/v1/{function,class,inline}/{python,java,javascript,go,cpp,c_sharp,c,rust,ruby,php}.zip\n\nFor example, download *class* of *Python*:\n\u003e https://ai4code.blob.core.windows.net/thevault/v1/class/python.zip\n\nOr download using the script [`download_dataset.py`](./resources/download_dataset.py):\n```bash\npython download_dataset.py \"\u003cpath/to/destination\u003e\" --set \"function\" # or class/inline\n```\n\n_Note: blob link currently only contains function-level version_\n\n# The Vault Toolkit\n## Getting Started\n\nTo setup environment and install dependencies via `pip`:\n```bash\npip -r install requirements.txt\n```\n\nInstall `codetext` parser to extract code using [tree-sitter](https://tree-sitter.github.io/tree-sitter/), via `pip`:\n```bash\npip install codetext\n```\n\nOr manually build `codetext` form source, see more at [`Codetext` repo](https://github.com/FSoft-AI4Code/CodeText-parser)\n```bash\ngit clone https://github.com/FSoft-AI4Code/CodeText-parser.git\ncd CodeText-parser\npip install -e .\n```\n\n## Processing Pipeline\nOur toolkit takes raw source code files as input and streamlines the extraction and generation of\ncode-text pairs, as illustrated in Figure above.\nThere are 3 seperate process:\n1. **Extracting Raw code:** By using Tree-sitter extractor to identify function/class/line node inside raw file and obtain their metadata\n2. **Extracting Docstring Style:** We implement a docstring style parser to capture all the informative section or param's description inside a docstring\n3. **Filtering Docstring:** From the docstring gathered from previous process, we use it as main factor to filter quality sample (remove empty docstring, uninformative docstring, etc). See more about cleaning methodologies at our [paper](https://arxiv.org/abs/2305.06156)\n\nWe seperated the source code into multiple steps (coresponde for each process). Or you can run the full pipeline follow [this tutorial](#processing-custom-dataset).\n\n![Extracting pipeline](./assets/pipeline.png)\n### Extracting Raw code\nFrom raw code, you can extract function, class using [`process_raw_node()`](./src/utils/utils.py#L138). An example structure of a raw code snippet show in the figure below. Inside a node are identifier, parameter or argument list, code and comment (docstring). \n\n```python\nfrom codetext.utils import parse_code\nfrom codetext.parser import PythonParser\n\ncode_snippet = \"\"\"\ndef sum2num(a: int, b: int):\n  '''\n  :param a: first number\n  :param b: second number\n  '''\n  return a + b # result\n\"\"\"\ncode_tree = parse_code(code_snippet, 'cpp')\n\nres = process_raw_node(\n    tree=code_tree, \n    blob=code_snippet,\n    language_parser=PythonParser(),\n    metadata={'repo': 'test'}  # Optional\n)\n\n# or extrating line\n\nres = get_line_definitions(\n    tree=code_tree, \n    blob=code_snippet,\n    language_parser=PythonParser(),\n    source_metadata={'repo': 'test'}  # Optional\n)\n```\n\nFor extracting raw inline comment, the function [`get_line_definitions()`](./src/utils/utils.py#L279) can help to extract line comment and return the parent code block, previous and next context (i.e. code block).\n\n```python\nfrom codetext.utils import parse_code\nfrom codetext.parser import PythonParser\n\ncode_snippet = \"\"\"\ndef sum2num(a: int, b: int):\n  '''\n  :param a: first number\n  :param b: second number\n  '''\n  return a + b\n\"\"\"\ncode_tree = parse_code(code_snippet, 'cpp')\n\nres = process_raw_node(\n    tree=code_tree, \n    blob=code_snippet,\n    language_parser=PythonParser(),\n    metadata={'repo': 'test'}  # Optional\n)\n```\n\n![Raw node structure](./assets/raw-node.png)\n\n### Filtering Extracted code snippet\nWith the result function or class node and their metadata extracted from previous process, execute [`get_node_definitions()`](./src/utils/utils.py#L238) to filtering sample based on their docstring. Heuristic rules will remove sample that not meet the minimum requirement (We detailedly describe it inside our \n).\n\n![](./assets/docstring-style.png)\n\nLastly, to extracting docstring style we implement a function call [`extract_docstring()`](./src/utils/utils.py#L514) that take docstring (in form of string) as input and result metadata of the docstring style as demonstrate in the figure above (e.g. param's docstring, type, return's docstring, etc.)\n\n\n### Processing Custom Dataset\nWe create a `.yaml` to define which field to load when processing data. Usually, only source code are needed, but in case there are other additional information about the raw code might be added using the `.yaml`.\n\nFor example, `CodeSearchNet` stores their data in structure:\n\n```yaml\n# CodeSearchNet jsonline format \n# https://github.com/github/CodeSearchNet#data-details\n\ncode: original_string # raw code\nrepo: repo # additional infor\npath: path # additional infor\nlanguage: language # additional infor\n```\nInside `processing.py` we merged extracting raw code, filtering docstring and extracting docstring style function into 1 simple pipeline for quickly extracting dataset from raw source data. You can use `processing.py` by:\n```bash\npython -m codetext.processing \n\u003cDATASET_PATH\u003e\n--save_path \u003cSAVE_PATH\u003e  # path to save dir\n\n--load_from_file  # load from file instead load from dataset cache\n--language Python  # or Java, JavaScript, ...\n--data_format './data/format/codeparot-format.yaml'  # load raw data format\n\n--n_split 20  # split original dataset into N subset\n--n_core -1  # number of multiple processor (default to 1) (-1 == using all core)\n```\n\nArguments list:\n```\npositional arguments:\n  data_path             data folder contain file.jsonl or huggingface dataset cache\n\noptions:\n  -h, --help            show this help message and exit\n  --save_path SAVE_PATH\n                        Processed data save path\n  --level LEVEL         Extract function/class/inline level or all\n  --language LANGUAGE   Declare processing language (e.g: Python, Java)\n  --data_format DATA_FORMAT\n                        Path to file .yaml contains data format\n  --load_from_file      Load from .json or .jsonl\n  --cons_from_raw       Continues from raw .jsonl (pass folder path to data)\n  --raw_only\n  --filtered_only\n  --extracted_only\n  --n_split N_SPLIT     Split all the raw data into N file and feed into process pool\n  --n_core N_CORE       Number of maximum process to create\n  --debug\n```\n\n# Citing The Vault\nMore details can be found in our [paper](https://arxiv.org/abs/2305.06156). \n\nIf you're using The Vault or the toolkit in your research or applications, please cite using this BibTeX:\n```bibtex\n@article{manh2023vault,\n  title={The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation},\n  author={Manh, Dung Nguyen and Hai, Nam Le and Dau, Anh TV and Nguyen, Anh Minh and Nghiem, Khanh and Guo, Jin and Bui, Nghi DQ},\n  journal={arXiv preprint arXiv:2305.06156},\n  year={2023}\n}\n```\n\n# Contact us\nIf you have any questions, comments or suggestions, please do not hesitate to contact us.\n- Website: [fpt-aicenter](https://www.fpt-aicenter.com/ai-residency/)\n- Email: support.ailab@fpt.com\n\n# License\n[MIT License](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFSoft-AI4Code%2FTheVault","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFSoft-AI4Code%2FTheVault","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFSoft-AI4Code%2FTheVault/lists"}