{"id":13411942,"url":"https://github.com/huggingface/datasets","last_synced_at":"2026-03-17T02:04:50.841Z","repository":{"id":36957767,"uuid":"250213286","full_name":"huggingface/datasets","owner":"huggingface","description":"🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools","archived":false,"fork":false,"pushed_at":"2025-04-28T14:04:03.000Z","size":89207,"stargazers_count":20066,"open_issues_count":904,"forks_count":2818,"subscribers_count":279,"default_branch":"main","last_synced_at":"2025-05-05T11:15:01.015Z","etag":null,"topics":["computer-vision","datasets","deep-learning","hacktoberfest","machine-learning","natural-language-processing","nlp","numpy","pandas","pytorch","speech","tensorflow"],"latest_commit_sha":null,"homepage":"https://huggingface.co/docs/datasets","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null,"zenodo":".zenodo.json"}},"created_at":"2020-03-26T09:23:22.000Z","updated_at":"2025-05-05T08:07:36.000Z","dependencies_parsed_at":"2023-01-17T08:30:49.839Z","dependency_job_id":"3fe60c11-fbee-4f66-857e-23903b1ac94f","html_url":"https://github.com/huggingface/datasets","commit_stats":{"total_commits":3906,"total_committers":604,"mean_commits":6.466887417218543,"dds":0.8143881208397338,"last_synced_commit":"d4422cc24a56dc7132ddc3fd6b285c5edbd60b8c"},"previous_names":["huggingface/nlp"],"tags_count":107,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdatasets","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdatasets/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdatasets/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fdatasets/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/datasets/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252498833,"owners_count":21757872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","datasets","deep-learning","hacktoberfest","machine-learning","natural-language-processing","nlp","numpy","pandas","pytorch","speech","tensorflow"],"created_at":"2024-07-30T20:01:18.978Z","updated_at":"2026-03-17T02:04:50.835Z","avatar_url":"https://github.com/huggingface.png","language":"Python","readme":"\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://huggingface.co/datasets/huggingface/documentation-images/raw/main/datasets-logo-dark.svg\"\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://huggingface.co/datasets/huggingface/documentation-images/raw/main/datasets-logo-light.svg\"\u003e\n    \u003cimg alt=\"Hugging Face Datasets Library\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/raw/main/datasets-logo-light.svg\" width=\"352\" height=\"59\" style=\"max-width: 100%;\"\u003e\n  \u003c/picture\u003e\n  \u003cbr/\u003e\n  \u003cbr/\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/huggingface/datasets/actions/workflows/ci.yml?query=branch%3Amain\"\u003e\u003cimg alt=\"Build\" src=\"https://github.com/huggingface/datasets/actions/workflows/ci.yml/badge.svg?branch=main\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/huggingface/datasets/blob/main/LICENSE\"\u003e\u003cimg alt=\"GitHub\" src=\"https://img.shields.io/github/license/huggingface/datasets.svg?color=blue\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://huggingface.co/docs/datasets/index.html\"\u003e\u003cimg alt=\"Documentation\" src=\"https://img.shields.io/website/http/huggingface.co/docs/datasets/index.html.svg?down_color=red\u0026down_message=offline\u0026up_message=online\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/huggingface/datasets/releases\"\u003e\u003cimg alt=\"GitHub release\" src=\"https://img.shields.io/github/release/huggingface/datasets.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://huggingface.co/datasets/\"\u003e\u003cimg alt=\"Number of datasets\" src=\"https://img.shields.io/endpoint?url=https://huggingface.co/api/shields/datasets\u0026color=brightgreen\"\u003e\u003c/a\u003e\n    \u003ca href=\"CODE_OF_CONDUCT.md\"\u003e\u003cimg alt=\"Contributor Covenant\" src=\"https://img.shields.io/badge/Contributor%20Covenant-2.0-4baaaa.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://zenodo.org/badge/latestdoi/250213286\"\u003e\u003cimg src=\"https://zenodo.org/badge/250213286.svg\" alt=\"DOI\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n🤗 Datasets is a lightweight library providing **two** main features:\n\n- **one-line dataloaders for many public datasets**: one-liners to download and pre-process any of the ![number of datasets](https://img.shields.io/endpoint?url=https://huggingface.co/api/shields/datasets\u0026color=brightgreen) major public datasets (image datasets, audio datasets, text datasets in 467 languages and dialects, etc.) provided on the [HuggingFace Datasets Hub](https://huggingface.co/datasets). With a simple command like `squad_dataset = load_dataset(\"rajpurkar/squad\")`, get any of these datasets ready to use in a dataloader for training/evaluating a ML model (Numpy/Pandas/PyTorch/TensorFlow/JAX),\n- **efficient data pre-processing**: simple, fast and reproducible data pre-processing for the public datasets as well as your own local datasets in CSV, JSON, text, PNG, JPEG, WAV, MP3, Parquet, HDF5, etc. With simple commands like `processed_dataset = dataset.map(process_example)`, efficiently prepare the dataset for inspection and ML model evaluation and training.\n\n[🎓 **Documentation**](https://huggingface.co/docs/datasets/) [🔎 **Find a dataset in the Hub**](https://huggingface.co/datasets) [🌟 **Share a dataset on the Hub**](https://huggingface.co/docs/datasets/share)\n\n\u003ch3 align=\"center\"\u003e\n    \u003ca href=\"https://hf.co/course\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/imgs/course_banner.png\"\u003e\u003c/a\u003e\n\u003c/h3\u003e\n\n🤗 Datasets is designed to let the community easily add and share new datasets.\n\n🤗 Datasets has many additional interesting features:\n\n- Thrive on large datasets: 🤗 Datasets naturally frees the user from RAM memory limitation, all datasets are memory-mapped using an efficient zero-serialization cost backend (Apache Arrow).\n- Smart caching: never wait for your data to process several times.\n- Lightweight and fast with a transparent and pythonic API (multi-processing/caching/memory-mapping).\n- Built-in interoperability with NumPy, PyTorch, TensorFlow 2, JAX, Pandas, Polars and more.\n- Native support for audio, image and video data.\n- Enable streaming mode to save disk space and start iterating over the dataset immediately.\n\n🤗 Datasets originated from a fork of the awesome [TensorFlow Datasets](https://github.com/tensorflow/datasets) and the HuggingFace team want to deeply thank the TensorFlow Datasets team for building this amazing library.\n\n# Installation\n\n## With pip\n\n🤗 Datasets can be installed from PyPi and has to be installed in a virtual environment (venv or conda for instance)\n\n```bash\npip install datasets\n```\n\n## With conda\n\n🤗 Datasets can be installed using conda as follows:\n\n```bash\nconda install -c huggingface -c conda-forge datasets\n```\n\nFollow the installation pages of TensorFlow and PyTorch to see how to install them with conda.\n\nFor more details on installation, check the installation page in the documentation: https://huggingface.co/docs/datasets/installation\n\n## Installation to use with Machine Learning \u0026 Data frameworks frameworks\n\nIf you plan to use 🤗 Datasets with PyTorch (2.0+), TensorFlow (2.6+) or JAX (0.4+) you should also install PyTorch, TensorFlow or JAX.\n🤗 Datasets is also well integrated with data frameworks like PyArrow, Pandas, Polars and Spark, which should be installed separately.\n\nFor more details on using the library with these frameworks, check the quick start page in the documentation: https://huggingface.co/docs/datasets/quickstart\n\n# Usage\n\n🤗 Datasets is made to be very simple to use - the API is centered around a single function, `datasets.load_dataset(dataset_name, **kwargs)`, that instantiates a dataset.\n\nThis library can be used for text/image/audio/etc. datasets. Here is an example to load a text dataset:\n\nHere is a quick example:\n\n```python\nfrom datasets import load_dataset\n\n# Print all the available datasets\nfrom huggingface_hub import list_datasets\nprint([dataset.id for dataset in list_datasets(limit=20)])\n\n# Load a dataset and print the first example in the training set\nsquad_dataset = load_dataset('rajpurkar/squad')\nprint(squad_dataset['train'][0])\n\n# Process the dataset - add a column with the length of the context texts\ndataset_with_length = squad_dataset.map(lambda x: {\"length\": len(x[\"context\"])})\n\n# Process the dataset - tokenize the context texts (using a tokenizer from the 🤗 Transformers library)\nfrom transformers import AutoTokenizer\ntokenizer = AutoTokenizer.from_pretrained('bert-base-cased')\n\ntokenized_dataset = squad_dataset.map(lambda x: tokenizer(x['context']), batched=True)\n```\n\nIf your dataset is bigger than your disk or if you don't want to wait to download the data, you can use streaming:\n\n```python\n# If you want to use the dataset immediately and efficiently stream the data as you iterate over the dataset\nimage_dataset = load_dataset('timm/imagenet-1k-wds', streaming=True)\nfor example in image_dataset[\"train\"]:\n    break\n```\n\nFor more details on using the library, check the quick start page in the documentation: https://huggingface.co/docs/datasets/quickstart and the specific pages on:\n\n- Loading a dataset: https://huggingface.co/docs/datasets/loading\n- What's in a Dataset: https://huggingface.co/docs/datasets/access\n- Processing data with 🤗 Datasets: https://huggingface.co/docs/datasets/process\n    - Processing audio data: https://huggingface.co/docs/datasets/audio_process\n    - Processing image data: https://huggingface.co/docs/datasets/image_process\n    - Processing text data: https://huggingface.co/docs/datasets/nlp_process\n- Streaming a dataset: https://huggingface.co/docs/datasets/stream\n- etc.\n\n# Add a new dataset to the Hub\n\nWe have a very detailed step-by-step guide to add a new dataset to the ![number of datasets](https://img.shields.io/endpoint?url=https://huggingface.co/api/shields/datasets\u0026color=brightgreen) datasets already provided on the [HuggingFace Datasets Hub](https://huggingface.co/datasets).\n\nYou can find:\n- [how to upload a dataset to the Hub using your web browser or Python](https://huggingface.co/docs/datasets/upload_dataset) and also\n- [how to upload it using Git](https://huggingface.co/docs/datasets/share).\n\n# Disclaimers\n\nYou can use 🤗 Datasets to load datasets based on versioned git repositories maintained by the dataset authors. For reproducibility reasons, we ask users to pin the `revision` of the repositories they use.\n\nIf you're a dataset owner and wish to update any part of it (description, citation, license, etc.), or do not want your dataset to be included in the Hugging Face Hub, please get in touch by opening a discussion or a pull request in the Community tab of the dataset page. Thanks for your contribution to the ML community!\n\n## BibTeX\n\nIf you want to cite our 🤗 Datasets library, you can use our [paper](https://huggingface.co/papers/2109.02846):\n\n```bibtex\n@inproceedings{lhoest-etal-2021-datasets,\n    title = \"Datasets: A Community Library for Natural Language Processing\",\n    author = \"Lhoest, Quentin  and\n      Villanova del Moral, Albert  and\n      Jernite, Yacine  and\n      Thakur, Abhishek  and\n      von Platen, Patrick  and\n      Patil, Suraj  and\n      Chaumond, Julien  and\n      Drame, Mariama  and\n      Plu, Julien  and\n      Tunstall, Lewis  and\n      Davison, Joe  and\n      {\\v{S}}a{\\v{s}}ko, Mario  and\n      Chhablani, Gunjan  and\n      Malik, Bhavitvya  and\n      Brandeis, Simon  and\n      Le Scao, Teven  and\n      Sanh, Victor  and\n      Xu, Canwen  and\n      Patry, Nicolas  and\n      McMillan-Major, Angelina  and\n      Schmid, Philipp  and\n      Gugger, Sylvain  and\n      Delangue, Cl{\\'e}ment  and\n      Matussi{\\`e}re, Th{\\'e}o  and\n      Debut, Lysandre  and\n      Bekman, Stas  and\n      Cistac, Pierric  and\n      Goehringer, Thibault  and\n      Mustar, Victor  and\n      Lagunas, Fran{\\c{c}}ois  and\n      Rush, Alexander  and\n      Wolf, Thomas\",\n    booktitle = \"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations\",\n    month = nov,\n    year = \"2021\",\n    address = \"Online and Punta Cana, Dominican Republic\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2021.emnlp-demo.21\",\n    pages = \"175--184\",\n    abstract = \"The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks. Datasets is a community library for contemporary NLP designed to support this ecosystem. Datasets aims to standardize end-user interfaces, versioning, and documentation, while providing a lightweight front-end that behaves similarly for small datasets as for internet-scale corpora. The design of the library incorporates a distributed, community-driven approach to adding datasets and documenting usage. After a year of development, the library now includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects and shared tasks. The library is available at https://github.com/huggingface/datasets.\",\n    eprint={2109.02846},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL},\n}\n```\n\nIf you need to cite a specific version of our 🤗 Datasets library for reproducibility, you can use the corresponding version Zenodo DOI from this [list](https://zenodo.org/search?q=conceptrecid:%224817768%22\u0026sort=-version\u0026all_versions=True).\n","funding_links":[],"categories":["Python","📂 Benchmarks \u0026 Datasets","🌟 Awesome Lists and Resource Hubs","Popular Libraries","HarmonyOS","Data Manipulation","🤗 Official Libraries","其他_机器学习与深度学习","Dataset Tools","General","数据读写与提取","Datasets","tensorflow","Data Loading \u0026 Extraction","Natural Language Processing (NLP)","📚 Skill Development \u0026 Career"],"sub_categories":["Windows Manager","Data Sources \u0026 Datasets"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fdatasets","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Fdatasets","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fdatasets/lists"}