{"id":18147343,"url":"https://github.com/huggingface/cosmopedia","last_synced_at":"2025-10-14T15:32:53.342Z","repository":{"id":223552201,"uuid":"759760559","full_name":"huggingface/cosmopedia","owner":"huggingface","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-20T10:42:28.000Z","size":11998,"stargazers_count":541,"open_issues_count":16,"forks_count":50,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-09-30T18:02:26.743Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-19T09:34:19.000Z","updated_at":"2025-09-22T13:44:07.000Z","dependencies_parsed_at":"2024-05-06T12:59:24.010Z","dependency_job_id":"5a454e19-b287-4941-a876-5bd27aa15647","html_url":"https://github.com/huggingface/cosmopedia","commit_stats":null,"previous_names":["huggingface/cosmopedia"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/huggingface/cosmopedia","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fcosmopedia","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fcosmopedia/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fcosmopedia/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fcosmopedia/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/cosmopedia/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fcosmopedia/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279019322,"owners_count":26086711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-01T22:06:15.096Z","updated_at":"2025-10-14T15:32:53.324Z","avatar_url":"https://github.com/huggingface.png","language":"Python","funding_links":[],"categories":["Projects \u0026 Blogs","Python"],"sub_categories":[],"readme":"# Cosmopedia\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./plots/cover.png\" alt=\"Description of Image\" width=\"500\" height=\"300\"\u003e\n    \u003cp\u003eImage generated by DALL-E, the \u003ca href=\"https://huggingface.co/datasets/HuggingFaceTB/miscellaneous/blob/main/cosmopedia_dalle_prompt_by_mixtral.txt\"\u003eprompt\u003c/a\u003e was generated by Mixtral-8x7B-Instruct-v0.1.\u003c/p\u003e\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\u003ca href=\"https://huggingface.co/datasets/HuggingFaceTB/cosmopedia\"\u003e[🤗 Cosmopedia dataset]\u003c/a\u003e | \u003ca href=\"https://huggingface.co/HuggingFaceTB/cosmopedian-1b\"\u003e[🤖 1B-LLM trained on Cosmopedia]\u003c/a\u003e  | \u003ca href=\"https://huggingface.co/blog/cosmopedia\"\u003e[📰 Blog post]\u003c/a\u003e \n\u003c/p\u003e\n blog post: \n\u003chr\u003e\n\n## Description\nHere you can find the code used for creating [Cosmopedia](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia), a dataset of synthetic textbooks, blogposts, stories, posts and WikiHow articles generated by Mixtral-8x7B-Instruct-v0.1. It contains over **30 million files and 25 billion tokens**, making it the largest open synthetic dataset to date.\n\nCosmopedia covers a variety of topics; we tried to map world knowledge present in Web datasets like RefinedWeb and RedPajama, and generate synthetic content that covers them. This is the v0.1 of Cosmopedia, with ample room for improvement and topics to be more comprehensively covered. We hope this dataset will help the community's research efforts in the increasingly intriguing domain of synthetic data.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./plots/clusters_map.png\" alt=\"clusters\" width=\"900\" height=\"600\"\u003e\n    \u003cp\u003eThe clusters of Cosmopedia.\u003c/p\u003e\n\u003c/div\u003e\n\nYou can also find a files frequency plot of single topic clusters in `plots/topic_distpng.png`.\n\n## Code structure\n- `prompts`: the code for building the prompts in each `seed_data` in Cosmopedia. In `web_samples`, you can also find pointers for the topic clustering we did.\n- `generation`: the code to run large scale synthetic generations with [llm-swarm](https://github.com/huggingface/llm-swarm) using the prompts you built. Cosmopedia consists of 25B tokens and was generated in \u003e 10k H100 GPU hours.\n- `deduplication`: the script we used to run MinHash deduplication with [datatrove](https://github.com/huggingface/datatrove).\n- `decontamination`: the code we used to run n-gram decontamination against evaluation benchmarks, when training models on the dataset like [cosmopedian-1b](https://huggingface.co/HuggingFaceTB/cosmopedian-1b). \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fcosmopedia","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Fcosmopedia","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fcosmopedia/lists"}