{"id":13676713,"url":"https://github.com/project-miracl/miracl","last_synced_at":"2026-01-17T10:37:08.042Z","repository":{"id":185191556,"uuid":"534334917","full_name":"project-miracl/miracl","owner":"project-miracl","description":"A large-scale multilingual dataset for Information Retrieval. Thorough human-annotations across 18 diverse languages.","archived":false,"fork":false,"pushed_at":"2024-07-31T21:17:17.000Z","size":109,"stargazers_count":183,"open_issues_count":2,"forks_count":4,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-29T07:39:47.227Z","etag":null,"topics":["benchmark","dataset","information-retrieval","multilingual"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/project-miracl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-08T18:04:27.000Z","updated_at":"2025-04-16T05:23:22.000Z","dependencies_parsed_at":"2024-01-14T14:49:25.857Z","dependency_job_id":"eb5f74d1-7ddb-424f-bb99-eec4bc652c83","html_url":"https://github.com/project-miracl/miracl","commit_stats":null,"previous_names":["project-miracl/miracl"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/project-miracl/miracl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-miracl%2Fmiracl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-miracl%2Fmiracl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-miracl%2Fmiracl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-miracl%2Fmiracl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/project-miracl","download_url":"https://codeload.github.com/project-miracl/miracl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-miracl%2Fmiracl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28506593,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T10:25:30.148Z","status":"ssl_error","status_checked_at":"2026-01-17T10:25:29.718Z","response_time":85,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","dataset","information-retrieval","multilingual"],"created_at":"2024-08-02T13:00:31.643Z","updated_at":"2026-01-17T10:37:03.022Z","avatar_url":"https://github.com/project-miracl.png","language":null,"funding_links":[],"categories":["Others","Benchmarks \u0026 Leaderboards"],"sub_categories":["Multimodal Embeddings"],"readme":"\u003c!-- \u003cp align=\"center\"\u003e \u003cspan\u003e\u003cfont size=\"32\"\u003e \u003cb\u003e MIRACL šŸŒšŸ™ŒšŸŒ \u003c/b\u003e\u003c/font\u003e\u003c/span\u003e\u003c/p\u003e --\u003e\n\u003ch1 align=\"center\"\u003e \n \u003cimg style=\"vertical-align:middle\" width=\"2000\" alt=\"image\" src=\"https://user-images.githubusercontent.com/31640436/210273474-7697ab15-c638-4513-8c80-12886941f520.png\"\u003e\n\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n \u003ca href=\"https://www.python.org/\"\u003e\n \u003cimg alt=\"Build\" src=\"https://img.shields.io/badge/Made%20with-Python-1f425f.svg?color=purple\"\u003e\n \u003c/a\u003e\n \u003ca href=\"https://github.com/beir-cellar/beir/blob/master/LICENSE\"\u003e\n \u003cimg alt=\"License\" src=\"https://img.shields.io/github/license/beir-cellar/beir.svg?color=green\"\u003e\n \u003c/a\u003e\n \u003ca href=\"https://github.com/beir-cellar/beir/\"\u003e\n \u003cimg alt=\"Downloads\" src=\"https://badges.frapsoft.com/os/v1/open-source.svg?v=103\"\u003e\n \u003c/a\u003e\n\u003c/p\u003e\n\n\u003ch4 align=\"center\"\u003e\n \u003cp\u003e\n \u003ca href=\"https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00595/117438/MIRACL-A-Multilingual-Retrieval-Dataset-Covering\"\u003ePaper\u003c/a\u003e |\n \u003ca href=\"#-baselines\"\u003eBaselines\u003c/a\u003e |\n \u003ca href=\"https://huggingface.co/datasets/miracl\"\u003eHuggingFace\u003c/a\u003e |\n \u003ca href=\"https://eval.ai/web/challenges/challenge-page/1881/leaderboard\"\u003eLeaderboard\u003c/a\u003e\n \u003c/p\u003e\n\u003c/h4\u003e\n\n\n\n## šŸ™Œ MIRACL\n\nMIRACL šŸŒšŸ™ŒšŸŒ (Multilingual Information Retrieval Across a Continuum of Languages) is an WSDM 2023 Cup challenge that focuses on search across 18 different languages, which collectively encompass over three billion native speakers around the world.\nThe website for the event can be found at [`miracl.ai`](http://miracl.ai).\nThis repo provides pointers to access the actual dataset.\n\nFor more details, check out our arXiv paper: [Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages](https://arxiv.org/abs/2210.09984).\n\nConnect with us!\n- šŸ“¬ [Mailing list](https://forms.gle/aCbjRQ9CPeXViWcaA)\n- šŸ’¬ [Slack Workspace](https://join.slack.com/t/miraclgroup/shared_invite/zt-1ph43qw40-gvoQI~jc_g2EPb_EgCF4Dg)\n- šŸ“£ [Twitter](https://twitter.com/project_miracl?s=21\u0026t=Qf9LrVerhhN1hsXs1gdWhw)\n\n## šŸ™Œ Corpora\n\nThe Wikipedia corpora used in MIRACL are available as a [HuggingFace Dataset](https://huggingface.co/datasets/miracl/miracl-corpus).\nSo far, we have released corpora for the 16 \"known languages\"; the remaining 2 \"surprise languages\" will be revealed later!\n\n+ šŸ¤— = direct link to HuggingFace Dataset\n+ šŸŒ = link to raw wiki dumps\n\n| Language | # of Passages | # of Articles | Links |\n|:----------------|--------------:|--------------:|:------|\n| Arabic (ar) | 2,061,414 | 656,982 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-ar) [šŸŒ](https://archive.org/download/arwiki-20190201/arwiki-20190201-pages-articles-multistream.xml.bz2)\n| Bengali (bn) | 297,265 | 63,762 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-bn) [šŸŒ](https://archive.org/download/bnwiki-20190201/bnwiki-20190201-pages-articles-multistream.xml.bz2)\n| English (en) | 32,893,221 | 5,758,285 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-en) [šŸŒ](https://archive.org/download/enwiki-20190201/enwiki-20190201-pages-articles-multistream.xml.bz2)\n| Spanish (es) | 10,373,953 | 1,669,181 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-es) [šŸŒ](https://archive.org/download/eswiki-20220301/eswiki-20220301-pages-articles-multistream.xml.bz2)\n| Persian (fa) | 2,207,172 | 857,827 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-fa) [šŸŒ](https://archive.org/download/fawiki-20220301/fawiki-20220301-pages-articles-multistream.xml.bz2)\n| Finnish (fi) | 1,883,509 | 447,815 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-fi) [šŸŒ](https://archive.org/download/fiwiki-20190201/fiwiki-20190201-pages-articles-multistream.xml.bz2)\n| French (fr) | 14,636,953 | 2,325,608 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-fr) [šŸŒ](https://archive.org/download/frwiki-20220301/frwiki-20220301-pages-articles-multistream.xml.bz2)\n| Hindi (hi) | 506,264 | 148,107 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-hi) [šŸŒ](https://archive.org/download/hiwiki-20220301/hiwiki-20220301-pages-articles-multistream.xml.bz2)\n| Indonesian (id) | 1,446,315 | 446,330 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-id) [šŸŒ](https://archive.org/download/idwiki-20190201/idwiki-20190201-pages-articles-multistream.xml.bz2)\n| Japanese (ja) | 6,953,614 | 1,133,444 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-ja) [šŸŒ](https://archive.org/download/jawiki-20190201/jawiki-20190201-pages-articles-multistream.xml.bz2)\n| Korean (ko) | 1,486,752 | 437,373 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-ko) [šŸŒ](https://archive.org/download/kowiki-20190201/kowiki-20190201-pages-articles-multistream.xml.bz2)\n| Russian (ru) | 9,543,918 | 1,476,045 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-ru) [šŸŒ](https://archive.org/download/ruwiki-20190201/ruwiki-20190201-pages-articles-multistream.xml.bz2)\n| Swahili (sw) | 131,924 | 47,793 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-sw) [šŸŒ](https://archive.org/download/swwiki-20190201/swwiki-20190201-pages-articles-multistream.xml.bz2)\n| Telugu (te) | 518,079 | 66,353 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-te) [šŸŒ](https://archive.org/download/tewiki-20190201/tewiki-20190201-pages-articles-multistream.xml.bz2)\n| Thai (th) | 542,166 | 128,179 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-th) [šŸŒ](https://archive.org/download/thwiki-20190101/thwiki-20190101-pages-articles-multistream.xml.bz2)\n| Chinese (zh) | 4,934,368 | 1,246,389 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl-corpus/tree/main/miracl-corpus-v1.0-zh) [šŸŒ](https://archive.org/download/zhwiki-20220301/zhwiki-20220301-pages-articles-multistream.xml.bz2)\n\nThe corpus for each language is prepared from a Wikipedia dump, where we keep only the plain text and discard images, tables, etc.\nEach article is segmented into multiple passages using [WikiExtractor](https://github.com/attardi/wikiextractor) based on natural discourse units (e.g., `\\n\\n` in the wiki markup).\nEach of these passages comprise a \"document\" or unit of retrieval.\nWe preserve the Wikipedia article title of each passage.\n\nThe corpus data files are in [JSON lines](https://jsonlines.org/) format, compressed with `gzip`.\nEach line in the file corresponds to a passage.\nConsider an example from the English corpus:\n\n```\n{\n \"docid\": \"39#0\",\n \"title\": \"Albedo\", \n \"text\": \"Albedo (meaning 'whiteness') is the measure of the diffuse reflection of solar radiation out of the total solar radiation received by an astronomical body (e.g. a planet like Earth). It is dimensionless and measured on a scale from 0 (corresponding to a black body that absorbs all incident radiation) to 1 (corresponding to a body that reflects all incident radiation).\"\n}\n```\n\nThe `docid` has the schema `X#Y`, where all passages with the same `X` come from the same Wikipedia article, whereas `Y` denotes the passage within that article, numbered sequentially.\nThe `text` field contains the text of the passage.\nThe `title` field contains the name of the article the passage comes from.\n\n## šŸ™Œ Topics and Relevance Judgments\n\nTopics (= queries) and relevance judgments (= relevance labels) of the MIRACL training sets and development sets for each of the 16 known languages are available on [HuggingFace Dataset](https://huggingface.co/datasets/miracl/miracl)! \n\nšŸ¤— = direct link to HuggingFace Dataset\n\n| | Train | | Dev | | |\n|:-----|------:|-------:|------:|-------:|:-----|\n| **Language** | **#Q**| **#J** |**#Q** |**#J** | **Links** |\n| Arabic (ar) | 3,495 | 25,382 | 2,896 | 29,197 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-ar) |\n| Bengali (bn) | 1,631 | 16,754 | 411 | 4,206 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-bn) |\n| English (en) | 2,863 | 29,416 | 799 | 8,350 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-en) |\n| Spanish (es) | 2,162 | 21,531 | 648 | 6,443 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-es) |\n| Persian (fa) | 2,107 | 21,844 | 632 | 6,571 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-fa) |\n| Finnish (fi) | 2,897 | 20,350 | 1,271 | 12,008 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-fi) |\n| French (fr) | 1,143 | 11,426 | 343 | 3,429 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-fr) |\n| Hindi (hi) | 1,169 | 11,668 | 350 | 3,494 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-hi) |\n| Indonesian (id) | 4,071 | 41,358 | 960 | 9,668 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-id) |\n| Japanese (ja) | 3,477 | 34,387 | 860 | 8,354 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-ja) |\n| Korean (ko) | 868 | 12,767 | 213 | 3,057 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-ko) |\n| Russian (ru) | 4,683 | 33,921 | 1,252 | 13,100 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-ru) |\n| Swahili (sw) | 1,901 | 9,359 | 482 | 5,092 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-sw) |\n| Telugu (te) | 3,452 | 18,608 | 828 | 1,606 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-te) |\n| Thai (th) | 2,972 | 21,293 | 733 | 7,573 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-th) |\n| Chinese (zh) | 1,312 | 13,113 | 393 | 3,928 | [šŸ¤—](https://huggingface.co/datasets/miracl/miracl/tree/main/miracl-v1.0-zh) |\n| **Total** | 40,203 | 343,177 | 13,071 | 126,076 |\n\nThe above table shows the number of queries (`#Q`) and the number of judgments (`#J`) in each (language, split) combination, where the judgments include both positive and negative labels.\n\nThe topics are formatted in TSV, with each line organized as follows:\n\n```\nqid\\tquery\n```\n\nThe relevance judgments are formatted in standard TREC qrels format, as follows:\n\n```\nqid Q0 docid relevance\n```\n\n## šŸ™Œ Baselines\n\n### Reproduce the results with Pyserini:\nWe have released baselines using BM25, mDPR, and hybrid of the two, as described in our [arXiv paper](https://arxiv.org/abs/2210.09984). \nReuslts of BM25 and mDPR could be reproduced using [Pyserini](https://github.com/castorini/pyserini).\n\nTo reproduce our baselines:\n\n1. Install the development version of Pyserini following [these instructions](https://github.com/castorini/pyserini/blob/master/docs/installation.md#development-installation). **(To run baselines on surprise languages, you'll need to re-build both Anserini and Pyserini)**\n2. Manually place all topics and qrels files under `tools/topics-and-qrels`. The topics and qrels files can be found under `miracl-v1.0-${lang}/topics` and `miracl-v1.0-${lang}/qrels` in the [HuggingFace dataset](https://huggingface.co/datasets/miracl/miracl). \n ```\n git clone https://huggingface.co/datasets/miracl/miracl\n mv miracl/*/*/* $PYSERINI_PATH/tools/topics-and-qrels/\n ```\n3. Following the commands in our [2-click-reproduction (2CR) website](https://castorini.github.io/pyserini/2cr/miracl.html).\n\nNote that the 2CR above is only for reproducing the search stage, where the indexes are pre-computed and loaded automatically by Pyserini. If you are interested in reproducing the indexing stage, please refer to [this](https://github.com/castorini/pyserini/blob/master/pyserini/resources/index-metadata/faiss.miracl-v1.0.20221004.2b2856.mdpr-tied-pft-msmarco-ft-all.README.md) documentation: \n\n### Checkpoints for dense models:\n* mDPR (w/o fine-tuning on MIRACL): `castorini/mdpr-tied-pft-msmarco`\n* mContriever (w/o fine-tuning on MIRACL): `facebook/mcontriever-msmarco`\n* mDPR (fine-tuned on MIRACL): `castorini/mdpr-tied-pft-msmarco-ft-miracl-{lang}`, where `{lang}` is the two-letter ISO code (e.g., `ar`, `bn`, ...)\n\n## šŸ™Œ Citation\nIf you find this dataset and repository helpful, please cite MIRACL as follows:\n```\n@article{10.1162/tacl_a_00595,\n author = {Zhang, Xinyu and Thakur, Nandan and Ogundepo, Odunayo and Kamalloo, Ehsan and Alfonso-Hermelo, David and Li, Xiaoguang and Liu, Qun and Rezagholizadeh, Mehdi and Lin, Jimmy},\n title = \"{MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages}\",\n journal = {Transactions of the Association for Computational Linguistics},\n volume = {11},\n pages = {1114-1131},\n year = {2023},\n month = {09},\n issn = {2307-387X},\n doi = {10.1162/tacl_a_00595},\n url = {https://doi.org/10.1162/tacl\\_a\\_00595},\n eprint = {https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl\\_a\\_00595/2157340/tacl\\_a\\_00595.pdf},\n}\n\n```\n\n## šŸ™Œ Contact\n\nIf you have any questions, feel free to email us (project.miracl [at] gmail.com) or start a Github issue under this repository. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproject-miracl%2Fmiracl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproject-miracl%2Fmiracl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproject-miracl%2Fmiracl/lists"}