{"id":13787082,"url":"https://github.com/commonsense/conceptnet-numberbatch","last_synced_at":"2025-10-07T17:59:14.234Z","repository":{"id":34948731,"uuid":"39023580","full_name":"commonsense/conceptnet-numberbatch","owner":"commonsense","description":null,"archived":false,"fork":false,"pushed_at":"2022-07-18T18:45:28.000Z","size":5743,"stargazers_count":1300,"open_issues_count":10,"forks_count":142,"subscribers_count":70,"default_branch":"master","last_synced_at":"2025-04-08T16:08:38.431Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":"Unmaintained","scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/commonsense.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-07-13T16:22:52.000Z","updated_at":"2025-04-07T15:25:49.000Z","dependencies_parsed_at":"2022-07-14T11:47:24.933Z","dependency_job_id":null,"html_url":"https://github.com/commonsense/conceptnet-numberbatch","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commonsense%2Fconceptnet-numberbatch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commonsense%2Fconceptnet-numberbatch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commonsense%2Fconceptnet-numberbatch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commonsense%2Fconceptnet-numberbatch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/commonsense","download_url":"https://codeload.github.com/commonsense/conceptnet-numberbatch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254485054,"owners_count":22078767,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T20:00:27.399Z","updated_at":"2025-10-07T17:59:09.215Z","avatar_url":"https://github.com/commonsense.png","language":"Python","funding_links":[],"categories":["**Tools, Libraries, Models**","Python","Language models","Urdu NLP Tools, Libraries and Models"],"sub_categories":["Word Embeddings","Word embeddings"],"readme":"![ConceptNet Numberbatch](conceptnet-numberbatch.png)\n\n\n## The best pre-computed word embeddings you can use\n\nConceptNet Numberbatch is a set of semantic vectors (also known as word embeddings)\nthat can be used directly as a representation of word meanings or as a starting point\nfor further machine learning.\n\nConceptNet Numberbatch is part of the [ConceptNet](http://conceptnet.io) open\ndata project. ConceptNet is a knowledge graph that provides lots of ways to\ncompute with word meanings, one of which is word embeddings, while ConceptNet\nNumberbatch is a snapshot of just the word embeddings.\n\nThese embeddings benefit from the fact that they have semi-structured, common\nsense knowledge from ConceptNet, giving them a way to learn about words that\nisn't _just_ observing them in context.\n\nNumberbatch is built using an ensemble that combines data from ConceptNet, word2vec,\nGloVe, and OpenSubtitles 2016, using a variation on retrofitting. It is\ndescribed in the paper [ConceptNet 5.5: An Open Multilingual Graph of General\nKnowledge][cn55-paper], presented at AAAI 2017.\n\nUnlike most embeddings, ConceptNet Numberbatch is **multilingual** from the\nground up.  Words in different languages share a common semantic space, and\nthat semantic space is informed by all of the languages.\n\n### Evaluation and publications\n\nConceptNet Numberbatch can be seen as a replacement for other precomputed\nembeddings, such as word2vec and GloVe, that do not include the graph-style\nknowledge in ConceptNet. Numberbatch outperforms these datasets on benchmarks\nof word similarity.\n\nConceptNet Numberbatch took first place in both subtasks at SemEval 2017 task\n2, \"[Multilingual and Cross-lingual Semantic Word Similarity][semeval17-2]\".\nWithin that task, it was also the first-place system in each of English,\nGerman, Italian, and Spanish. The result is described in our ACL 2017 SemEval\npaper, \"[Extending Word Embeddings with Multilingual Relational Knowledge][semeval-paper]\".\n\n[cn55-paper]: https://arxiv.org/abs/1612.03975\n[semeval17-2]: http://alt.qcri.org/semeval2017/task2/\n[semeval-paper]: https://arxiv.org/abs/1704.03560\n\nThe code and papers were created as a research project of [Luminoso\nTechnologies, Inc.][luminoso], by Robyn Speer, Joshua Chin, Catherine Havasi, and\nJoanna Lowry-Duda.\n\n![Graph of performance on English evaluations](eval-graph.png)\n\n### Now with more fairness\n\nWord embeddings are prone to learn human-like stereotypes and prejudices.\nConceptNet Numberbatch 17.04 and later counteract this as part of the build\nprocess, leading to word vectors that are less prejudiced than competitors such\nas word2vec and GloVe. See [our blog post on reducing\nbias](https://blog.conceptnet.io/2017/04/24/conceptnet-numberbatch-17-04-better-less-stereotyped-word-vectors/).\n\n![Graph of biases](bias-graph.png)\n\nA paper by Chris Sweeney and Maryam Najafian, [\"A Transparent Framework for\nEvaluating Unintended Demographic Bias in Word Embeddings\"][sweeney-paper],\nindependently evaluates bias in precomputed word embeddings, and finds that\nConceptNet Numberbatch is less likely than GloVe to inherently lead to\ndemographic discrimination.\n\n[sweeney-paper]: (https://www.aclweb.org/anthology/P19-1162)\n\n## Code\n\nSince 2016, the code for building ConceptNet Numberbatch is part of the [ConceptNet\ncode base][conceptnet5], in the `conceptnet5.vectors` package.\n\nThe only code contained in _this_ repository is `text_to_uri.py`, which\nnormalizes natural-language text into the ConceptNet URI representation,\nallowing you to look up rows in these tables without requiring the entire\nConceptNet codebase. For all other purposes, please refer to the [ConceptNet\ncode][conceptnet5].\n\n[conceptnet5]: https://github.com/commonsense/conceptnet5\n\n\n## Out-of-vocabulary strategy\n\nConceptNet Numberbatch is evaluated with an out-of-vocabulary strategy that\nhelps its performance in the presence of unfamiliar words. The strategy is\nimplemented in the [ConceptNet code base][conceptnet5]. It can be summarized\nas follows:\n\n- Given an unknown word whose language is not English, try looking up the\n  equivalently-spelled word in the English embeddings (because English words\n  tend to end up in text of all languages).\n- Given an unknown word, remove a letter from the end, and see if that is\n  a prefix of known words. If so, average the embeddings of those known words.\n- If the prefix is still unknown, continue removing letters from the end until\n  a known prefix is found. Give up when a single character remains.\n\n\n## Downloads\n\n[ConceptNet Numberbatch 19.08][nb1908-main] is the current recommended download.\n\nThis table lists the downloads and formats available for multiple recent versions:\n\n| Version  | Multilingual                            | English-only                              | HDF5                         |\n| -------- | --------------------------------------- | ----------------------------------------- | ---------------------------- |\n| **19.08**| [numberbatch-19.08.txt.gz][nb1908-main] | [numberbatch-en-19.08.txt.gz][nb1908-en]  | [19.08/mini.h5][nb1908-mini] |\n| 17.06    | [numberbatch-17.06.txt.gz][nb1706-main] | [numberbatch-en-17.06.txt.gz][nb1706-en]  | [17.06/mini.h5][nb1706-mini] |\n| 17.04    | [numberbatch-17.04.txt.gz][nb1704-main] | [numberbatch-en-17.04b.txt.gz][nb1704-en] | [17.05/mini.h5][nb1704-mini] |\n| 17.02    | [numberbatch-17.02.txt.gz][nb1704-main] | [numberbatch-en-17.02.txt.gz][nb1702-en]  |                              |\n| 16.09    |                                         |                                           | [16.09/numberbatch.h5][nb1609-h5] |\n\nThe 16.09 version was the version published at AAAI 2017. You can reproduce its results using a Docker snapshot of the conceptnet5 repository.\nSee the instructions on the [ConceptNet wiki](https://github.com/commonsense/conceptnet5/wiki/Running-your-own-copy#reproducing-the-word-embedding-evaluation).\n\n[nb1908-main]: https://conceptnet.s3.amazonaws.com/downloads/2019/numberbatch/numberbatch-19.08.txt.gz\n[nb1908-en]: https://conceptnet.s3.amazonaws.com/downloads/2019/numberbatch/numberbatch-en-19.08.txt.gz\n[nb1908-mini]: http://conceptnet.s3.amazonaws.com/precomputed-data/2016/numberbatch/19.08/mini.h5\n\n[nb1706-main]: https://conceptnet.s3.amazonaws.com/downloads/2017/numberbatch/numberbatch-17.06.txt.gz\n[nb1706-en]: https://conceptnet.s3.amazonaws.com/downloads/2017/numberbatch/numberbatch-en-17.06.txt.gz\n[nb1706-mini]: http://conceptnet.s3.amazonaws.com/precomputed-data/2016/numberbatch/17.06/mini.h5\n\n[nb1704-main]: https://conceptnet.s3.amazonaws.com/downloads/2017/numberbatch/numberbatch-17.04.txt.gz\n[nb1704-en]: https://conceptnet.s3.amazonaws.com/downloads/2017/numberbatch/numberbatch-en-17.04b.txt.gz\n[nb1704-mini]: http://conceptnet.s3.amazonaws.com/precomputed-data/2016/numberbatch/17.05/mini.h5\n\n[nb1702-main]: http://conceptnet.s3.amazonaws.com/downloads/2017/numberbatch/numberbatch-17.02.txt.gz\n[nb1702-en]: http://conceptnet.s3.amazonaws.com/downloads/2017/numberbatch/numberbatch-en-17.02.txt.gz\n\n[nb1609-h5]: http://conceptnet.s3.amazonaws.com/precomputed-data/2016/numberbatch/16.09/numberbatch.h5\n\n\nThe .txt.gz files of term vectors are in the text format used by word2vec, GloVe, and fastText.\n\nThe first line of the file contains the dimensions of the matrix:\n\n    9161912 300\n\nEach line contains a term label followed by 300 floating-point numbers,\nseparated by spaces:\n\n    /c/en/absolute_value -0.0847 -0.1316 -0.0800 -0.0708 -0.2514 -0.1687 -...\n    /c/en/absolute_zero 0.0056 -0.0051 0.0332 -0.1525 -0.0955 -0.0902 0.07...\n    /c/en/absoluteless 0.2740 0.0718 0.1548 0.1118 -0.1669 -0.0216 -0.0508...\n    /c/en/absolutely 0.0065 -0.1813 0.0335 0.0991 -0.1123 0.0060 -0.0009 0...\n    /c/en/absolutely_convergent 0.3752 0.1087 -0.1299 -0.0796 -0.2753 -0.1...\n\nThe HDF5 files are the format that ConceptNet uses internally. They are data\ntables that can be loaded into Python using a library such as `pandas` or\n`pytables`.\n\nThe \"mini.h5\" files trade off a little bit of accuracy for a lot of\nmemory savings, taking up less than 150 MB in RAM, and are used to power the\n[ConceptNet API](https://github.com/commonsense/conceptnet5/wiki/API).\n\n\n## License and attribution\n\nThese vectors are distributed under the [CC-By-SA 4.0][cc-by-sa] license. In\nshort, if you distribute a transformed or modified version of these vectors,\nyou must release them under a compatible Share-Alike license and give due\ncredit to [Luminoso][luminoso].\n\nSome suggested text:\n\n    This data contains semantic vectors from ConceptNet Numberbatch, by\n    Luminoso Technologies, Inc. You may redistribute or modify the\n    data under the terms of the CC-By-SA 4.0 license.\n\n[cc-by-sa]: https://creativecommons.org/licenses/by-sa/4.0/\n[luminoso]: http://luminoso.com\n\nIf you build on this data, you should cite it. Here is a straightforward citation:\n\n\u003e Robyn Speer, Joshua Chin, and Catherine Havasi (2017). \"ConceptNet 5.5: An Open Multilingual Graph of General Knowledge.\" In proceedings of AAAI 2017.\n\nIn BibTeX form, the citation is:\n\n    @inproceedings{speer2017conceptnet,\n        title = {{ConceptNet} 5.5: An Open Multilingual Graph of General Knowledge},\n        url = {http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14972},\n        author = {Speer, Robyn and Chin, Joshua and Havasi, Catherine},\n        year = {2017},\n        pages = {4444--4451}\n    }\n\nThis data is itself built on:\n\n  - [ConceptNet 5.7][conceptnet], which contains data from Wiktionary,\n    WordNet, and many contributors to Open Mind Common Sense projects,\n    edited by Robyn Speer\n\n  - [GloVe][glove], by Jeffrey Pennington, Richard Socher, and Christopher\n    Manning\n\n  - [word2vec][], by Tomas Mikolov and Google Research\n\n  - Parallel text from [OpenSubtitles 2016][opensubtitles], by Pierre Lison\n    and Jörg Tiedemann, analyzed using [fastText][], by Piotr Bojanowski,\n    Edouard Grave, Armand Joulin, and Tomas Mikolov\n\n[conceptnet]: http://conceptnet.io/\n[glove]: http://nlp.stanford.edu/projects/glove/\n[word2vec]: https://code.google.com/archive/p/word2vec/\n[opensubtitles]: http://opus.lingfil.uu.se/OpenSubtitles2016.php\n[fastText]: https://github.com/facebookresearch/fastText\n\n\n## Language statistics\n\nThe multilingual data in ConceptNet Numberbatch represents 78 different language\ncodes, though some have vocabularies with much more coverage than others. The following\ntable lists the languages and their vocabulary size.\n\nYou may notice a focus on even the smaller and historical languages of Europe,\nand under-representation of widely-spoken languages from outside Europe, which\nis an effect of the availability of linguistic resources for these languages.\nWe would like to change this, but it requires finding good source data for\nConceptNet in these under-represented languages.\n\nBecause Numberbatch contains word forms, inflected languages end up with larger\nvocabularies.\n\nThese vocabulary sizes were updated for ConceptNet Numberbatch 19.08.\n\n| code | language                       | vocab size |\n|:-----|:-------------------------------|-----------:|\n|   fr | French                         |    1388686 |\n|   la | Latin                          |     855294 |\n|   es | Spanish                        |     651859 |\n|   de | German                         |     594456 |\n|   it | Italian                        |     557743 |\n|   en | English                        |     516782 |\n|   ru | Russian                        |     455325 |\n|   zh | Chinese                        |     307441 |\n|   fi | Finnish                        |     267307 |\n|   pt | Portuguese                     |     262904 |\n|   ja | Japanese                       |     256648 |\n|   nl | Dutch                          |     190221 |\n|   bg | Bulgarian                      |     178508 |\n|   sv | Swedish                        |     167321 |\n|   pl | Polish                         |     152949 |\n|   no | Norwegian Bokmål               |     105689 |\n|   eo | Esperanto                      |      96255 |\n|   th | Thai                           |      95342 |\n|   sl | Slovenian                      |      91134 |\n|   ms | Malay                          |      90554 |\n|   cs | Czech                          |      88613 |\n|   ca | Catalan                        |      87508 |\n|   ar | Arabic                         |      85325 |\n|   hu | Hungarian                      |      74384 |\n|   se | Northern Sami                  |      67601 |\n|   sh | Serbian                        |      66746 |\n|   el | Greek                          |      65905 |\n|   gl | Galician                       |      59006 |\n|   da | Danish                         |      57119 |\n|   fa | Persian                        |      53984 |\n|   ro | Romanian                       |      51437 |\n|   tr | Turkish                        |      51308 |\n|   is | Icelandic                      |      48639 |\n|   eu | Basque                         |      44151 |\n|   ko | Korean                         |      42106 |\n|   vi | Vietnamese                     |      39802 |\n|   ga | Irish                          |      36988 |\n|  grc | Ancient Greek                  |      36977 |\n|   uk | Ukrainian                      |      36851 |\n|   lv | Latvian                        |      36333 |\n|   he | Hebrew                         |      33435 |\n|   mk | Macedonian                     |      33370 |\n|   ka | Georgian                       |      32338 |\n|   hy | Armenian                       |      29844 |\n|   sk | Slovak                         |      29376 |\n|   lt | Lithuanian                     |      28826 |\n|  ast | Asturian                       |      28401 |\n|   mg | Malagasy                       |      26865 |\n|   et | Estonian                       |      26525 |\n|   oc | Occitan                        |      26095 |\n|  fil | Filipino                       |      25088 |\n|   io | Ido                            |      25004 |\n|  hsb | Upper Sorbian                  |      24852 |\n|   hi | Hindi                          |      23538 |\n|   te | Telugu                         |      22173 |\n|   be | Belarusian                     |      22117 |\n|  fro | Old French                     |      21249 |\n|   sq | Albanian                       |      20493 |\n|  mul | (Multilingual, such as emoji)  |      19376 |\n|   cy | Welsh                          |      18721 |\n|  xcl | Classical Armenian             |      18420 |\n|   az | Azerbaijani                    |      17184 |\n|   kk | Kazakh                         |      16979 |\n|   gd | Scottish Gaelic                |      16827 |\n|   af | Afrikaans                      |      16132 |\n|   fo | Faroese                        |      15973 |\n|  ang | Old English                    |      15700 |\n|   ku | Kurdish                        |      13804 |\n|   vo | Volapük                        |      12731 |\n|   ta | Tamil                          |      12690 |\n|   ur | Urdu                           |      12006 |\n|   sw | Swahili                        |      11150 |\n|   sa | Sanskrit                       |      11081 |\n|  nrf | Norman French                  |      10048 |\n|  non | Old Norse                      |       8536 |\n|   gv | Manx                           |       8425 |\n|   nv | Navajo                         |       8232 |\n|  rup | Aromanian                      |       5107 |\n\n\n## Referred here from an old version?\n\nAn unpublished paper of ours described the \"ConceptNet Vector Ensemble\", and refers to\na repository that now redirects here, and an attached store of data that is no\nlonger hosted. We apologize, but we're not supporting the unpublished paper.\nPlease use a newer version and use the currently supported\n[ConceptNet build process](https://github.com/commonsense/conceptnet5/wiki/Build-process).\n\n\n## Image credit\n\nThe otter logo was designed by [Christy\nPresler](https://thenounproject.com/cnpresler/) for The Noun Project, and is\nused under a Creative Commons Attribution license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcommonsense%2Fconceptnet-numberbatch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcommonsense%2Fconceptnet-numberbatch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcommonsense%2Fconceptnet-numberbatch/lists"}