{"id":13767597,"url":"https://github.com/coosto/dutch-word-embeddings","last_synced_at":"2026-05-15T21:21:08.093Z","repository":{"id":33108539,"uuid":"139824080","full_name":"coosto/dutch-word-embeddings","owner":"coosto","description":"Dutch word embeddings, trained on a large collection of Dutch social media messages and news/blog/forum posts.","archived":false,"fork":false,"pushed_at":"2022-02-22T11:15:27.000Z","size":14,"stargazers_count":44,"open_issues_count":1,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-11T21:33:04.153Z","etag":null,"topics":["coosto","dutch","nlp","word2vec","word2vec-model","wordembeddings"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coosto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-07-05T09:08:58.000Z","updated_at":"2024-10-12T20:41:55.000Z","dependencies_parsed_at":"2022-08-07T20:00:32.782Z","dependency_job_id":null,"html_url":"https://github.com/coosto/dutch-word-embeddings","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coosto%2Fdutch-word-embeddings","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coosto%2Fdutch-word-embeddings/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coosto%2Fdutch-word-embeddings/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coosto%2Fdutch-word-embeddings/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coosto","download_url":"https://codeload.github.com/coosto/dutch-word-embeddings/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253497296,"owners_count":21917683,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coosto","dutch","nlp","word2vec","word2vec-model","wordembeddings"],"created_at":"2024-08-03T16:01:10.137Z","updated_at":"2026-05-15T21:21:08.046Z","avatar_url":"https://github.com/coosto.png","language":"Python","funding_links":[],"categories":["Uncategorized"],"sub_categories":["Uncategorized"],"readme":"\u003ch1\u003e \u003ca href=\"https://www.coosto.com/en\"\u003e\u003cimg align=\"right\" src=\"https://www.coosto.com/themes/coosto/assets/images/misc/logo-coosto.png\" width=\"180px\"\u003e\u003c/a\u003e Dutch Word2Vec Model \u003c/h1\u003e\n\nThis repository contains a Word2Vec model trained on a large Dutch corpus, comprised of social media messages and posts from Dutch news, blog and fora. Finding pre-trained Dutch models online can often be quite difficult, especially since most online models are trained on neatly written texts like Wikipedia or newspaper archives. When working with noisy text sources these models usually underperform due to the large number of out-of-vocabulary words used on social platforms and their short-message writing style. By training on a combination of both large and short texts from multiple online sources we've tried to create a model more suited for these types of texts.\n\nWe are sharing this model to help research using Dutch data sources, so feel free to use it for your projects! If you would like a more up-to-date model, or a model with specific preprocessing steps, we'd be happy to help! Please contact the current maintainer ([*@Alexander Nieuwenhuijse*](https://github.com/severun)) if you'd like to use this model for commercial products.\n\n## Installation\nThe model can be downloaded using the provided utils script\n```sh\n$ git clone https://github.com/coosto/dutch-word-embeddings.git\n$ cd dutch-word2vec-model\n$ python3 utils.py download\n```\nOr directly as an asset from [release page](https://github.com/coosto/dutch-word-embeddings/releases).\n\n## Usage\nTo run a demo (using [gensim](https://github.com/RaRe-Technologies/gensim)) for the downloaded model run following command. It will first output some example analogies and afterward present an interactive prompt to query for nearest neighbour terms.\n```sh\n$ python3 utils.py demo --model model.bin\nLoading model...\nModel loaded\n...\n````\n\n## Examples\nIf we query for \"tomaat\" (Dutch for tomato) we get a lot of Dutch vegetables:  \n\u003cpre\u003e\nEnter word or sentence: tomaat\n     Term     |      Distance\n-----------------------------------\npaprika       |0.8452869653701782   (Bell pepper)\nkomkommer     |0.7932491898536682   (Cucumber)\ncourgette     |0.771128237247467    (Zucchini)\nspinazie      |0.7697550058364868   (Spinach)\naubergine     |0.7646535634994507   (Eggplant)\nrucola        |0.7631270885467529   (Arugula)\navocado       |0.7610437273979187   (Avocado)\nradijs        |0.7554484605789185   (Radish)\ntomaatjes     |0.7549760341644287   (Tomatoes)\ntomaten       |0.7525067925453186   (Tomatoes)\n\u003c/pre\u003e\n\nAnother interesting case is \"lidl\" (A supermarket chain), which returns a list of other supermarkets:\n\u003cpre\u003e\nEnter word or sentence: lidl\n      Term     |      Distance\n------------------------------------\naldi           |0.9053274989128113\njumbo          |0.7790680527687073\n\u003cb\u003ealbert_heijn   |0.7582876086235046\u003c/b\u003e\nsupermarkt     |0.7522222995758057\n#lidl          |0.7320866584777832\n\u003cb\u003ealbert_hein    |0.7161234617233276\u003c/b\u003e\n\u003cb\u003ealbert_heyn    |0.7076783180236816\u003c/b\u003e\nekoplaza       |0.6897568702697754\nvomar          |0.6830324530601501\nnettorama      |0.6739382147789001\n\u003c/pre\u003e\nThis example shows some common typographical errors for a supermarket chain called Albert Heijn. This type of errors would not be in the model is it was trained only on neatly written text, like the Dutch Wikipedia data or newspaper articles, but is included in this model because these errors are made a lot on social media.\n\n\u003chr\u003e\n\n# Data selection \u0026 Modeling\nThe model was trained using ~600 million individual messages, comprised of Dutch social media messages (624 million messages) and Dutch news, blog and fora posts (36 million messages). All messages were published between 01/01/2017 and 31/12/2017. To improve the quality of the model some basic preprocessing was applied to all the messages, described below.\n\n### Splitting sentences\nEvery individual message was split into separate sentences by searching for punctuation marks, which are only considered to be an end-of-line character if it is not used in the following exceptions:\n* (Roman) Numerals\n* Single letters\n* Abbriviations\n\n### Clean up\nGiven that social media messages are usually not neatly formatted some cleaning of the text is applied:\n1. Converting to lowercase\n2. Removing HTML/XML tags\n3. Replacing *URLs* with the *\\\u003curl\\\u003e* token\n4. Replacing *@-mentions* with the *\\\u003cmention\\\u003e* token\n5. Removing punctuation marks, emojis and unwanted unicode characters\n7. Removing sentences with less than 5 tokens.\n\nThe URLs and @-mentions are removed for both privacy and performance reasons. The last step is applied to remove very small text messages, since they usually do not provide enough relevant context to learn from.\n\n### Deduplication\nLastly the entire set of cleaned up sentences is removed from any duplicate training examples and shuffeled into a random order. This results in a training set of 490 million unique preprocessed sentences.\n\n### Training\nThe model was trained using [Google's Word2Vec implementation](https://code.google.com/p/word2vec/). We've selected the Continuous Bag-of-Words (CBOW) model and generate vectors of size 300. The min-count parameter was chosen based on manual inspection of the vocabulary and to limit the size of the model.\n```sh\nword2vec -train input.txt -output model.bin -size 300 -window 10 -negative 10 -hs 0 -cbow 1 -sample 1e-5 -iter 5 -min-count 300\n```\nThe resulting model contains 250479 vectors and was not pruned or altered in any way.\n\nLicense\n----\nThis work is licensed under a [Creative Commons Attribution-NonCommercial 4.0 International License ](https://creativecommons.org/licenses/by-nc/4.0/).\n\nPlease contact the current maintainer ([*@Alexander Nieuwenhuijse*](https://github.com/severun)) if you wish to use this model with a different license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoosto%2Fdutch-word-embeddings","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoosto%2Fdutch-word-embeddings","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoosto%2Fdutch-word-embeddings/lists"}