{"id":25126104,"url":"https://github.com/stephantul/wordninja2","last_synced_at":"2025-10-12T17:25:24.182Z","repository":{"id":254672025,"uuid":"564318676","full_name":"stephantul/wordninja2","owner":"stephantul","description":null,"archived":false,"fork":false,"pushed_at":"2024-08-25T19:36:06.000Z","size":1134,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-08T09:17:57.193Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stephantul.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-10T13:12:59.000Z","updated_at":"2024-09-26T11:10:14.000Z","dependencies_parsed_at":"2024-08-25T08:48:21.811Z","dependency_job_id":null,"html_url":"https://github.com/stephantul/wordninja2","commit_stats":null,"previous_names":["stephantul/wordninja2"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephantul%2Fwordninja2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephantul%2Fwordninja2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephantul%2Fwordninja2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephantul%2Fwordninja2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stephantul","download_url":"https://codeload.github.com/stephantul/wordninja2/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246871264,"owners_count":20847454,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-08T09:17:59.078Z","updated_at":"2025-10-12T17:25:19.142Z","avatar_url":"https://github.com/stephantul.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# wordninja2\n\nwordninja2 is a faster version of [wordninja](https://github.com/keredson/wordninja). Wordninja is a word-based unigram LM that splits strings that contain words without spaces into words, as follows:\n\n```python\n\u003e\u003e\u003e from wordninja2 import split\n\u003e\u003e\u003e split(\"waldorfastorianewyork\")\n['waldorf', 'astoria', 'new', 'york']\n\u003e\u003e\u003e split(\"besthotelpricesyoucanfind\")\n['best', 'hotel', 'prices', 'you', 'can', 'find']\n\n```\n\nWordninja was originally defined in [a stackoverflow thread](https://stackoverflow.com/questions/8870261/how-to-split-text-without-spaces-into-list-of-words/11642687#11642687), and then rewritten into a Python package.\n\nAs the original wordninja isn't really maintained, and contains some inconsistencies, I decided to rewrite it. See below for a comparison between wordninja and wordninja2.\n\n## Usage\n\nwordninja2 is packaged with a wordlist, which allows you to use it out of the box. To facilitate migrating from wordninja to wordninja2, we use the exact same wordlist.\n\n```python\n\u003e\u003e\u003e from wordninja2 import split\n\u003e\u003e\u003e split(\"HelloIfoundanewhousewiththreebedroomswouldwebeabletoshareit?\")\n['Hello',\n 'I',\n 'found',\n 'a',\n 'new',\n 'house',\n 'with',\n 'three',\n 'bedrooms',\n 'would',\n 'we',\n 'be',\n 'able',\n 'to',\n 'share',\n 'it',\n '?']\n\n ```\n\nUsing wordninja2 with your own wordlist is easy, and works regardless of punctuation in tokens or the languages of those tokens.\n\n```python\n\u003e\u003e\u003e from wordninja2 import WordNinja\n\u003e\u003e\u003e my_words = [\"dog\", \"cat\", \"房子\"]\n\u003e\u003e\u003e wn = WordNinja(my_words)\n\u003e\u003e\u003e wn.split(\"idogcat房子house\")\n[\"i\", \"dog\", \"cat\", \"房子\", \"h\", \"o\", \"u\", \"s\", \"e\"]\n\n```\n\nNote that any wordlist you supply should be in descending order of importance. That is, wordninja assumes that words higher in the list should get precedence in segmentation over words that are lower in the list. The example that follows shows what happens.\n\n```python\n\u003e\u003e\u003e from wordninja2 import WordNinja\n\u003e\u003e\u003e my_words = [\"dog\", \"s\", \"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"dogs\"]\n\u003e\u003e\u003e wn = WordNinja(my_words)\n\u003e\u003e\u003e wn.split(\"dogs\")\n[\"dog\", \"s\"]\n\n\u003e\u003e\u003e my_words = [\"dogs\", \"dog\", \"s\"]\n\u003e\u003e\u003e wn = WordNinja(my_words)\n\u003e\u003e\u003e wn.split(\"dogsdog\")\n[\"dogs\", \"dog\"]\n\n```\n\n### Wordfreq integration\n\nIf you want multilingual wordlists, or a better English wordlist, you can download [wordfreq](https://pypi.org/project/wordfreq/), by the great [rspeer](https://github.com/rspeer) (go give it a star on [Github](https://github.com/rspeer/wordfreq)). This works as follows:\n\n```python\n\u003e\u003e\u003e from wordfreq import top_n_list\n\u003e\u003e\u003e from wordninja2 import WordNinja\n\u003e\u003e\u003e wordlist = top_n_list(\"de\", 500_000)\n\u003e\u003e\u003e print(wordlist[:10])\n['die', 'der', 'und', 'in', 'das', 'ich', 'ist', 'nicht', 'zu', 'den']\n\n\u003e\u003e\u003e wn = WordNinja(wordlist)\n\u003e\u003e\u003e wn.split(\"erinteressiertsichfüralles,aberbesondersfürschmetterlingeundandereinsekten\")\n['er',\n 'interessiert',\n 'sich',\n 'für',\n 'alles',\n ',',\n 'aber',\n 'besonders',\n 'für',\n 'schmetterlinge',\n 'und',\n 'andere',\n 'insekten']\n\n```\n\nOne interesting avenue is that you could segment strings using different languages, and take the best one.\n\n```python\n\u003e\u003e\u003e from wordfreq import top_n_list\n\u003e\u003e\u003e from wordninja2 import WordNinja\n\u003e\u003e\u003e wns = {}\n\u003e\u003e\u003e for language in [\"de\", \"nl\", \"en\", \"fr\"]:\n        wordlist = top_n_list(language, 500_000)\n        wns[language] = WordNinja(wordlist)\n\n\u003e\u003e\u003e # This is a dutch string.\n\u003e\u003e\u003e string = \"ditiseennederlandsetekstmeteenheelmooiverhaalofmeerdereverhalen\"\n\u003e\u003e\u003e segmentations = {}\n\u003e\u003e\u003e for language, model in wns.items():\n        segmentation = model.split_with_cost(string)\n        segmentations[language] = segmentation\n\n\u003e\u003e\u003e for language, segmentation in sorted(segmentations.items(), key=lambda x: x[1].cost):\n        print(language)\n        print(segmentation.tokens)\nnl\n['dit', 'is', 'een', 'nederlandse', 'tekst', 'meteen', 'heel', 'mooi', 'verhaal', 'of', 'meerdere', 'verhalen']\nen\n['diti', 'seen', 'nederlandse', 'tekst', 'me', 'teen', 'heel', 'moo', 'iver', 'haal', 'of', 'meer', 'der', 'ever', 'halen']\nfr\n['dit', 'ise', 'en', 'nederlandse', 'tekst', 'me', 'te', 'en', 'heel', 'mooi', 'verh', 'aal', 'of', 'meer', 'de', 'rever', 'halen']\nde\n['dit', 'i', 'seen', 'nederlandse', 'tekst', 'me', 'teen', 'heel', 'mooi', 'verha', 'al', 'of', 'meer', 'der', 'ever', 'halen']\n\n```\n\n## Differences with wordninja\n\nIn this section I'll highlight some differences between `wordninja` and `wordninja2`.\n\n### Consistency\n\nThe original `wordninja` is not self-consistent, that is, the following assert fails.\n\n```python\nstring = \"this,string-split it\"\nassert \"\".join(split(string)) == string\n\n```\n\nThis is because `wordninja` removes all non-word characters from the string before processing it. This also has the consequence of `wordninja` never being able to detect words with these special characters in them.\n\n`wordninja2` is completely self-consistent, and does not remove any special characters from a string.\n\n### Speed\n\n`wordninja2` is twice as fast than `wordninja`. Segmenting the entire text of Mary Shelley's Frankenstein (which you can download [here](https://www.gutenberg.org/ebooks/84)):\n\n```python\n\u003e\u003e\u003e import re\n\n\u003e\u003e\u003e from wordninja2 import split\n\u003e\u003e\u003e from wordninja import split as old_split\n\n\u003e\u003e\u003e # Remove all spaces.\n\u003e\u003e\u003e txt = re.sub(r\"\\s\", \"\", open(\"pg84.txt\").read())\n\u003e\u003e\u003e %timeit split(txt)\n299 ms ± 4.32 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n\u003e\u003e\u003e %timeit old_split(txt)\n1.89 s ± 36.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n\n```\n\nThe original wordninja has an algorithm that backtracks up to the length of the longest word for each character in the string. Thus, if your wordlist has even a single long word, the entire algorithm will start taking a really long time. Coincidentally, the default wordlist used in `wordninja` has a really long word: `llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch`, see [here](https://www.atlasobscura.com/places/llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch) for additional background.\n\nTo avoid backtracking, `wordninja2` uses the [aho-corasick algorithm](https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm). We use a fast implementation in Rust: [aho-corasick](https://github.com/BurntSushi/aho-corasick), with python bindings: [aho-corasick-rs](https://github.com/G-Research/ahocorasick_rs/).\n\n## Dependencies\n\nSee the [pyproject.toml](pyproject.toml) file. We only rely on the aforementioned aho-corasick implementation and numpy.\n\n## Installation\n\nClone the repo and run `make install`. I might put this on `pypi` later.\n\n## Tests\n\n`wordninja2` has 100% test coverage, run `make test` to run the tests.\n\n## License\n\nMIT\n\n## Authors\n\n* Stéphan Tulkens\n* The original code is by [keredson](https://github.com/keredson)\n* The original algorithm was written by [Generic Human](https://stackoverflow.com/users/1515832/generic-human)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephantul%2Fwordninja2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstephantul%2Fwordninja2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephantul%2Fwordninja2/lists"}