{"id":13631067,"url":"https://github.com/winkjs/wink-nlp","last_synced_at":"2025-05-13T22:01:56.950Z","repository":{"id":33780885,"uuid":"161860407","full_name":"winkjs/wink-nlp","owner":"winkjs","description":"Developer friendly Natural Language Processing ✨","archived":false,"fork":false,"pushed_at":"2025-02-18T04:42:42.000Z","size":28128,"stargazers_count":1267,"open_issues_count":1,"forks_count":59,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-04-28T14:00:26.460Z","etag":null,"topics":["bm25","chatbot","custom-entity-detection","hacktoberfest","named-entity-extraction","natural-language-processing","negation-handling","ner","nlp","pattern-matching","pos-tagging","sbd","sentence-boundary-detection","sentiment-analysis","tokenize","vectorizer","visualization","wink","wink-nlp","word-vectors"],"latest_commit_sha":null,"homepage":"https://winkjs.org/wink-nlp/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/winkjs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-15T02:02:08.000Z","updated_at":"2025-04-24T08:23:56.000Z","dependencies_parsed_at":"2023-10-10T20:19:24.159Z","dependency_job_id":"62ef7e51-d9f5-4f14-9668-8a9fd6a1f139","html_url":"https://github.com/winkjs/wink-nlp","commit_stats":{"total_commits":216,"total_committers":7,"mean_commits":"30.857142857142858","dds":0.07870370370370372,"last_synced_commit":"5d913ab11b7abc6060d69b4f57652b282fb9314d"},"previous_names":[],"tags_count":42,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/winkjs%2Fwink-nlp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/winkjs%2Fwink-nlp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/winkjs%2Fwink-nlp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/winkjs%2Fwink-nlp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/winkjs","download_url":"https://codeload.github.com/winkjs/wink-nlp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254036807,"owners_count":22003651,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bm25","chatbot","custom-entity-detection","hacktoberfest","named-entity-extraction","natural-language-processing","negation-handling","ner","nlp","pattern-matching","pos-tagging","sbd","sentence-boundary-detection","sentiment-analysis","tokenize","vectorizer","visualization","wink","wink-nlp","word-vectors"],"created_at":"2024-08-01T22:02:09.296Z","updated_at":"2025-05-13T22:01:56.942Z","avatar_url":"https://github.com/winkjs.png","language":"JavaScript","funding_links":[],"categories":["JavaScript","⚙️ Backend \u0026 APIs"],"sub_categories":[],"readme":"# winkNLP\n\n### [![Build Status](https://github.com/winkjs/wink-nlp/actions/workflows/node.js.yml/badge.svg)](https://github.com/winkjs/wink-nlp/actions/workflows/node.js.yml/) [![Coverage Status](https://coveralls.io/repos/github/winkjs/wink-nlp/badge.svg?branch=master)](https://coveralls.io/github/winkjs/wink-nlp?branch=master) [![Known Vulnerabilities](https://snyk.io/test/github/winkjs/wink-nlp/badge.svg)](https://snyk.io/test/github/winkjs/wink-nlp) [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/6035/badge)](https://bestpractices.coreinfrastructure.org/projects/6035) [![Gitter](https://img.shields.io/gitter/room/nwjs/nw.js.svg)](https://gitter.im/winkjs/Lobby) [![Follow on Twitter](https://img.shields.io/twitter/follow/winkjs_org?style=social)](https://twitter.com/winkjs_org)\n\n## Developer friendly Natural Language Processing ✨\n[\u003cimg align=\"right\" src=\"https://decisively.github.io/wink-logos/logo-title.png\" width=\"100px\" \u003e](https://winkjs.org/)\n\nWinkNLP is a JavaScript library for Natural Language Processing (NLP). Designed specifically to make development of NLP applications **easier** and **faster**, winkNLP is optimized for the right balance of performance and accuracy. \n\nIts [word embedding](https://github.com/winkjs/wink-embeddings-sg-100d?tab=readme-ov-file#wink-embeddings-sg-100d) support unlocks deeper text analysis. Represent words and text as numerical vectors with ease, bringing higher accuracy in tasks like semantic similarity, text classification, and beyond – even within a **[browser](https://observablehq.com/@winkjs/how-to-use-word-embedding-with-winknlp)**.\n\nIt is built ground up with [no external dependency](https://snyk.io/test/github/winkjs/wink-nlp?tab=dependencies) and has a [lean code base of ~10Kb minified \u0026 gzipped](https://bundlephobia.com/package/wink-nlp). A test coverage of [~100%](https://coveralls.io/github/winkjs/wink-nlp?branch=master) and compliance with the [Open Source Security Foundation best practices](https://bestpractices.coreinfrastructure.org/en/projects/6035) make winkNLP the ideal tool for building production grade systems with confidence.\n\nWinkNLP with full [Typescript support](https://github.com/winkjs/wink-nlp/blob/master/types/index.d.ts), runs on Node.js, [web browsers](https://github.com/winkjs/wink-nlp#how-to-install-for-web-browser) and [Deno](https://github.com/winkjs/wink-nlp#how-to-run-on-deno).\n\n## Build amazing apps quickly\n| [Wikipedia article timeline](https://observablehq.com/@winkjs/how-to-visualize-timeline-of-a-wiki-article) | [Context aware word cloud](https://observablehq.com/@winkjs/how-to-create-a-context-aware-word-cloud) | [Key sentences detection](https://observablehq.com/@winkjs/how-to-visualize-key-sentences-in-a-document) |\n| --- | --- | --- |\n| [\u003cimg src=\"https://user-images.githubusercontent.com/29990/202497363-19c30578-8146-4f36-9c4b-4de613610837.png\"\u003e](https://observablehq.com/@winkjs/how-to-visualize-timeline-of-a-wiki-article)| [\u003cimg src=\"https://user-images.githubusercontent.com/29990/202506181-1a926ee0-788f-4aa1-aeac-a097f09fe747.png\"\u003e](https://observablehq.com/@winkjs/how-to-create-a-context-aware-word-cloud)|[\u003cimg src=\"https://user-images.githubusercontent.com/29990/202506490-7f999d12-8319-4969-b92b-0649559ffbe6.png\"\u003e](https://observablehq.com/@winkjs/how-to-visualize-key-sentences-in-a-document)|\n\nHead to  [live examples](https://winkjs.org/examples.html) to explore further.\n\n## Blazing fast\nWinkNLP can easily process large amount of raw text at speeds over **650,000 tokens/second**\u0026nbsp; on a M1 Macbook Pro in both browser and Node.js environments. It even runs smoothly on a low-end smartphone's browser.\n\n| Environment | Benchmarking Command |\n|--- | --- |\n| Node.js | [node benchmark/run](https://github.com/winkjs/wink-nlp/tree/master/benchmark) |\n| Browser | [How to measure winkNLP's speed on browsers?](https://observablehq.com/@winkjs/how-to-measure-winknlps-speed-on-browsers) |\n\n## Features\nWinkNLP has a [comprehensive natural language processing (NLP) pipeline](https://winkjs.org/wink-nlp/processing-pipeline.html) covering tokenization, sentence boundary detection (sbd), negation handling, sentiment analysis, part-of-speech (pos) tagging, named entity recognition (ner), custom entities recognition (cer). It offers a rich feature set:\n\n\u003ctable\u003e\n\u003ctr\u003e\u003ctd width=\"330px;\"\u003e🐎 Fast, lossless \u0026 multilingual tokenizer \u003c/td\u003e\u003ctd\u003eFor example, the multilingual text string \u003cb\u003e\u003ccode style=\"font-size: 0.9em\"\u003e\"¡Hola! नमस्कार! Hi! Bonjour chéri\"\u003c/code\u003e\u003c/b\u003e  is tokenized as \u003ccode style=\"font-size: 0.9em\"\u003e[\"¡\", \"Hola\", \"!\", \"नमस्कार\", \"!\", \"Hi\", \"!\", \"Bonjour\", \"chéri\"]\u003c/code\u003e.  The tokenizer processes text at a speed close to \u003cb\u003e4 million\u003c/b\u003e tokens/second on a M1 MBP's browser.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e✨ Developer friendly and intuitive \u003ca href=\"https://winkjs.org/wink-nlp/getting-started.html\"\u003eAPI\u003c/a\u003e\u003c/td\u003e\u003ctd\u003eWith winkNLP, process any text using a simple, declarative syntax; most \u003ca href=\"https://observablehq.com/@winkjs/how-to-build-a-naive-wikification-tool?collection=@winkjs/winknlp-recipes\"\u003elive examples\u003c/a\u003e have \u003cb\u003e30-40\u003c/b\u003e lines of code.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e🖼 Best-in-class \u003ca href=\"https://winkjs.org/wink-nlp/visualizing-markup.html\"\u003etext visualization\u003c/a\u003e\u003c/td\u003e\u003ctd\u003eProgrammatically \u003cb\u003e\u003ca href=\"https://winkjs.org/wink-nlp/markup.html\"\u003emark\u003c/a\u003e\u003c/b\u003e tokens, sentences, entities, etc. using HTML mark or any other tag of your choice.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e♻️ Extensive text processing features\u003c/td\u003e\u003ctd\u003eRemove and/or retain tokens with specific attributes such as part-of-speech, named entity type, token type, stop word, shape and many more; compute Flesch reading ease score; generate n-grams; normalize, lemmatise or stem. Checkout how with the right kind of text preprocessing, even \u003ca href=\"https://github.com/winkjs/wink-naive-bayes-text-classifier#readme\"\u003eNaive Bayes classifier\u003c/a\u003e achieves \u003cb\u003eimpressive (≥90%)\u003c/b\u003e accuracy in sentiment analysis and chatbot intent classification tasks.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e🔠 Pre-trained \u003ca href=\"https://winkjs.org/wink-nlp/language-models.html\"\u003elanguage models\u003c/a\u003e\u003c/td\u003e\u003ctd\u003eCompact sizes starting from \u003ca href=\"https://bundlephobia.com/package/wink-eng-lite-web-model\"\u003e~1MB (minified \u0026 gzipped)\u003c/a\u003e – reduce model loading time drastically down to ~1 second on a 4G network.\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e↗️ \u003ca href=\"https://github.com/winkjs/wink-embeddings-sg-100d?tab=readme-ov-file#wink-embeddings-sg-100d\"\u003eWord vectors\u003c/a\u003e\u003c/td\u003e\u003ctd\u003e100-dimensional English word embeddings for over 350K English words, which are optimized for winkNLP. Allows easy computation of sentence or document embeddings.\u003c/td\u003e\u003c/tr\u003e\n\u003c/table\u003e\n\n### Utilities \u0026 Tools 💼\n- [BM25 Vectorizer](https://winkjs.org/wink-nlp/bm25-vectorizer.html)\n- [Similarity methods](https://winkjs.org/wink-nlp/similarity.html) – Cosine, Tversky, Sørensen-Dice, Otsuka-Ochiai\n- [its \u0026 as helpers](https://winkjs.org/wink-nlp/its-as-helper.html) to get Bag of Words, Frequency table, Lemma, Stem, Stop word removal, Negation handling and many more.\n\n\n## Documentation\n- [Concepts](https://winkjs.org/wink-nlp/getting-started.html) — everything you need to know to get started.\n- [API Reference](https://winkjs.org/wink-nlp/read-doc.html) — explains usage of APIs with examples.\n- [Change log](https://github.com/winkjs/wink-nlp/blob/master/CHANGELOG.md) — version history along with the details of breaking changes, if any.\n- [Examples](https://winkjs.org/examples.html) — live examples with code to give you a head start.\n\n## Installation\n\nUse [npm](https://www.npmjs.com/package/wink-nlp) install:\n\n```shell\nnpm install wink-nlp --save\n```\n\nIn order to use winkNLP after its installation, you also need to install a language model according to the node version used. The table below outlines the version specific installation command:\n\n| Node.js Version |Installation |\n| --- | --- |\n| 16 or 18 | `npm install wink-eng-lite-web-model --save` |\n| 14 or 12 | `node -e \"require('wink-nlp/models/install')\"` |\n\nThe [wink-eng-lite-web-model](https://github.com/winkjs/wink-eng-lite-web-model) is designed to work with Node.js version 16 or 18. It can also work on browsers as described in the next section. This is the **recommended** model.\n\nThe second command installs the [wink-eng-lite-model](https://github.com/winkjs/wink-eng-lite-model), which works with Node.js version 14 or 12. \n\n### How to configure TypeScript project\n\nEnable [`esModuleInterop`](https://www.typescriptlang.org/tsconfig#esModuleInterop) and [`allowSyntheticDefaultImports`](https://www.typescriptlang.org/tsconfig#allowSyntheticDefaultImports) in the `tsconfig.json` file:\n\n```\n\"compilerOptions\": {\n    \"esModuleInterop\": true,\n    \"allowSyntheticDefaultImports\": true,\n    ...\n}\n```\n\n### How to install for Web Browser\nIf you’re using winkNLP in the browser use the [wink-eng-lite-web-model](https://www.npmjs.com/package/wink-eng-lite-web-model). Learn about its installation and usage in our [guide to using winkNLP in the browser](https://winkjs.org/wink-nlp/wink-nlp-in-browsers.html). Explore **[winkNLP recipes](https://observablehq.com/collection/@winkjs/winknlp-recipes)** on [Observable](https://observablehq.com/) for live browser based examples.\n\n### How to run on [Deno](https://deno.land/)\nFollow the [example on replit](https://replit.com/@sanjayaksaxena/wink-nlp-deno-example#index.ts).\n\n### Get started\nHere is the \"Hello World!\" of winkNLP:\n\n```javascript\n// Load wink-nlp package.\nconst winkNLP = require( 'wink-nlp' );\n// Load english language model.\nconst model = require( 'wink-eng-lite-web-model' );\n// Instantiate winkNLP.\nconst nlp = winkNLP( model );\n// Obtain \"its\" helper to extract item properties.\nconst its = nlp.its;\n// Obtain \"as\" reducer helper to reduce a collection.\nconst as = nlp.as;\n \n// NLP Code.\nconst text = 'Hello   World🌎! How are you?';\nconst doc = nlp.readDoc( text );\n \nconsole.log( doc.out() );\n// -\u003e Hello   World🌎! How are you?\n \nconsole.log( doc.sentences().out() );\n// -\u003e [ 'Hello   World🌎!', 'How are you?' ]\n \nconsole.log( doc.entities().out( its.detail ) );\n// -\u003e [ { value: '🌎', type: 'EMOJI' } ]\n \nconsole.log( doc.tokens().out() );\n// -\u003e [ 'Hello', 'World', '🌎', '!', 'How', 'are', 'you', '?' ]\n \nconsole.log( doc.tokens().out( its.type, as.freqTable ) );\n// -\u003e [ [ 'word', 5 ], [ 'punctuation', 2 ], [ 'emoji', 1 ] ]\n```\nExperiment with winkNLP on [RunKit](https://npm.runkit.com/wink-nlp).\n\n## Speed \u0026 Accuracy\nThe [winkNLP](https://winkjs.org/wink-nlp/) processes raw text at **~650,000 tokens per second** with its [wink-eng-lite-web-model](https://github.com/winkjs/wink-eng-lite-web-model), when [benchmarked](https://github.com/bestiejs/benchmark.js) using \"Ch 13 of Ulysses by James Joyce\" on a M1 Macbook Pro machine with 16GB RAM. The processing included the entire NLP pipeline — tokenization, sentence boundary detection, negation handling, sentiment analysis, part-of-speech tagging, and named entity extraction. This speed is way ahead of the prevailing speed benchmarks.\n\nThe benchmark was conducted on [Node.js versions 16, and 18](https://nodejs.org/en/about/releases/).\n\nIt pos tags a subset of WSJ corpus with an accuracy of **~95%** — this includes *tokenization of raw text prior to pos tagging*. The present state-of-the-art is at ~97% accuracy but at lower speeds and is generally computed using gold standard pre-tokenized corpus.\n\nIts general purpose sentiment analysis delivers a [f-score](https://en.wikipedia.org/wiki/F1_score) of **~84.5%**, when validated using Amazon Product Review [Sentiment Labelled Sentences Data Set](https://archive.ics.uci.edu/ml/machine-learning-databases/00331/) at [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/index.php). The current benchmark accuracy for **specifically trained** models can range around 95%.\n\n## Memory Requirement\nWink NLP delivers this performance with the minimal load on RAM. For example, it processes the entire [History of India Volume I](https://en.wikisource.org/wiki/History_of_India/Volume_1) with a total peak memory requirement of under **80MB**. The book has around 350 pages which translates to over 125,000 tokens.\n\n\n## Need Help?\n\n### Usage query 👩🏽‍💻\nPlease ask at [Stack Overflow](https://stackoverflow.com/) or discuss at [Wink JS GitHub Discussions](https://github.com/winkjs/wink-nlp/discussions) or chat with us at [Wink JS Gitter Lobby](https://gitter.im/winkjs/Lobby).\n\n### Bug report 🐛\nIf you spot a bug and the same has not yet been reported, raise a new [issue](https://github.com/winkjs/wink-nlp/issues) or consider fixing it and sending a PR.\n\n### New feature 🌟\nLooking for a new feature, request it via the [new features \u0026 ideas](https://github.com/winkjs/wink-nlp/discussions/categories/new-features-ideas) discussion forum  or consider becoming a [contributor](https://github.com/winkjs/wink-nlp/blob/master/CONTRIBUTING.md).\n\n\n## About winkJS\n[WinkJS](https://winkjs.org/) is a family of open source packages for **Natural Language Processing**, **Machine Learning**, and **Statistical Analysis** in NodeJS. The code is **thoroughly documented** for easy human comprehension and has a **test coverage of ~100%** for reliability to build production grade solutions.\n\n## Copyright \u0026 License\n\n**Wink NLP** is copyright 2017-25 [GRAYPE Systems Private Limited](https://graype.in/).\n\nIt is licensed under the terms of the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwinkjs%2Fwink-nlp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwinkjs%2Fwink-nlp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwinkjs%2Fwink-nlp/lists"}