{"id":22081578,"url":"https://github.com/vunb/vntk","last_synced_at":"2025-04-05T12:09:06.019Z","repository":{"id":57393608,"uuid":"67094470","full_name":"vunb/vntk","owner":"vunb","description":"Vietnamese NLP Toolkit for Node","archived":false,"fork":false,"pushed_at":"2024-02-26T07:26:52.000Z","size":3734,"stargazers_count":207,"open_issues_count":2,"forks_count":61,"subscribers_count":22,"default_branch":"master","last_synced_at":"2024-04-26T14:06:01.435Z","etag":null,"topics":["language-identification","named-entity-recognition","natural-language-processing","pos-tagging","tf-idf","vietnamese","vietnamese-nlp","vietnamese-text-classification","vietnamese-tokenizer"],"latest_commit_sha":null,"homepage":"https://gitter.im/vntk/Lobby","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vunb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-09-01T03:23:55.000Z","updated_at":"2024-06-21T14:18:22.124Z","dependencies_parsed_at":"2024-06-21T14:18:19.818Z","dependency_job_id":"35555bc9-26e2-4551-bca6-f2cb56bceb85","html_url":"https://github.com/vunb/vntk","commit_stats":{"total_commits":117,"total_committers":2,"mean_commits":58.5,"dds":0.02564102564102566,"last_synced_commit":"7f8376d20eb1d847e0d4bde68498bb925bf20b49"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vunb%2Fvntk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vunb%2Fvntk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vunb%2Fvntk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vunb%2Fvntk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vunb","download_url":"https://codeload.github.com/vunb/vntk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247332612,"owners_count":20921853,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["language-identification","named-entity-recognition","natural-language-processing","pos-tagging","tf-idf","vietnamese","vietnamese-nlp","vietnamese-text-classification","vietnamese-tokenizer"],"created_at":"2024-11-30T23:24:38.246Z","updated_at":"2025-04-05T12:09:06.000Z","avatar_url":"https://github.com/vunb.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VNTK\n\nVietnamese NLP Toolkit for Node\n\n[![Join the chat at https://gitter.im/vntk/Lobby](https://badges.gitter.im/vntk/Lobby.svg)](https://gitter.im/vntk/Lobby)\n[![npm version](https://img.shields.io/npm/v/vntk.svg?style=flat)](https://www.npmjs.com/package/vntk)\n[![npm downloads](https://img.shields.io/npm/dm/vntk.svg)](https://www.npmjs.com/package/vntk)\n[![Travis](https://travis-ci.org/vunb/vntk.svg?branch=master)](https://travis-ci.org/vunb/vntk)\n[![Appveyor](https://ci.appveyor.com/api/projects/status/3xej24k2f9qlm2v8/branch/master?svg=true)](https://ci.appveyor.com/project/vunb/vntk/branch/master)\n\n\n# Installation In A Nutshell\n\n1. Install [Node.js](http://nodejs.org/)\n2. Run: `$ npm install vntk --save`\n\nIf you are interested in contributing to **vntk**, or just hacking on it, then fork it away!\n\nJump to guide: [How to build an NLP API Server using Vntk](#nlp-api-server).\n\n# Documentation\n\n* [**CLI Utilities**](#cli-utilities)\n  * [1. Installation](#1-installation)\n  * [2. Usage Example](#2-usage-example)\n* [**API Usage**](#api-usage)\n  * [1. Tokenizer](#1-tokenizer)\n  * [2. Word Segmentation](#2-word-segmentation)\n  * [3. POS Tagging](#3-pos-tagging)\n  * [4. Chunking](#4-chunking)\n  * [5. Named Entity Recognition](#5-named-entity-recognition)\n    * [PER LOC ORG](#ner-per-loc-org)\n    * [Date time](#ner-date-time)\n    * [Custom NER](#ner-custom)\n  * [6. Utility](#6-utility)\n    * [Dictionary](#dictionary)\n    * [Clean html](#clean-html)\n  * [7. TF-IDF](#7-tf-idf)\n  * [8. Classifiers](#8-classifiers)\n    * [Naive Bayes](#bayes-classifier)\n    * [fastText](#fasttext-classifier)\n  * [9. Language identification](#9-language-identification)\n  * [10. CRFSuite](#10-crfsuite)\n* [**NLP API Server**](#nlp-api-server)\n* [**Contributing**](#contributing)\n* [**License**](#license)\n\n# CLI Utilities\n\n## 1. Installation\n\nVntk cli will install nice and easy with:\n\n\u003e npm install -g @vntk/cli\n\nThen you need to pay attention to how to use these cli utilities to preprocess text from files, especially vietnamese that **describe at the end of each apis usage**. If you wish to improve the tool, please fork and make it better [here](https://github.com/vntk/vntk-cli).\n\n## 2. Usage Example\n\nAfter the CLI has installed, you need to open your `Terminal` (or Command Prompt on Windows) and type command you need to use.\n\nFor instance, the following command will open a file and process it by using Word Tokenizer to tokenize each lines in the file.\n\n```bash\n# Process a text file or a folder\n$ vntk ws input.txt --output output.txt\n\n# Output file will contain lines which have tokenized.\n```\n\n# API Usage\n\n## 1. Tokenizer\n\n\u003e Regex Tokenizer using Regular Expression.  \n\u003e Tokenizer is provided to break text into arrays of tokens!\n\nExample:\n\n```js\nvar vntk = require('vntk');\nvar tokenizer = vntk.tokenizer();\n\nconsole.log(tokenizer.tokenize('Giá khuyến mãi: 140.000đ / kg  ==\u003e giảm được 20%'))\n// [ 'Giá', 'khuyến', 'mãi', ':', '140.000', 'đ', '/', 'kg', '==\u003e', 'giảm', 'được', '20', '%' ]\n\nconsole.log(tokenizer.stokenize('Giá khuyến mãi: 140.000đ / kg  ==\u003e giảm được 20%'))\n// Giá khuyến mãi : 140.000 đ / kg ==\u003e giảm được 20 %\n```\n\nCommand line: `vntk tok \u003cfile_name.txt\u003e`\n\n## 2. Word Segmentation\n\n\u003e Vietnamese Word Segmentation using Conditional Random Fields, called: `Word Tokenizer`.  \n\u003e Word Tokenizer helps break text into arrays of words!\n\n```js\nvar vntk = require('vntk');\nvar tokenizer = vntk.wordTokenizer();\n\nconsole.log(tokenizer.tag('Chào mừng các bạn trẻ tới thành phố Hà Nội'));\n// [ 'Chào mừng', 'các', 'bạn', 'trẻ', 'tới', 'thành phố', 'Hà Nội' ]\n```\n\nLoad custom trained model:\n\n```js\nvar vntk = require('vntk');\nvar tokenizer = vntk.wordTokenizer(new_model_path);\n\nconsole.log(tokenizer.tag('Chào mừng các bạn trẻ tới thành phố Hà Nội', 'text'));\n// Chào_mừng các bạn trẻ tới thành_phố Hà_Nội\n```\n\nCommand line: `vntk ws \u003cfile_name.txt\u003e`\n\n## 3. POS Tagging\n\n\u003e Vietnamese Part of Speech Tagging using Conditional Random Fields, called: `posTag`.  \n\u003e Pos_Tag helps labeling the part of speech of sentences!\n\n```js\nvar vntk = require('vntk');\nvar pos_tag = vntk.posTag();\n\nconsole.log(pos_tag.tag('Chợ thịt chó nổi tiếng ở TP Hồ Chí Minh bị truy quét'))\n// [ [ 'Chợ', 'N' ],\n//   [ 'thịt', 'N' ],\n//   [ 'chó', 'N' ],\n//   [ 'nổi tiếng', 'A' ],\n//   [ 'ở', 'E' ],\n//   [ 'TP', 'N' ],\n//   [ 'Hồ', 'Np' ],\n//   [ 'Chí', 'Np' ],\n//   [ 'Minh', 'Np' ],\n//   [ 'bị', 'V' ],\n//   [ 'truy quét', 'V' ] ]\n```\n\nLoad custom trained model:\n\n```js\nvar vntk = require('vntk');\nvar pos_tag = vntk.posTag(new_model_path);\n\nconsole.log(pos_tag.tag('Cán bộ xã và những chiêu \"xin làm hộ nghèo\" cười ra nước mắt', 'text'))\n// [N Cán bộ] [N xã] [C và] [L những] [N chiêu] [CH \"] [V xin] [V làm] [N hộ] [A nghèo] [CH \"] [V cười] [V ra] [N nước mắt]\n```\n\nCommand line: `vntk pos \u003cfile_name.txt\u003e`\n\n## 4. Chunking\n\n\u003e Vietnamese Chunking using Conditional Random Fields  \n\u003e Chucking helps labeling the part of speech of sentences and short phrases (like noun phrases)!\n\n```js\nvar vntk = require('vntk');\nvar chunking = vntk.chunking();\n\nconsole.log(chunking.tag('Nhật ký SEA Games ngày 21/8: Ánh Viên thắng giòn giã ở vòng loại.'))\n// [ [ 'Nhật ký', 'N', 'B-NP' ],\n//   [ 'SEA', 'N', 'B-NP' ],\n//   [ 'Games', 'Np', 'B-NP' ],\n//   [ 'ngày', 'N', 'B-NP' ],\n//   [ '21/8', 'M', 'B-NP' ],\n//   [ ':', 'CH', 'O' ],\n//   [ 'Ánh', 'Np', 'B-NP' ],\n//   [ 'Viên', 'Np', 'I-NP' ],\n//   [ 'thắng', 'V', 'B-VP' ],\n//   [ 'giòn giã', 'N', 'B-NP' ],\n//   [ 'ở', 'E', 'B-PP' ],\n//   [ 'vòng', 'N', 'B-NP' ],\n//   [ 'loại', 'N', 'B-NP' ],\n//   [ '.', 'CH', 'O' ] ]\n```\n\nLoad custom trained model:\n\n```js\nvar vntk = require('vntk');\nvar chunking = vntk.chunking(new_model_path);\n\nconsole.log(chunking.tag('Nhật ký SEA Games ngày 21/8: Ánh Viên thắng giòn giã ở vòng loại.', 'text'));\n// [NP Nhật ký] [NP SEA] [NP Games] [NP ngày] [NP 21/8] : [NP Ánh Viên] [VP thắng] [NP giòn giã] [PP ở] [NP vòng] [NP loại] .\n```\n\nCommand line: `vntk chunk \u003cfile_name.txt\u003e`\n\n## 5. Named Entity Recognition\n\n\u003e Vietnamese Named Entity Recognition (NER) using Conditional Random Fields  \n\u003e In NER, your goal is to find named entities, which tend to be noun phrases (though aren't always)\n\n```js\nvar vntk = require('vntk');\nvar ner = vntk.ner();\n\nconsole.log(ner.tag('Chưa tiết lộ lịch trình tới Việt Nam của Tổng thống Mỹ Donald Trump'))\n// [ [ 'Chưa', 'R', 'O', 'O' ],\n//   [ 'tiết lộ', 'V', 'B-VP', 'O' ],\n//   [ 'lịch trình', 'V', 'B-VP', 'O' ],\n//   [ 'tới', 'E', 'B-PP', 'O' ],\n//   [ 'Việt Nam', 'Np', 'B-NP', 'B-LOC' ],\n//   [ 'của', 'E', 'B-PP', 'O' ],\n//   [ 'Tổng thống', 'N', 'B-NP', 'O' ],\n//   [ 'Mỹ', 'Np', 'B-NP', 'B-LOC' ],\n//   [ 'Donald', 'Np', 'B-NP', 'B-PER' ],\n//   [ 'Trump', 'Np', 'B-NP', 'I-PER' ] ]\n```\n\nLoad custom trained model:\n\n```js\nvar vntk = require('vntk');\nvar ner = vntk.ner(new_model_path);\n\nconsole.log(ner.tag('Chưa tiết lộ lịch trình tới Việt Nam của Tổng thống Mỹ Donald Trump', 'text'))\n// Chưa  tiết lộ  lịch trình  tới [LOC Việt Nam] của  Tổng thống [LOC Mỹ] [PER Donald Trump]\n```\n\nCommand line: `vntk ner \u003cfile_name.txt\u003e`\n\n## 6. Utility\n\n### Dictionary\n\n* Check a word is exists in dictionary\n\n```js\nvar vntk = require('vntk');\nvar dictionary = vntk.dictionary();\n\ndictionary.has('chào');\n// true\n```\n\n* Lookup word definitons\n\n```js\nvar vntk = require('vntk');\nvar dictionary = vntk.dictionary();\n\nvar senses = dictionary.lookup('chào');\nconsole.log(senses);\n\n// Output\n[ { example: 'chào thầy giáo ~ con chào mẹ',\n    sub_pos: 'Vt',\n    definition: 'tỏ thái độ kính trọng hoặc quan tâm đối với ai bằng lời nói hay cử chỉ, khi gặp nhau hoặc khi từ biệt',\n    pos: 'V' },\n    { example: 'đứng nghiêm làm lễ chào cờ',\n    sub_pos: 'Vu',\n    definition: 'tỏ thái độ kính cẩn trước cái gì thiêng liêng, cao quý',\n    pos: 'V' },\n    { example: 'chào hàng ~ lời chào cao hơn mâm cỗ (tng)',\n    sub_pos: 'Vu',\n    definition: 'mời ăn uống hoặc mua hàng',\n    pos: 'V' }]\n```\n\n### Clean html\n\n```javascript\nvar vntk = require('vntk');\nvar util = vntk.util();\n\nutil.clean_html('\u003cspan style=\"color: #4b67a1;\"\u003eXin chào!!!\u003c/span\u003e');\n// Xin chào!!!\n```\n\n```bash\n# command line\nvntk clean \u003cfile_name1.txt\u003e\n```\n\n## 7. TF-IDF\n\n[Term Frequency–Inverse Document Frequency (tf-idf)](http://en.wikipedia.org/wiki/Tf%E2%80%93idf) is implemented to determine how important a word (or words) is to a document relative to a corpus. See following example.\n\n```js\nvar vntk = require('vntk');\nvar tfidf = new vntk.TfIdf();\n\ntfidf.addDocument('Đại tướng Trần Đại Quang - Ủy viên Bộ Chính trị, Bí thư Đảng ủy Công an Trung ương, Bộ trưởng Bộ Công an.');\ntfidf.addDocument('Thượng tướng Tô Lâm - Ủy viên Bộ Chính trị - Thứ trưởng Bộ Công an.');\ntfidf.addDocument('Thượng tướng Lê Quý Vương - Ủy viên Trung ương Đảng - Thứ trưởng Bộ Công an.');\ntfidf.addDocument('Thiếu tướng Bùi Mậu Quân - Phó Tổng cục trưởng Tổng cục An ninh');\n\nconsole.log('Bộ Công an --------------------------------');\ntfidf.tfidfs('Bộ Công an', function(i, measure) {\n    console.log('document #' + i + ' is ' + measure);\n});\n\nconsole.log('Tổng cục An ninh --------------------------------');\ntfidf.tfidfs('Tổng cục An ninh', function(i, measure) {\n    console.log('document #' + i + ' is ' + measure);\n});\n```\n\nThe above output:\n\n```text\nBộ Công an --------------------------------\ndocument #0 is 6.553712897371581\ndocument #1 is 3.7768564486857903\ndocument #2 is 2.7768564486857903\ndocument #3 is 0.7768564486857903\nTổng cục An ninh --------------------------\ndocument #0 is 1.5537128973715806\ndocument #1 is 0.7768564486857903\ndocument #2 is 0.7768564486857903\ndocument #3 is 9.242592351485516\n```\n\n## 8. Classifiers\n\n[Naive Bayes](http://en.wikipedia.org/wiki/Naive_Bayes_classifier), [fastText](https://github.com/facebookresearch/fastText) are classifiers currently supported.\n\n### Bayes Classifier\n\nThe following examples use the **BayesClassifier** class:\n\n```js\nvar vntk = require('vntk');\n\nvar classifier = new vntk.BayesClassifier();\n\nclassifier.addDocument('khi nào trận chiến đã kết thúc?', 'when');\nclassifier.addDocument('tàu rời đi lúc mấy giờ?', 'when');\nclassifier.addDocument('trận đấu diễn ra vào thời gian nào?', 'when');\nclassifier.addDocument('anh ấy rời đi vào lúc mấy giờ?', 'when');\nclassifier.addDocument('bao giờ thì đến lễ hội hóa trang?', 'when');\nclassifier.addDocument('ai phát hiện ra điện ?', 'who');\nclassifier.addDocument('người sáng lập ra microsoft là ai?', 'who');\nclassifier.addDocument('ai kiếm được tiền của họ một cách chăm chỉ ?', 'who');\nclassifier.addDocument('người phát minh tạo ra.', 'who');\nclassifier.addDocument('gia đình bạn gồm những ai?', 'who');\n\nclassifier.train();\n\n\nconsole.log(classifier.classify('chiến tranh thế giới bắt đầu vào lúc nào?'));\n// output: when\n\nconsole.log(classifier.classify('kẻ thù của luffy là ai?'));\n// output: who\n```\n\n### FastText Classifier\n\nAccording to [fasttext.cc](https://fasttext.cc/docs/en/supervised-tutorial.html). We have a simple classifier for executing prediction models about `cooking` from stackexchange questions:\n\n```js\nconst path = require('path');\nconst vntk = require('vntk');\n\nconst model = path.resolve(__dirname, './model_cooking.bin');\nconst classifier = new vntk.FastTextClassifier(model);\n\nclassifier.predict('Why not put knives in the dishwasher?', 5, (err, res) =\u003e {\n    if (err) {\n        console.error(err);\n    } else if (res.length \u003e 0) {\n        let tag = res[0].label; // __label__knives\n        let confidence = res[0].value // 0.8787146210670471\n        console.log('classify', tag, confidence, res);\n    } else {\n        console.log('No matches');\n    }\n});\n```\n\n## 9. Language identification\n\n**VNTK Langid** can identify 176 languages from text samples and return confidence scores for each (see the list of ISO codes below). This model was trained by [fastText](https://fasttext.cc/docs/en/language-identification.html) on data from Wikipedia, Tatoeba and SETimes, used under CC-BY-SA.\n\nApi usage example:\n\n* langid.detect([input])\n* langid.getLanguages([input, num, callback])\n* langid.langids - list of supported languages\n\n```js\nconst langid = require('vntk').langid();\n\n// returns the most accuracy language detected\nlangid.detect('sử dụng vntk với fastext rất tuyệt?')\n    .then((lid) =\u003e {\n        console.log(lid)\n        // vi\n    });\n\n// returns the list of detectable languages\nlangid.getLanguages('Wie lange bleiben Sie?', 5)\n    .then((res) =\u003e {\n        let lid = res[0].label;\n        t.equal(lid, 'de', 'German');\n        t.equal(res.length, 5, 'number of languagues are detected');\n        console.log(res)\n    });\n\n// returns list of supported languagues\nconsole.log(langid.langids)\n```\n\nLoad custom trained model:\n\n```js\nvar vntk = require('vntk');\nvar langid = vntk.langid(new_model_path);\n```\n\nList of supported languages\n\n\u003e af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs bxr ca cbk ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es et eu fa fi fr frr fy ga gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia id ie ilo io is it ja jbo jv ka kk km kn ko krc ku kv kw ky la lb lez li lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms mt mwl my myv mzn nah nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu rm ro ru rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl tr tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh\n\n\n## 10. CRFSuite\n\nFor quick access to `CRFSuite` which shipped with `vntk` we can refer to it via following api.\n\n\u003e var crfsuite = require('vntk').crfsuite()\n\nThen create a `Tagger` or `Trainer`:\n\n```js\nvar crfsuite = require('vntk').crfsuite()\nvar tagger = new crfsuite.Tagger()\nvar trainer = new crfsuite.Trainer()\n```\n\nFor detail documentation, click [here](https://github.com/vunb/node-crfsuite).\n\n# NLP API Server\n\nFollow these steps to quickly serve an NLP API server using vntk:\n\n```bash\n# Clone the repository\ngit clone https://github.com/vunb/vntk\n\n# Move to source code folder\ncd vntk\n\n# Install dependencies\nnpm install\n\n# Run NLP API server\nnpm run server\n\n# Copy and paste the following link to your browser to see result in action\n# http://localhost:3000/api/tok/Phó Thủ tướng Vương Đình Huệ yêu cầu điều chỉnh tên gọi “trạm thu giá” BOT\n```\n\nDetail checkout: [./server](./server)\n\n# Contributing\n\nPull requests and stars are highly welcome.\n\nFor bugs and feature requests, please [create an issue](https://github.com/vunb/vntk/issues/new).\n\nLICENSE\n========\n\nMIT.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvunb%2Fvntk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvunb%2Fvntk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvunb%2Fvntk/lists"}