{"id":19098439,"url":"https://github.com/winkjs/wink-tokenizer","last_synced_at":"2025-10-28T01:48:40.289Z","repository":{"id":26846175,"uuid":"110212879","full_name":"winkjs/wink-tokenizer","owner":"winkjs","description":"Multilingual tokenizer that automatically tags each token with its type","archived":false,"fork":false,"pushed_at":"2023-03-04T02:27:58.000Z","size":2147,"stargazers_count":61,"open_issues_count":2,"forks_count":12,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-19T13:51:31.043Z","etag":null,"topics":["devanagari","french","german","hindi","konkani","latin","marathi","multilingual","tagging","tokenization","tokenizer","wink"],"latest_commit_sha":null,"homepage":"http://winkjs.org/wink-tokenizer/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/winkjs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-10T06:47:27.000Z","updated_at":"2025-04-12T09:42:16.000Z","dependencies_parsed_at":"2024-06-18T15:27:21.675Z","dependency_job_id":"be4c3142-5983-4be5-97c3-36de2375f0a1","html_url":"https://github.com/winkjs/wink-tokenizer","commit_stats":{"total_commits":100,"total_committers":6,"mean_commits":"16.666666666666668","dds":0.13,"last_synced_commit":"1479974749efed366401eac79e6abca48315786d"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/winkjs%2Fwink-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/winkjs%2Fwink-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/winkjs%2Fwink-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/winkjs%2Fwink-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/winkjs","download_url":"https://codeload.github.com/winkjs/wink-tokenizer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252931391,"owners_count":21827104,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["devanagari","french","german","hindi","konkani","latin","marathi","multilingual","tagging","tokenization","tokenizer","wink"],"created_at":"2024-11-09T03:45:46.519Z","updated_at":"2025-10-28T01:48:35.254Z","avatar_url":"https://github.com/winkjs.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# wink-tokenizer\n\nMultilingual tokenizer that automatically tags each token with its type\n\n### [![Build Status](https://api.travis-ci.org/winkjs/wink-tokenizer.svg?branch=master)](https://travis-ci.org/winkjs/wink-tokenizer) [![Coverage Status](https://coveralls.io/repos/github/winkjs/wink-tokenizer/badge.svg?branch=master)](https://coveralls.io/github/winkjs/wink-tokenizer?branch=master) [![Gitter](https://img.shields.io/gitter/room/nwjs/nw.js.svg)](https://gitter.im/winkjs/Lobby)\n\n[\u003cimg align=\"right\" src=\"https://decisively.github.io/wink-logos/logo-title.png\" width=\"100px\" \u003e](http://winkjs.org/)\n\nTokenize sentences in Latin and Devanagari scripts using **`wink-tokenizer`**. Some of it's top feature are outlined below:\n\n1. Support for English, French, German, Hindi, Sanskrit, Marathi and many more.\n\n1. Intelligent tokenization of sentence containing words in more than one language.\n\n1. Automatic detection \u0026 tagging of different types of tokens based on their features:\n     - These include word, punctuation, email, mention, hashtag, emoticon, and emoji etc.\n     - User definable token types.\n\n1. High performance – tokenizes a typical english sentence at speed of over **2.4 million tokens/second** and a complex tweet containing hashtags, emoticons, emojis, mentions, e-mail at a speed of over **1.5 million tokens/second** (benchmarked on 2.2 GHz Intel Core i7 machine with 16GB RAM).\n\n\n### Installation\n\nUse [npm](https://www.npmjs.com/package/wink-tokenizer) to install:\n\n    npm install wink-tokenizer --save\n\n### Getting Started\n```javascript\n// Load tokenizer.\nvar tokenizer = require( 'wink-tokenizer' );\n// Create it's instance.\nvar myTokenizer = tokenizer();\n\n// Tokenize a tweet.\nvar s = '@superman: hit me up on my email r2d2@gmail.com, 2 of us plan party🎉 tom at 3pm:) #fun';\nmyTokenizer.tokenize( s );\n// -\u003e [ { value: '@superman', tag: 'mention' },\n//      { value: ':', tag: 'punctuation' },\n//      { value: 'hit', tag: 'word' },\n//      { value: 'me', tag: 'word' },\n//      { value: 'up', tag: 'word' },\n//      { value: 'on', tag: 'word' },\n//      { value: 'my', tag: 'word' },\n//      { value: 'email', tag: 'word' },\n//      { value: 'r2d2@gmail.com', tag: 'email' },\n//      { value: ',', tag: 'punctuation' },\n//      { value: '2', tag: 'number' },\n//      { value: 'of', tag: 'word' },\n//      { value: 'us', tag: 'word' },\n//      { value: 'plan', tag: 'word' },\n//      { value: 'party', tag: 'word' },\n//      { value: '🎉', tag: 'emoji' },\n//      { value: 'tom', tag: 'word' },\n//      { value: 'at', tag: 'word' },\n//      { value: '3pm', tag: 'time' },\n//      { value: ':)', tag: 'emoticon' },\n//      { value: '#fun', tag: 'hashtag' } ]\n\n// Tokenize a French sentence.\ns = 'Mieux vaut prévenir que guérir:-)';\nmyTokenizer.tokenize( s );\n// -\u003e [ { value: 'Mieux', tag: 'word' },\n//      { value: 'vaut', tag: 'word' },\n//      { value: 'prévenir', tag: 'word' },\n//      { value: 'que', tag: 'word' },\n//      { value: 'guérir', tag: 'word' },\n//      { value: ':-)', tag: 'emoticon' } ]\n\n// Tokenize a sentence containing Hindi and English.\ns = 'द्रविड़ ने टेस्ट में ३६ शतक जमाए, उनमें 21 विदेशी playground पर हैं।';\nmyTokenizer.tokenize( s );\n// -\u003e [ { value: 'द्रविड़', tag: 'word' },\n//      { value: 'ने', tag: 'word' },\n//      { value: 'टेस्ट', tag: 'word' },\n//      { value: 'में', tag: 'word' },\n//      { value: '३६', tag: 'number' },\n//      { value: 'शतक', tag: 'word' },\n//      { value: 'जमाए', tag: 'word' },\n//      { value: ',', tag: 'punctuation' },\n//      { value: 'उनमें', tag: 'word' },\n//      { value: '21', tag: 'number' },\n//      { value: 'विदेशी', tag: 'word' },\n//      { value: 'playground', tag: 'word' },\n//      { value: 'पर', tag: 'word' },\n//      { value: 'हैं', tag: 'word' },\n//      { value: '।', tag: 'punctuation' } ]\n```\n\n### Documentation\nCheck out the [tokenizer API documentation](http://winkjs.org/wink-tokenizer/) to learn more.\n\n### Need Help?\n\nIf you spot a bug and the same has not yet been reported, raise a new [issue](https://github.com/winkjs/wink-tokenizer/issues) or consider fixing it and sending a pull request.\n\n### About wink\n[Wink](http://winkjs.org/) is a family of open source packages for **Statistical Analysis**, **Natural Language Processing** and **Machine Learning** in NodeJS. The code is **thoroughly documented** for easy human comprehension and has a **test coverage of ~100%** for reliability to build production grade solutions.\n\n\n### Copyright \u0026 License\n\n**wink-tokenizer** is copyright 2017-21 [GRAYPE Systems Private Limited](http://graype.in/).\n\nIt is licensed under the terms of the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwinkjs%2Fwink-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwinkjs%2Fwink-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwinkjs%2Fwink-tokenizer/lists"}