https://github.com/gitbookio/tokenize-english

Javascript tokenizer for english sentences
https://github.com/gitbookio/tokenize-english

Last synced: 9 months ago
JSON representation

Javascript tokenizer for english sentences

• Host: GitHub
• URL: https://github.com/gitbookio/tokenize-english
• Owner: GitbookIO
• License: apache-2.0
• Created: 2015-10-07T11:49:48.000Z (over 10 years ago)
• Default Branch: master
• Last Pushed: 2015-10-15T10:15:06.000Z (about 10 years ago)
• Last Synced: 2025-04-23T02:39:28.088Z (9 months ago)
• Language: JavaScript
• Homepage:
• Size: 128 KB
• Stars: 14
• Watchers: 5
• Forks: 4
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# tokenize-english

[![Build Status](https://travis-ci.org/GitbookIO/tokenize-english.png?branch=master)](https://travis-ci.org/GitbookIO/tokenize-english)

[![NPM version](https://badge.fury.io/js/tokenize-english.svg)](http://badge.fury.io/js/tokenize-english)

Javascript tokenizer for english sentences.

### Installation

```

$ npm install tokenize-english

```

### Usage

```js

var tokenize = require('tokenize-text')();

var tokenizeEnglish = require('tokenize-english')(tokenize);

var tokens = tokenizeEnglish.sentences("On Jan. 20, former Sen. Barack Obama became the 44th President of the U.S. Millions attended the Inauguration.")

/*

[ { index: 0,

    value: 'On Jan. 20, former Sen. Barack Obama became the 44th President of the U.S.',

    offset: 74 },

  { index: 74,

    value: ' Millions attended the Inauguration.',

    offset: 36 } ]

*/

```