https://github.com/reinfer/blingfire-rs

Rust wrapper for the BlingFire tokenization library
https://github.com/reinfer/blingfire-rs

machine-learning nlp rust rust-wrapper tokenizer

Last synced: 3 months ago
JSON representation

Rust wrapper for the BlingFire tokenization library

• Host: GitHub
• URL: https://github.com/reinfer/blingfire-rs
• Owner: reinfer
• License: mit
• Created: 2019-07-06T13:50:14.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2020-06-23T16:31:02.000Z (over 4 years ago)
• Last Synced: 2024-07-18T04:39:03.124Z (4 months ago)
• Topics: machine-learning, nlp, rust, rust-wrapper, tokenizer
• Language: Rust
• Size: 24.4 KB
• Stars: 15
• Watchers: 11
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

        
[![Build Status](https://travis-ci.com/reinfer/blingfire-rs.svg?branch=master)](https://travis-ci.com/reinfer/blingfire-rs)

[![Documentation](https://docs.rs/blingfire/badge.svg)](https://docs.rs/blingfire)

[![Crate](https://meritbadge.herokuapp.com/blingfire)](https://crates.io/crates/blingfire)

# BlingFire in Rust

`blingfire` is a thin Rust wrapper for the [BlingFire](https://github.com/microsoft/BlingFire) tokenization library.

Add the library to `Cargo.toml` to get started

```bash

cargo add blingfire

```

The library exposes two functions `text_to_words` and `text_to_sentences`

```rust

use blingfire;

fn main() {

    let mut parsed = String::new();

    blingfire::text_to_words("Cat,sat on   the mat.", &mut parsed).unwrap();

    assert_eq!(parsed.as_str(), "Cat , sat on the mat .");

    blingfire::text_to_sentences("Cat sat. Dog barked.", &mut parsed).unwrap();

    assert_eq!(parsed.as_str(), "Cat sat.\nDog barked.");

}

```

The code is licensed under the MIT License.