https://github.com/narsil/esaxx-rs

Bindings to copy of SentencePiece esaxx library (fast suffix array and frequent substrings).
https://github.com/narsil/esaxx-rs

Last synced: 5 months ago
JSON representation

Bindings to copy of SentencePiece esaxx library (fast suffix array and frequent substrings).

• Host: GitHub
• URL: https://github.com/narsil/esaxx-rs
• Owner: Narsil
• License: apache-2.0
• Created: 2020-06-06T07:47:06.000Z (about 5 years ago)
• Default Branch: main
• Last Pushed: 2024-06-30T06:41:08.000Z (11 months ago)
• Last Synced: 2024-12-27T01:05:24.112Z (5 months ago)
• Language: C++
• Size: 207 KB
• Stars: 5
• Watchers: 3
• Forks: 16
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
![](https://github.com/Narsil/esaxx-rs/workflows/build/badge.svg)

# esaxx-rs

This code implements a fast suffix tree / suffix array.

This code is taken from ![sentencepiece](https://github.com/google/sentencepiece) 

and to be used by ![hugging face](https://github.com/huggingface/tokenizers/).

Small wrapper around sentencepiece's esaxx suffix array C++ library.

Usage

```rust

let string = "abracadabra";

let suffix = esaxx_rs::suffix(string).unwrap();

let chars: Vec<_> = string.chars().collect();

let mut iter = suffix.iter();

assert_eq!(iter.next().unwrap(), (&chars[..4], 2)); // abra

assert_eq!(iter.next(), Some((&chars[..1], 5))); // a

assert_eq!(iter.next(), Some((&chars[1..4], 2))); // bra

assert_eq!(iter.next(), Some((&chars[2..4], 2))); // ra

assert_eq!(iter.next(), Some((&chars[..0], 11))); // ''

assert_eq!(iter.next(), None);

```