https://github.com/euskadi31/go-tokenizer

A Text Tokenizer library for Golang
https://github.com/euskadi31/go-tokenizer

go golang golang-library machine-learning nlp text tokenizer

Last synced: 3 months ago
JSON representation

A Text Tokenizer library for Golang

• Host: GitHub
• URL: https://github.com/euskadi31/go-tokenizer
• Owner: euskadi31
• License: mit
• Created: 2018-03-18T14:22:53.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2025-03-17T14:45:18.000Z (3 months ago)
• Last Synced: 2025-03-17T15:52:11.862Z (3 months ago)
• Topics: go, golang, golang-library, machine-learning, nlp, text, tokenizer
• Language: Go
• Size: 46.9 KB
• Stars: 11
• Watchers: 2
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

        
# Text Tokenizer for Golang ![Last release](https://img.shields.io/github/release/euskadi31/go-tokenizer.svg)

[![Go Report Card](https://goreportcard.com/badge/github.com/euskadi31/go-tokenizer)](https://goreportcard.com/report/github.com/euskadi31/go-tokenizer)

| Branch | Status                                                                                                                                                    | Coverage                                                                                                                                             |

| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |

| master | [![Go](https://github.com/euskadi31/go-tokenizer/actions/workflows/go.yml/badge.svg)](https://github.com/euskadi31/go-tokenizer/actions/workflows/go.yml) | [![Coveralls](https://img.shields.io/coveralls/euskadi31/go-tokenizer/master.svg)](https://coveralls.io/github/euskadi31/go-tokenizer?branch=master) |

```sh

go get -u github.com/euskadi31/go-tokenizer/v3

```

```go

import (

    "fmt"

    "github.com/euskadi31/go-tokenizer/v3"

)

func main() {

    t := tokenizer.New()

    tokens := t.Tokenize("I believe life is an intelligent thing: that things aren't random.")

    fmt.Print(tokens) // []string{"I", "believe", "life", "is", "an", "intelligent", "thing", "that", "things", "aren't", "random"}

}

```

## License

go-tokenizer is licensed under [the MIT license](LICENSE.md).