https://github.com/wmentor/tokens

Text to tokes
https://github.com/wmentor/tokens

go golang golang-library text-processing text-processor tokenizer tokens

Last synced: about 1 year ago
JSON representation

Text to tokes

• Host: GitHub
• URL: https://github.com/wmentor/tokens
• Owner: wmentor
• License: mit
• Created: 2020-07-12T17:18:59.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2023-12-26T18:58:45.000Z (over 2 years ago)
• Last Synced: 2025-01-14T06:54:13.023Z (about 1 year ago)
• Topics: go, golang, golang-library, text-processing, text-processor, tokenizer, tokens
• Language: Go
• Homepage:
• Size: 27.3 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# tokens

![test](https://github.com/wmentor/tokens/workflows/test/badge.svg)

[![Coverage Status](https://coveralls.io/repos/github/wmentor/tokens/badge.svg?branch=master&v=1.0.4)](https://coveralls.io/github/wmentor/tokens?branch=master)

[![https://goreportcard.com/report/github.com/wmentor/tokens](https://goreportcard.com/badge/github.com/wmentor/tokens)](https://goreportcard.com/report/github.com/wmentor/tokens)

[![https://pkg.go.dev/github.com/wmentor/tokens](https://pkg.go.dev/badge/github.com/wmentor/tokens.svg)](https://pkg.go.dev/github.com/wmentor/tokens)

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Text to tokens Go library.

# Token get insensitive mode

```go

package main

import (

	"fmt"

	"strings"

	"github.com/wmentor/tokens"

)

func main() {

	txt := "Hello, my little friend!"

	tokenizer := tokens.New(strings.NewReader(txt))

	for {

		tok, err := range tokenizer.Token()

		if err != nil { // io.EOF

			break

		}

		fmt.Println(tok)

	}

}

```

Result:

```

hello

,

my

little

friend

!

```

Case sensitive mode:

```go

package main

import (

	"fmt"

	"strings"

	"github.com/wmentor/tokens"

)

func main() {

	txt := "Hello, my little friend!"

	tokenizer := tokens.New(strings.NewReader(txt), tokens.WithCaseSensitive())

	for {

		tok, err := range tokenizer.Token()

		if err != nil { // io.EOF

			break

		}

		fmt.Println(tok)

	}

}

```

Result:

```

Hello

,

my

liTTle

friend

!

```