https://github.com/osamingo/shamoji

The shamoji (杓文字) is a word filtering package
https://github.com/osamingo/shamoji

go tokenizer word-filter

Last synced: about 1 month ago
JSON representation

The shamoji (杓文字) is a word filtering package

• Host: GitHub
• URL: https://github.com/osamingo/shamoji
• Owner: osamingo
• License: mit
• Created: 2017-07-23T06:38:42.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2022-12-01T19:56:27.000Z (about 2 years ago)
• Last Synced: 2024-10-25T05:24:20.639Z (about 2 months ago)
• Topics: go, tokenizer, word-filter
• Language: Go
• Homepage:
• Size: 38.1 KB
• Stars: 13
• Watchers: 2
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

• awesome-go - shamoji - The shamoji is word filtering package written in Go. (Natural Language Processing / Tokenizers)
• zero-alloc-awesome-go - shamoji - The shamoji is word filtering package written in Go. (Natural Language Processing / Tokenizers)
• awesome-go-extra - shamoji - 07-23T06:38:42Z|2022-05-04T07:58:17Z| (Bot Building / Tokenizers)
• awesome-go-zh - shamoji
• awesome-go - shamoji - The shamoji (杓文字) is word filtering package - ★ 9 (Natural Language Processing)

README

        
# shamoji

[![GitHub Actions](https://github.com/osamingo/shamoji/workflows/CI/badge.svg?branch=master)](https://github.com/osamingo/shamoji/actions?query=workflow%3ACI+branch%3Amaster)

[![codecov](https://codecov.io/gh/osamingo/shamoji/branch/master/graph/badge.svg)](https://codecov.io/gh/osamingo/shamoji)

[![Go Report Card](https://goreportcard.com/badge/github.com/osamingo/shamoji)](https://goreportcard.com/report/github.com/osamingo/shamoji)

[![codebeat badge](https://codebeat.co/badges/9d9fdf3d-0c6d-455f-8444-8399a07d49ae)](https://codebeat.co/projects/github-com-osamingo-shamoji-master)

[![GoDoc](https://godoc.org/github.com/osamingo/shamoji?status.svg)](https://godoc.org/github.com/osamingo/shamoji)

[![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/osamingo/shamoji/master/LICENSE)

## About

The shamoji (杓文字) is word filtering package.

## Install

```

$ go get github.com/osamingo/shamoji@latest

```

## Usage

```go

package main

import (

	"fmt"

	"sync"

	"github.com/osamingo/shamoji"

	"github.com/osamingo/shamoji/filter"

	"github.com/osamingo/shamoji/tokenizer"

	"golang.org/x/text/unicode/norm"

)

var (

	o sync.Once

	s *shamoji.Serve

)

func main() {

	yes, word := Contains("我が生涯に一片の悔い無し")

	fmt.Printf("Result: %v, Word: %s", yes, word)

}

func Contains(sentence string) (bool, string) {

	o.Do(func() {

		tok, err := tokenizer.NewKagomeTokenizer(norm.NFKC)

		if err != nil {

			panic(err)

		}

		s = &shamoji.Serve{

			Tokenizer: tok,

			Filer:     filter.NewCuckooFilter("涯に", "悔い"),

		}

	})

	return s.Do(sentence)

}

```

## License

Released under the [MIT License](https://github.com/osamingo/shamoji/blob/master/LICENSE).