https://github.com/awsong/MMSEGO

Chinese word splitting algorithm MMSEG in GO
https://github.com/awsong/MMSEGO

Last synced: about 2 months ago
JSON representation

Chinese word splitting algorithm MMSEG in GO

• Host: GitHub
• URL: https://github.com/awsong/MMSEGO
• Owner: awsong
• License: other
• Created: 2012-04-18T04:06:21.000Z (over 12 years ago)
• Default Branch: darts
• Last Pushed: 2012-04-18T04:18:51.000Z (over 12 years ago)
• Last Synced: 2024-08-03T23:07:00.521Z (4 months ago)
• Language: Go
• Homepage:
• Size: 683 KB
• Stars: 63
• Watchers: 5
• Forks: 14
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-go - MMSEGO - This is a GO implementation of [MMSEG](http://technology.chtsai.org/mmseg/) which a Chinese word splitting algorithm. (Natural Language Processing / Tokenizers)
• zero-alloc-awesome-go - MMSEGO - This is a GO implementation of [MMSEG](http://technology.chtsai.org/mmseg/) which a Chinese word splitting algorithm. (Natural Language Processing / Tokenizers)
• awesome-go - MMSEGO - Chinese word splitting algorithm MMSEG in GO - ★ 57 (Natural Language Processing)
• awesome-go-extra - MMSEGO - 04-18T04:06:21Z|2012-04-18T04:18:51Z| (Bot Building / Tokenizers)

README

        
MMSEGO

=====

This is a GO implementation of [MMSEG](http://technology.chtsai.org/mmseg/) which a Chinese word splitting algorithm.

TO DO list

----------

* Documentation/comments

* Benchmark

Usage

---------

#Input Dictionary Format

```sh

Key\tFreq

```

Each key occupies one line. The file should be utf-8 encoded, please refer to [go-darts](https://github.com/awsong/go-darts)

#Code example

```go

package main

import (

    "fmt"

    "time"

    "os"

    "mmsego"

    "bufio"

    "log"

    )

func main() {

    var s = new(mmsego.Segmenter)

    s.Init("darts.lib")

    if err != nil {

	log.Fatal(err)

    }

    t := time.Now()

    offset := 0

    unifile, _ := os.Open("/tmp/a.txt")

    uniLineReader := bufio.NewReaderSize(unifile, 4000)

    line, bufErr := uniLineReader.ReadString('\n')

    for nil == bufErr {

	//takeWord := func(off int, length int){ fmt.Printf("%s ", string(line[off-offset:off-offset+length])) }

	takeWord := func(off, length int){ }

	s.Mmseg(line[:], offset, takeWord, nil, false)

	offset += len(line)

	line, bufErr = uniLineReader.ReadString('\n')

    }

    takeWord := func(off int, length int){ fmt.Printf("%s ", string(line[off-offset:off-offset+length])) }

    s.Mmseg(line, offset, takeWord, nil, true)

    fmt.Printf("Duration: %v\n", time.Since(t))

}

```

LICENSE

-----------

Apache License 2.0