https://github.com/rekram1-node/text-processor
NLP utility library to interact with text. Initially this is to process English documents, essays, etc
https://github.com/rekram1-node/text-processor
Last synced: 11 months ago
JSON representation
NLP utility library to interact with text. Initially this is to process English documents, essays, etc
- Host: GitHub
- URL: https://github.com/rekram1-node/text-processor
- Owner: rekram1-node
- License: mit
- Created: 2023-01-28T15:41:52.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-01-29T19:03:54.000Z (over 3 years ago)
- Last Synced: 2023-07-28T15:48:02.922Z (almost 3 years ago)
- Language: Go
- Homepage:
- Size: 161 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Text Processor
[](https://goreportcard.com/report/github.com/rekram1-node/text-processor) [](https://github.com/rekram1-node/text-processor/blob/main/LICENSE) 
NLP utility library to interact with text documents using a [Word2vec Model](https://developer.syn.co.in/tutorial/bot/oscova/pretrained-vectors.html) Library parses out sentences and paragraphs, removes stop words and tokenizes sentences in order to be consumed by the word2vec comparison functions
## Features
* Extract Sentences and Paragraphs from text
* Show how similar sentences and paragraphs are using word2vec model
* Tokenization, stop word removing, vectorization (done internally)
* Getting key phrases and general sentiment
## How it works
## Getting Started
### Prerequisites
- [Go](https://go.dev/)
- [Word2vec Model](https://developer.syn.co.in/tutorial/bot/oscova/pretrained-vectors.html)
- Note: model must be unzipped and in working directory!
## Installing Model
Go to [Word2vec Model](https://developer.syn.co.in/tutorial/bot/oscova/pretrained-vectors.html) and select one of their models to download, I use the 300 Dimension Google News One
### Getting blinkgo
With [Go module](https://github.com/golang/go/wiki/Modules) support, simply add the following import
```go
import "github.com/rekram1-node/text-processor/text"
```
to your code, and then `go [build|run|test]` will automatically fetch the necessary dependencies.
Otherwise, run the following to install the `text-processor` library
```shell
$ go get -u github.com/rekram1-node/text-processor/text
```
## Usage
### Basic Text Comparison
```go
package main
import (
"fmt"
"log"
"github.com/rekram1-node/text-processor/text"
)
func main() {
t1 := "So much of modern-day life revolves around using opposable thumbs, from holding a hammer to build a home to ordering food delivery on our smartphones. But for our ancestors, the uses were much simpler. Strong and nimble thumbs meant that they could better create and wield tools, stones and bones for killing large animals for food"
t2 := "A lot of life today involves using opposable thumbs, from using a hammer to build a house to ordering something on our smartphones. But for our predecessors, the uses were much more simple. Powerful and dexterous thumbs meant that they could better make and use tools, stones and bones for killing large animals to eat"
// load the word2vec model
m, err := text.LoadModel("yourModelFile")
if err != nil {
log.Fatal(err)
}
// extract paragraphs and sentences from the text
t1Paragraphs, t1Sentences, err := text.ExtractAll(t1)
if err != nil {
log.Fatal(err)
}
// extract paragraphs and sentences from the text
t2Paragraphs, t2Sentences, err := text.ExtractAll(t2)
if err != nil {
log.Fatal(err)
}
// compare the two texts and a map of sentences (from 1st document)
// paired to sentences (from 2nd document) along with a similarity score
sim, err := m.MostSimilarSentences(t1Sentences, t2Sentences)
if err != nil {
log.Fatal(err)
}
// iterate over sentence array and display data
for _, sentence := range t1Sentences {
simSentence := sim[sentence]
if simSentence.Sentence != "" {
fmt.Println()
fmt.Println(sentence, "is most similar to:", simSentence.Sentence)
fmt.Printf("similarity: %v\n", simSentence.Score)
fmt.Println()
}
}
// compare the two texts and a map of paragraphs (from 1st document)
// paired to paragraphs (from 2nd document) along with a similarity score
simPara, err := m.MostSimilarParagraphs(t1Paragraphs, t2Paragraphs)
if err != nil {
log.Fatal(err)
}
// iterate over paragraph array and display data
for _, para := range t1Paragraphs {
simParagraph := simPara[para]
if simParagraph.Paragraph != "" {
fmt.Println()
fmt.Println(para, "is most similar to:", simParagraph.Paragraph)
fmt.Printf("similarity: %v\n", simParagraph.Score)
fmt.Println()
}
}
}
```
## Issues
If you have an issue: report it on the [issue tracker](https://github.com/rekram1-node/text-processor/issues)