https://github.com/tokestermw/spacy_kenlm

:game_die: KenLM extension for spaCy 2.0.
https://github.com/tokestermw/spacy_kenlm

kenlm language-model nlp spacy spacy-extension spacy-nlp

Last synced: 2 months ago
JSON representation

:game_die: KenLM extension for spaCy 2.0.

• Host: GitHub
• URL: https://github.com/tokestermw/spacy_kenlm
• Owner: tokestermw
• License: mit
• Created: 2017-12-04T23:53:08.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2017-12-06T05:23:52.000Z (about 7 years ago)
• Last Synced: 2024-10-01T07:42:54.301Z (3 months ago)
• Topics: kenlm, language-model, nlp, spacy, spacy-extension, spacy-nlp
• Language: Python
• Homepage:
• Size: 8.79 KB
• Stars: 16
• Watchers: 5
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
## spacy_kenlm: KenLM extension for spaCy 2.0

This package adds [kenLM](https://github.com/kpu/kenlm) support

as a [spaCy 2.0 extension](https://spacy.io/usage/processing-pipelines#extensions).

## Usage

Train a `kenLM` language model first (or use the test model from `test.arpa`).

Add the spaCyKenLM to the spaCy pipeline to return scores.

```

import spacy

from spacy_kenlm import spaCyKenLM

nlp = spacy.load('en_core_web_sm')

spacy_kenlm = spaCyKenLM()  # default model from test.arpa

nlp.add_pipe(spacy_kenlm)

doc = nlp('How are you?')

# doc score

doc._.kenlm_score

# span score

doc[:2]._.kenlm_score

# token score

doc[2]._.kenlm_score

```

## Installation

Install from the pip package.

```

pip install spacy_kenlm

```