https://github.com/liyuanlucasliu/lightner
Inference with state-of-the-art models (pre-trained by LD-Net / AutoNER / VanillaNER / ...)
https://github.com/liyuanlucasliu/lightner
inference ner neural-network sequence-labeling
Last synced: about 1 month ago
JSON representation
Inference with state-of-the-art models (pre-trained by LD-Net / AutoNER / VanillaNER / ...)
- Host: GitHub
- URL: https://github.com/liyuanlucasliu/lightner
- Owner: LiyuanLucasLiu
- License: apache-2.0
- Created: 2018-09-30T22:08:39.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2018-12-15T04:22:10.000Z (over 6 years ago)
- Last Synced: 2025-04-15T03:54:16.542Z (about 1 month ago)
- Topics: inference, ner, neural-network, sequence-labeling
- Language: Python
- Homepage:
- Size: 76.2 KB
- Stars: 115
- Watchers: 9
- Forks: 4
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: HISTORY.rst
- License: LICENSE
Awesome Lists containing this project
README
# LightNER
[](https://opensource.org/licenses/Apache-2.0)
[](https://badge.fury.io/py/LightNER)
[](https://pepy.tech/project/lightner)
**Check Our New NER Toolkit🚀🚀🚀**
- **Inference**:
- **[LightNER](https://github.com/LiyuanLucasLiu/LightNER)**: inference w. models pre-trained / trained w. *any* following tools, *efficiently*.
- **Training**:
- **[LD-Net](https://github.com/LiyuanLucasLiu/LD-Net)**: train NER models w. efficient contextualized representations.
- **[VanillaNER](https://github.com/LiyuanLucasLiu/Vanilla_NER)**: train vanilla NER models w. pre-trained embedding.
- **Distant Training**:
- **[AutoNER](https://github.com/shangjingbo1226/AutoNER)**: train NER models w.o. line-by-line annotations and get competitive performance.
--------------------------------
This package supports to conduct inference with models pre-trained by:
- [Vanilla_NER](https://github.com/LiyuanLucasLiu/Vanilla_NER): vanilla sequence labeling models.
- [LD-Net](https://github.com/LiyuanLucasLiu/LD-Net): sequence labeling models w. efficient contextualized representation.
- [AutoNER](https://github.com/shangjingbo1226/AutoNER): distant supervised named entity recognition models (*no line-by-line annotations for training*).
We are in an early-release beta. Expect some adventures and rough edges.
## Quick Links
- [Installation](#installation)
- [Usage](#usage)
## Installation
To install via pypi:
```
pip install lightner
```
To build from source:
```
pip install git+https://github.com/LiyuanLucasLiu/LightNER
```
or
```
git clone https://github.com/LiyuanLucasLiu/LightNER.git
cd LightNER
python setup.py install
```
## Usage
### Pre-trained Models
| | Model | Task | Performance |
| ------------- |------------- | ------------- | ------------- |
| [LD-Net](https://github.com/LiyuanLucasLiu/LD-Net) | [pner1.th](http://dmserv4.cs.illinois.edu/pner1.th) | NER for (PER, LOC, ORG & MISC) | F1 92.21 |
| [LD-Net](https://github.com/LiyuanLucasLiu/LD-Net) | [pnp0.th](http://dmserv4.cs.illinois.edu/pnp0.th) | Chunking | F1 95.79 |
| Vanilla_NER | | NER for (PER, LOC, ORG & MISC) | |
| Vanilla_NER | | Chunking | |
| [AutoNER](https://github.com/shangjingbo1226/AutoNER) | [autoner0.th](http://dmserv4.cs.illinois.edu/bioner_models/autoner0.th) | Distant NER trained w.o. line-by-line annotations (Disease, Chemical) | F1 85.30 |
### Decode API
The decode api can be called in the following way:
```
from lightner import decoder_wrapper
model = decoder_wrapper()
model.decode(["Ronaldo", "won", "'t", "score", "more", "than", "30", "goals", "for", "Juve", "."])
```
The ```decode()``` method also can conduct decoding at document level (takes list of list of ```str``` as input) or corpus level (takes list of list of list of ```str``` as input).
The ```decoder_wrapper``` method can be customized by choosing a different pre-trained model or passing an additional ```configs``` file as:
```
model = decoder_wrapper(URL_OR_PATH_TO_CHECKPOINT, configs)
```
And you can access the config options by:
```
lightner decode -h
```
### Console
After installing and downloading the pre-trained mdoels, conduct the inference by
```
lightner decode -m MODEL_FILE -i INPUT_FILE -o OUTPUT_FILE
```
You can find more options by:
```
lightner decode -h
```
The current accepted paper format is as below (tokenized by line break and ```-DOCSTART-``` is optional):
```
-DOCSTART-
Ronaldo
won
't
score
more
30
goals
for
Juve
.
```
The output would be:
```
Ronaldo won 't score more than 30 goals for Juve .
```