https://github.com/mbforbes/textmetrics
Automatic text metrics (BLEU, ROUGE, METEOR, +++)
https://github.com/mbforbes/textmetrics
bleu meteor metrics ngrams nlp rouge text text-metrics vocab vocabulary
Last synced: 5 months ago
JSON representation
Automatic text metrics (BLEU, ROUGE, METEOR, +++)
- Host: GitHub
- URL: https://github.com/mbforbes/textmetrics
- Owner: mbforbes
- License: mit
- Created: 2018-02-21T17:55:31.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2018-08-07T23:46:08.000Z (almost 8 years ago)
- Last Synced: 2025-07-29T10:31:04.022Z (11 months ago)
- Topics: bleu, meteor, metrics, ngrams, nlp, rouge, text, text-metrics, vocab, vocabulary
- Language: Python
- Homepage:
- Size: 63.7 MB
- Stars: 5
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# textmetrics
Automatic text metrics---BLEU, ROUGE, and METEOR, plus extras like vocab and
ngrams.
## Usage
```bash
# Compares each candidate (c) separately against all references (r).
python -m textmetrics.main c1.txt c2.txt --references r1.txt r2.txt r3.txt
```
## Installation
Requires:
- Perl (for BLEU)
- Java 1.8 (for METEOR)
- Python 3.6+
```bash
pip install textmetrics
```
## Features
- [x] BLEU
- [x] ROUGE
- [x] METEOR
## Notes
BLEU and METEOR use the refernce implementations (in Perl and Java,
respectively). We originally used the reference Perl implementation for ROUGE
as well, but it ran so slowly that we opted for a Python reimplementation
instead. (ROUGE's original Perl implementation is also more difficult to setup,
even with wrapper libraries.)
## Worklist
- [ ] pypi
- [ ] API support (possible to have interface for passing strings?)
- [ ] ROUGE crashes things if it decides there aren't sentences (e.g., run with
README.md as input and reference)
- [ ] Add back in orig ROUGE for completeness (place behind switch)
- [ ] BLEU perl script fails if the filename ends in `gz` because it tries to
un-gzip it, which happens eventually when creating a lot of files. we should
wrap the filename creation so this doesn't happen
- [ ] ngrams has divide by zero error. With two simple files (two lines each,
same first line, differing second line) running with `2.txt --references
1.txt 1.txt` triggered this divide by zero
- [ ] Demo + guide for better README (should cover file + API usage)
- [ ] Tests
- [ ] Early check in each module for whether program runnable + nice error
message (e.g., no java or bad version, no perl or bad version, etc.)
Note to self: I followed this guide for [packaging to
pypi](https://packaging.python.org/tutorials/packaging-projects/), and future
uploads will probably look like:
```bash
# (1) ensure tests pass
# (2) bump version in setup.py
# (3) commit + push to github
# (4) generate distribution
python setup.py sdist bdist_wheel
# (5) Upload
twine upload dist/*
```