https://github.com/touhi99/n-gram-language-model
Programming for NLP Project - Implement a basic n-gram language model and generate sentence using beam search
https://github.com/touhi99/n-gram-language-model
beam-search ngram ngram-language-model perplexity
Last synced: about 1 year ago
JSON representation
Programming for NLP Project - Implement a basic n-gram language model and generate sentence using beam search
- Host: GitHub
- URL: https://github.com/touhi99/n-gram-language-model
- Owner: touhi99
- Created: 2019-01-28T23:20:59.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2020-03-10T14:32:16.000Z (over 6 years ago)
- Last Synced: 2025-03-29T18:21:57.093Z (over 1 year ago)
- Topics: beam-search, ngram, ngram-language-model, perplexity
- Language: Python
- Homepage:
- Size: 1.2 MB
- Stars: 12
- Watchers: 2
- Forks: 5
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## N-gram Language Model
### Data
- train_shakespeare.txt (train file)
- dev_shakespeare.txt (test file)
- new_shakespeare.txt (generated file, based on bigram, beam size 30)
### Files
- main.py
- corpus.py
- lm.py
- beam.py
- test_ngram.py
### Model
- trained_model_ngram.pkl (Saved after training)
### Requirement
`pip install docopt`
`pip install click`
### How-to
`python
main.py train [--n ] [--path ]
main.py generate [--lines ]
main.py perplexity [--path ]
main.py common [--number ]
main.py testcase
main.py --help`
`Options:
--n Number of n-gram
--path Train/Test file path
--lines No. of lines to be generated
--number No. of n-gram to show
--help Show this screen`
`python test_ngram.py`