https://github.com/touhi99/n-gram-language-model

Programming for NLP Project - Implement a basic n-gram language model and generate sentence using beam search
https://github.com/touhi99/n-gram-language-model

beam-search ngram ngram-language-model perplexity

Last synced: about 5 hours ago
JSON representation

Programming for NLP Project - Implement a basic n-gram language model and generate sentence using beam search

• Host: GitHub
• URL: https://github.com/touhi99/n-gram-language-model
• Owner: touhi99
• Created: 2019-01-28T23:20:59.000Z (almost 6 years ago)
• Default Branch: master
• Last Pushed: 2020-03-10T14:32:16.000Z (over 4 years ago)
• Last Synced: 2023-11-26T21:28:18.153Z (12 months ago)
• Topics: beam-search, ngram, ngram-language-model, perplexity
• Language: Python
• Homepage:
• Size: 1.2 MB
• Stars: 13
• Watchers: 3
• Forks: 5
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
## N-gram Language Model

### Data

- train_shakespeare.txt (train file)

- dev_shakespeare.txt (test file)

- new_shakespeare.txt (generated file, based on bigram, beam size 30)

### Files

- main.py

- corpus.py

- lm.py

- beam.py

- test_ngram.py

### Model

- trained_model_ngram.pkl (Saved after training)

### Requirement

	`pip install docopt`

	`pip install click`

### How-to

`python

  main.py train [--n ] [--path ]

  main.py generate [--lines ] 

  main.py perplexity [--path ]

  main.py common [--number ]

  main.py testcase

  main.py --help`

`Options:

  --n      		Number of n-gram

  --path      Train/Test file path

  --lines 		No. of lines to be generated

  --number 		No. of n-gram to show

  --help     		Show this screen`

`python test_ngram.py`