https://github.com/rhsimplex/spectrm-challenge-ryan
My work for the Spectrm challenge
https://github.com/rhsimplex/spectrm-challenge-ryan
Last synced: about 1 month ago
JSON representation
My work for the Spectrm challenge
- Host: GitHub
- URL: https://github.com/rhsimplex/spectrm-challenge-ryan
- Owner: rhsimplex
- Created: 2016-09-10T17:15:55.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2016-09-23T11:13:59.000Z (over 9 years ago)
- Last Synced: 2025-07-15T07:40:10.967Z (11 months ago)
- Language: Jupyter Notebook
- Homepage:
- Size: 3.34 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Spectrm Challenge
My entry to the [Spectrm Challenge](https://github.com/cod3licious/spectrm-challenge). Uses a very simple word-frequencey model and achieves recall rates around 5-8%. For an explanation on how it works, [see the Jupyter notebook.](https://github.com/rhsimplex/spectrm-challenge-ryan/blob/master/Word%20Frequency%20Model.ipynb)
## Setup
`spectrm-challenge-ryan` only requires basic scientific python dependencies (numpy, scipy, pandas, matplotlib, nltk). I recommend using a pre-packaged distribution like Anaconda for multiprocessing/memory efficiency. However, you can use `pip` to set up the dependencies:
```bash
$ pip install -r requirements.txt
```
## Running the model
Note: Though the matrices are pretty sparse, I used the regular `numpy` matrix implementation. That means generating a model can be quite memory intensive. All my tests were on a desktop with 10 cores and 16GB of memory. A nice improvement would to have this code use the `scipy` sparse matrix implementation.
To run the model on the unlabeled examples, simply run:
```bash
$ python match_dialogs.py
```
It knows the default locations. If you want to run on other datasets, for instance the training set, you can specify:
```bash
$ python match_dialogs.py challenge_data/train_dialogs.txt challenge_data/train_missing.txt
```