https://github.com/i008/nyyelp
predicting yelp review rating using recurrent neural networks
https://github.com/i008/nyyelp
deep-learning nlp python recurrent-neural-networks yelp-dataset
Last synced: 12 months ago
JSON representation
predicting yelp review rating using recurrent neural networks
- Host: GitHub
- URL: https://github.com/i008/nyyelp
- Owner: i008
- Created: 2017-02-21T22:54:55.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2017-02-26T11:58:24.000Z (over 9 years ago)
- Last Synced: 2025-06-10T14:49:51.923Z (about 1 year ago)
- Topics: deep-learning, nlp, python, recurrent-neural-networks, yelp-dataset
- Language: Jupyter Notebook
- Homepage:
- Size: 18.9 MB
- Stars: 22
- Watchers: 1
- Forks: 4
- Open Issues: 0
-
Metadata Files:
- Readme: README.MD
Awesome Lists containing this project
README
# Project objective:
Predict review rating (how many stars the reviewer gave to given POI) given the textual-content of the review using pretrained **GLOVE** word embeddings and a 2 layers deep **GRU(LSTM)** recurrent neural network. Visualize clusters on a 2D space using PCA and t-SNE.
#### simple yelp-dataset exploration
yelp data: https://www.yelp.de/dataset_challenge
notebook used to load and explore basic properties of this dataset can be found here:
https://github.com/i008/nyyelp/blob/master/exploration.ipynb
#### NLP project
The notebook for the NLP part of this project can be found here:
https://github.com/i008/nyyelp/blob/master/nlp.ipynb
#### Steps(rough plan):
1) Process .text data from yelp review-documents
- filter to leave english-only reviews
- balance the dataset (use similar amount of data for each star rating (1,2,3,4,5*)
- balance the dataset by review length, so the length distribution of reviews for each rating-class is atleast similar (bucketing might be a good idea)
2) Process text for deep-learning
- bring GLOVE embeddings to a reasonable form
- Tokenize each review
- transform reviews into tokenized sequences
- Pad sequences to a fixed length
- Prepare the embedding weight-matrix to be used in the Embedding layer (we will not train this layer)
- split into test and train
3) Deep learning
- loss function, regression(rmse) and/or classification(logloss)
- prepare architecture (2 LSTM/GRU layers -> Dense)
- train, wish for the best
4) Post learning
- extract features from last LSTM layer
- casting to 2D space using t-SNE
5) Remarks
- we have to limit the amount of data, bc of memory issues possible solution is to save the sequences into HDF and flowing from disk during training
# Summary
### text Data:
- For performance and memory reasons the final model was trained on a balanced and downsampled (by a factor of 10) subset of the reviews leaving us around 150,000 reviews (in total)
- Review data was tokenized (20,000 words)
- Max sequence length was set to 1000.
- Embedding dimensions were set to 100 (GLOVE-100d representation was used) wich means every words is a vector with shape (100,)
### Deep Learning Model
- The Embedding layer used GLOVE (http://nlp.stanford.edu/projects/glove/) weights. Training of this layer was disabled
- The objective function was logloss wich means that it was a classification task , altough regression would also be a valid option
- After the 3rd-4th epoch we can observe that the model starts to overfit
### Results
fig1. confusion matrix :
fig2. classification report:
fig3. Learning process:
As we can see on the images above the model managed to learn (something) and achieved decent accuracy(~0.62).
Key takeways:
- Extreme(1,5) reviews are easier to classify and understand, people write them in a specific way.
- As we can see on fig1. The model makes mistakes almost only "by 1-star" this is a very good sign and leads to the conclusion that the "grduation" of the data was learned to some degree
### t-SNE and PCA on GRU features
t-SNE and PCA are dimmensionality reduction techniques, used on the features from last GRU layer they can give us an "2D"
idea of the problem we are tackling.
fig4. t-SNE
fig5. PCA
### Learning Performance
- One epoch (~150,000 reviews) with a batch size of 128 reviews took around 15mins to train on a GTX1070.
### Docker app (try it yourself)
```bash
docker run i008/nyyelp:latest python predict.py --review "this place really sucks, food is terrible"
```
>number of stars: 1
```bash
docker run i008/nyyelp:latest python predict.py --review "i have mixed feeling about this place, on one hand its good on the other not really"
```
>number of stars: 3
### TODO, Ideas
- Treat this problem as regression and see what happens
- Cluster GRU features using DBSCAN or similar methods, might give interesting results.
- Use more data to train the models.
- Balance the dataset considering the length of the review.
- Interactive plot for t-SNE (click on a data point should show the review plotly?)