{"id":19668432,"url":"https://github.com/pat-alt/neuralnets","last_synced_at":"2026-04-12T16:12:14.161Z","repository":{"id":119125559,"uuid":"331854878","full_name":"pat-alt/neuralNets","owner":"pat-alt","description":"A small project on the use of deep learning for natural language processing (NLP).","archived":false,"fork":false,"pushed_at":"2021-02-26T11:56:10.000Z","size":9999,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-14T05:15:54.787Z","etag":null,"topics":["deep-learning","lstm","rnn","tensorflow","word2vec"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pat-alt.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-22T06:15:57.000Z","updated_at":"2021-12-05T12:14:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"4854a9ad-7660-4322-9a1a-af944a36d0d7","html_url":"https://github.com/pat-alt/neuralNets","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pat-alt/neuralNets","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pat-alt%2FneuralNets","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pat-alt%2FneuralNets/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pat-alt%2FneuralNets/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pat-alt%2FneuralNets/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pat-alt","download_url":"https://codeload.github.com/pat-alt/neuralNets/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pat-alt%2FneuralNets/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265871377,"owners_count":23842026,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","lstm","rnn","tensorflow","word2vec"],"created_at":"2024-11-11T16:35:26.502Z","updated_at":"2026-04-12T16:12:14.115Z","avatar_url":"https://github.com/pat-alt.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n```{r setup, include=FALSE}\nknitr::opts_chunk$set(echo = FALSE, message = FALSE, warning = FALSE)\nlibrary(kableExtra)\nlibrary(data.table)\n```\n\n# neuralNets\n\nA small project on the use of deep learning for natural language processing (NLP).\n\n## Prerequisites\n\nImplementation of deep learning is done through TensorFlow. To install TensorFlow 2 see [here](https://www.tensorflow.org/install/).\n\n## NLP with deep learning\n\nThe [notebook](notebook.ipynb) trains various deep learning models to a simple, binary text classification problem. The [data set](data/data.txt) contains restaurant reviews which are labelled as either positive ($=1$) or negative ($=0$):\n\n```{r}\nkable(head(fread(\"data/data.txt\")), col.names = c(\"Review\", \"Label\"))\n```\n\nBelow you may find a high-level overview of the motivation, methodology and results. All code along with comments is contained in the [notebook](notebook.ipynb) which is divided into the following sections:\n\n1. Load and preprocess data \n2. A first run\n3. Model selection through hyperparameter tuning\n4. Final predictions\n\n### Methodology\n\nBoth a convolutional neural network (CNN) and several recurrent neural networks (RNN) are trained to word-wise and and character-wise classification of the labels. Both deep learning approaches rely on an unsupervised neural language model at the first stage, which learns the word vectors  -- essentially a representation of the text in vector form. The whole process can be summarized broadly as follows: a corpus of text $\\mathbf{X}^{'}$ is fed to a `word2vec` model. The output $\\mathbf{X}$ and corresponding vector of target labels $\\mathbf{y}$ is then fed to a supervised deep learning model, which produces the final predictions $\\mathbf{\\hat{y}}$.\n\n![](www/process_flow.png)\n\nFor model selection through hyperparameter tuning a simple grid search framework is built from scratch.^[Alternatively, one can perform grid search through existing libraries, for example, `sklearn` or `ray`. The latter was tested, but I ran into issues. The low-level custom grid search ultimately worked very well.]\n\n#### Embeddings\n\nWith respect to the word embeddings three different approaches are used: *random* -- words are randomly initialized, *non-static* -- embeddings are pre-trained through `word2vec` and fine-tuned during training, and *GloVe* -- pre-trained global word vectors are used. The latter are obtained from [GloVe](https://nlp.stanford.edu/projects/glove/). In particular, the 50-$d$ Wikipedia 2014 + Gigaword 5 vectors are used here. \n\n#### CNN\n\nConvolutional neural networks were originally invented in the context of computer vision, but have subsequently been successfully applied to NLP. CNNs involve layers of filters that convolve over a tensor of input features -- in the animation below this is illustrated as the green filter sliding over the blue matrix of inputs^[Animation of convoluting filter in action. Source: [towards data science](https://towardsdatascience.com/applied-deep-learning-part-4-convolutional-neural-networks-584bc134c1e2)]. In the underlying case of text classification the filters convolve over the padding sequences contained in $\\mathbf{X}$ and produce a tensor of features (red matrix in the animation below) which is then fed to the subsequent dense layers of the main sequential model. Finally a sigmoid layer produces a probability distribution over labels.\n\n![](www/cnn.gif)\n\n#### RNN\n\nRecurrent neural networks are based on the idea of persistent thoughts: thinking is modelled as a continuous process that instead of continuously reinventing itself and starting from scratch, evolves gradually and at each step uses information about its prior states. This hierarchical, chain-like nature of RNNs makes them particularly useful for problems that involves sequences, for example, speech recognition or time series analysis. The former is the focus of this small project, so let us dwell on this a little further. Consider the sentence fragment, which is lifted directly from the [data set](data/data.txt) we treat in this project: \"[...] will not meet your expectations.\" Without further context, if you had to classify the sentiment of this text fragment, you would probably label it as negative. When the full sequence of words is revealed the label switches to positive: \"Great Subway, in fact it's so good when you come here every other Subway will not meet your expectations.\" This demonstrates the importance of using prior information that emerges from the context. The sentence also demonstrates that it can be difficult to learn the *context* of a single word just from its nearest neighbours: to understand the role of the word \"expectations\" in the context of this sentence, it is not enough to look at a few words preceding it. In fact, as we saw above, too small a choice of the context window may lead to wrong conclusions about the sentiment label. In order to account for long-term dependencies we can use a Long Short Term Memory (LSTM) network, a special kind of RNN.\n\n## Results\n\nThe best CNN and RNN classifiers have been serialized and saved to disk (`cnn.json` and `rnn.json`, respectively). To load and use a model and its corresponding weights, you can simply run the below from within the root folder of this repository:\n\n```{r, echo=FALSE}\nlibrary(reticulate)\n```\n\n```{python, echo=TRUE, eval=FALSE}\n# load json and create model\njson_file = open('cnn.json', 'r')\nloaded_model_json = json_file.read()\njson_file.close()\nloaded_model = model_from_json(loaded_model_json)\n# load weights into new model\nloaded_model.load_weights(\"cnn.h5\")\n```\n\nBoth models ended up with roughly the same overall accuracy, both reaching perfect in-sample accuracy after only 2-3 epochs and an average validation accuracy of \u003e80% over the first 10 epochs. \n\nIn terms of hyper parameter tuning and model architecture the following observations can be made: \n\n- **CNN**: the best model in terms of validation accuracy used 2 filter sizes, 100 filters and just 1 hidden layer of 150 neurons. In terms of word embeddings, best performance was achieved using GloVe. \n\n- **RNN**: the best model turns out to be a Gated Recurrent Unit (GRU). GRUs are very similar to LSTMs and generally have been found to yield similar performance when applied to NLP. For small data sets, as we have here, GRUs often outperform LSTMs. The optimal number of hidden neurons identified by the grid search is 100. Contrary to the results for the CNN above, we find that in this case the randomly initialized word vectors outperform the pre-trained GloVe vectors.\n\n```{r}\ndt_pred \u003c- fread(\"predictions.csv\")[,V1:=NULL]\ncor_models \u003c- cor(dt_pred$pred_cnn, dt_pred$pred_rnn)\n```\n\nSince independently of each other both the CNN and RNN produce perfect in-sample predictions of the discrete labels, it is not surprising that the correlation between their respective probabilistic predictions is `r round(cor_models,2)*100` percent. The figure below illustrates this further. It plots the twenty most frequent words occurring in sentences that were labelled as either positive (=1) or negative (=0) by the CNN and RNN, respectively^[Stop words have been removed.]. Evidently there are certain similarities: both models appear to have learned that negative reviews tend revolve around timing and positive reviews tend to be packed with positive attributes describing either staff, food or the atmosphere.\n\n```{r clouds, fig.width=10, fig.height=5}\nlibrary(ggwordcloud)\nlibrary(tidytext)\nlibrary(dplyr)\nstop_words \u003c- tidytext::stop_words\nstop_words \u003c- rbind(\n  data.table(stop_words),\n  data.table(word=c(\"food\", \"service\", \"restaurant\")), \n  fill=TRUE\n)\ndt_plot \u003c- dt_pred %\u003e%\n    tidytext::unnest_tokens(word, sentence) %\u003e%\n    dplyr::anti_join(stop_words)\ndt_plot \u003c- melt(\n  dt_plot, \n  id.vars = c(\"word\", \"true\"), \n  measure.vars = c(\"pred_cnn\", \"pred_rnn\"), \n  value.name = \"sentiment\",\n  variable.name = \"model\"\n)\ndt_plot \u003c- unique(dt_plot[,.(.N, sentiment=mean(sentiment)), by=.(word, true, model)])\nsetorder(dt_plot, true, model, -N)\ndt_plot \u003c- dt_plot[,idx:=1:.N,by=.(true,model)][idx\u003c=20][,idx:=NULL]\nlevels(dt_plot$model) \u003c- c(\"CNN\", \"RNN\")\ndt_plot[,sentiment:=factor(round(sentiment))]\nggplot(dt_plot, ggplot2::aes(label = word, size = N, colour=sentiment)) +\n  geom_text_wordcloud() +\n  scale_size_area(max_size = 5) +\n  scale_colour_manual(\n    guide=FALSE,\n    values = c(\"coral\", \"darkgreen\")\n  ) +\n  facet_grid(\n    cols = vars(model),\n    rows = vars(sentiment)\n  )\n```\n\n# Notes\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpat-alt%2Fneuralnets","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpat-alt%2Fneuralnets","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpat-alt%2Fneuralnets/lists"}