Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/brianspiering/awesome-dl4nlp

A curated list of awesome Deep Learning (DL) for Natural Language Processing (NLP) resources
https://github.com/brianspiering/awesome-dl4nlp

List: awesome-dl4nlp

Last synced: 10 days ago
JSON representation

A curated list of awesome Deep Learning (DL) for Natural Language Processing (NLP) resources

Awesome Lists containing this project

README 

        
Awesome Deep Learning for Natural Language Processing (NLP) [![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome)

====



Table of Contents

----



- __[Courses](#courses)__

- __[Books](#books)__

- __[Tutorials](#tutorials)__

- __[Talks / Lectures](#talks)__

- __[Frameworks / Models](#frameworks)__

- __[Papers](#papers)__

- __[Blog Posts](#blog-posts)__

- __[Datasets](#datasets)__

- __[Word Embeddings / Word Vectors](#word-embeddings)__

- __[Contributing](#contributing)__



Courses

----

1. NLP with Deep Learning / CS224N from Stanford (Winter 2019)

	- [Course homepage](http://web.stanford.edu/class/cs224n/index.html) A complete survey of the field with videos, lecture slides, and sample student projects.

	- [Course lectures](https://www.youtube.com/watch?v=8rXD5-xhemo&list=PLoROMvodv4rOhcuXMZkNm7j3fVwBBY42z) Video playlist.

	- [Previous course notes](https://github.com/stanfordnlp/cs224n-winter17-notes) Probably the best "book" on DL for NLP.

	- [Course code](https://github.com/DSKSD/DeepNLP-models-Pytorch) Pytorch implementations of various Deep NLP models in cs-224n.

1. Neural Networks for NLP from Carnegie Mellon University

	- [Course homepage](http://phontron.com/class/nn4nlp2017/)

	- [Course lectures](https://www.youtube.com/user/neubig/videos)

	- [Course code](https://github.com/neubig/nn4nlp2017-code/)

1. Deep Learning for Natural Language Processing from University of Oxford and DeepMind

	- [Course homepage](https://www.cs.ox.ac.uk/teaching/courses/2016-2017/dl/)

	- [Course slides](https://github.com/oxford-cs-deepnlp-2017/lectures)

	- [Course lectures](https://www.youtube.com/playlist?list=PL613dYIGMXoZBtZhbyiBqb0QtgK6oJbpm)



Books

-----

1. [Deep Learning with Text: Natural Language Processing (Almost) from Scratch with Python and spaCy](https://www.amazon.com/Deep-Learning-Text-Approach-Processing/dp/1491984414) by Patrick Harrison and Matthew Honnibal 

1. [Neural Network Methods in Natural Language Processing](https://www.amazon.com/gp/product/1627052984) by Yoav Goldberg and Graeme Hirst

1. [Deep Learning in Natural Language Processing](http://www.springer.com/us/book/9789811052088) by Li Deng and Yang Liu

1. [Natural Language Processing in Action](https://www.manning.com/books/natural-language-processing-in-action) by Hobson Lane, Cole Howard, and Hannes Hapke

1. Deep Learning: Natural Language Processing in Python by The LazyProgrammer (Kindle only)

	1. [Word2Vec and Word Embeddings in Python and Theano](https://www.amazon.com/Deep-Learning-Language-Processing-Embeddings-ebook/dp/B01KQ0ZN0A)

	1. [From Word2Vec to GLoVe in Python and Theano](https://www.amazon.com/Deep-Learning-Language-Processing-Word2Vec-ebook/dp/B01KRBOO4Y/)

	1. [Recursive Neural Networks: Recursive Neural (Tensor) Networks in Theano](https://www.amazon.com/Deep-Learning-Language-Processing-Recursive-ebook/dp/B01KS5AEXO)

1. [Applied Natural Language Processing with Python](https://www.amazon.ca/Applied-Natural-Language-Processing-Python/dp/1484237323) by  Taweh Beysolow II

1. [Deep Learning Cookbook](https://www.amazon.ca/Deep-Learning-Cookbook-Practical-Recipes/dp/149199584X) by Douwe Osinga

1. [Deep Learning for Natural Language Processing: Creating Neural Networks with Python](https://www.amazon.ca/Deep-Learning-Natural-Language-Processing/dp/148423684X) by Palash Goyal, Sumit Pandey, Karan Jain

1. [Machine Learning for Text](https://www.amazon.ca/Machine-Learning-Text-Charu-Aggarwal/dp/3319735306) by Charu C. Aggarwal

1. [Natural Language Processing with TensorFlow](https://www.amazon.ca/Natural-Language-Processing-TensorFlow-language-ebook/dp/B077Q3VZFR) by Thushan Ganegedara

1. [fastText Quick Start Guide: Get started with Facebook's library for text representation and classification](https://www.amazon.ca/fastText-Quick-Start-Guide-representation/dp/1789130999)

1. [Hands-On Natural Language Processing with Python](https://www.amazon.ca/Hands-Natural-Language-Processing-Python/dp/178913949X)

1. [Natural Language Processing in Action, Seond Edition](https://www.manning.com/books/natural-language-processing-in-action-second-edition) by Hobson Lane and Maria Dyshel

1. [Getting Started with Natural Language Processing in Action](https://www.manning.com/books/getting-started-with-natural-language-processing) by Ekaterina Kochmar

2. [Deep Learning for Natural Language Processing in Action](https://www.manning.com/books/deep-learning-for-natural-language-processing) by Stephan Raaijmakers



Tutorials

-----



1. [Text classification guide from Google](https://developers.google.com/machine-learning/guides/text-classification/)

1. [Deep Learning for NLP with PyTorch](https://pytorch.org/tutorials/beginner/deep_learning_nlp_tutorial.html)



Talks

----

1. [Deep Learning for Natural Language Processing (without Magic)](http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutorial)

1. [A Primer on Neural Network Models for Natural Language Processing](https://arxiv.org/abs/1510.00726)

1. [Deep Learning for Natural Language Processing: Theory and Practice (Tutorial)](https://www.microsoft.com/en-us/research/publication/deep-learning-for-natural-language-processing-theory-and-practice-tutorial/)

1. [TensorFlow Tutorials](https://www.tensorflow.org/tutorials/mandelbrot)

1. [Practical Neural Networks for NLP](https://github.com/clab/dynet_tutorial_examples) from EMNLP 2016 using DyNet framework

1. [Recurrent Neural Networks with Word Embeddings](http://deeplearning.net/tutorial/rnnslu.html)

1. [LSTM Networks for Sentiment Analysis](http://deeplearning.net/tutorial/lstm.html)

1. [TensorFlow demo using the Large Movie Review Dataset](http://ai.stanford.edu/~amaas/data/sentiment/)

1. [LSTMVis: Visual Analysis for Recurrent Neural Networks](http://lstm.seas.harvard.edu/client/index.html)

1. Using deep learning in natural language processing by Rob Romijnders from PyData Amsterdam 2017

	- [video](https://www.youtube.com/watch?v=HVdPWoZ_swY)

	- [slides](https://github.com/RobRomijnders/talks/blob/master/pydata_DL_NLP.pdf)

1. [Richard Socher's talk on sentiment analysis, question answering, and sentence-image embeddings](https://www.youtube.com/watch?v=tdLmf8t4oqM)

1. [Deep Learning, an interactive introduction for NLP-ers](http://www.slideshare.net/roelofp/220115dlmeetup)

1. [Deep Natural Language Understanding](http://videolectures.net/deeplearning2016_cho_language_understanding/)

1. [Deep Learning Summer School, Montreal 2016](http://videolectures.net/deeplearning2016_montreal/) Includes state-of-art language modeling.

1. Tackling the Limits of Deep Learning for NLP by Richard Socher

	- [video](https://www.youtube.com/watch?v=JYwNmSe4HqE)

	- [slides](https://berkeley-deep-learning.github.io/cs294-131-s17/slides/socher-talk.pdf)



Frameworks

----

1. [Overview of DL frameworks for NLP](https://medium.com/@datamonsters/13-deep-learning-frameworks-for-natural-language-processing-in-python-2b84a6b6cd98)

1. General Frameworks

	1. [Keras](https://keras.io/) - _The Python Deep Learning library_ Emphasis on user friendliness, modularity, easy extensibility, and Pythonic.

	1. [TensorFlow](https://www.tensorflow.org/) - A cross-platform, general purpose Machine Intelligence library with Python and C++ API.

	1. [PyTorch](http://pytorch.org/) - PyTorch is a deep learning framework that puts Python first. "Tensors and Dynamic neural networks in Python with strong GPU acceleration."



1. Specific Frameworks

	1. [SpaCy](https://spacy.io/) - A Python package designed for speed, getting things dones, and interoperates with other Deep Learning frameworks

	1. [Genism: Topic modeling for humans](https://pypi.python.org/pypi/gensim) - A Python package that includes word2vec and doc2vec implementations.

	1. [fasttext](https://github.com/facebookresearch/fastText) Facebook's library for fast text representation and classification.

	1. Built on TensorFlow

		1. [SyntaxNet](https://github.com/tensorflow/models/tree/master/research/syntaxnet) - A toolkit for natural language understanding (NLU).

		1. [textsum](https://github.com/tensorflow/models/tree/master/research/textsum) - A Sequence-to-Sequence with Attention Model for Text Summarization.

		1. [Skip-Thought Vectors](https://github.com/tensorflow/models/tree/master/research/skip_thoughts) implementation in TensorFlow.

		1. [ActiveQA: Active Question Answering](https://github.com/google/active-qa) - Using reinforcement learning to train artificial agents for question answering

		1. [BERT](https://github.com/google-research/bert) - Bidirectional Encoder Representations from Transformers for pre-trained models

	1. Built on PyTorch

		1. [PyText](https://github.com/facebookresearch/pytext) - A deep-learning based NLP modeling framework by Facebook

		1. [AllenNLP](https://allennlp.org/) - An open-source NLP research library

		1. [Flair](https://github.com/zalandoresearch/flair) - A very simple framework for state-of-the-art NLP

		1. [fairseq](https://github.com/pytorch/fairseq) - A Sequence-to-Sequence Toolkit

		1. [fastai](http://docs.fast.ai/text.html) - Simplifies training fast and accurate neural nets using modern best practices

		1. [Transformer model](http://nlp.seas.harvard.edu/2018/04/03/attention.html) - Annotated notebook implementation

	1. [Deeplearning4j’s NLP framework](http://deeplearning4j.org/nlp) - Java implementation.

	1. [DyNet](https://github.com/clab/dynet) - _The Dynamic Neural Network Toolkit_ "work well with networks that have dynamic structures that change for every training instance".

	1. [deepnl](https://github.com/attardi/deepnl) - A Python library for NLP based on Deep Learning neural network architecture.



Papers

----

1. [Deep or shallow, NLP is breaking out](http://dl.acm.org/citation.cfm?id=2874915) - General overview of how Deep Learning is impacting NLP.

1. [Natural Language Processing from Research at Google](http://research.google.com/pubs/NaturalLanguageProcessing.html) - Not all Deep Learning (but mostly).

1. [Context Dependent Recurrent Neural Network Language Model](http://www.msr-waypoint.com/pubs/176926/rnn_ctxt.pdf)

1. [Translation Modeling with Bidirectional Recurrent Neural Networks](https://www-i6.informatik.rwth-aachen.de/publications/download/936/SundermeyerMartinAlkhouliTamerWuebkerJoernNeyHermann--TranslationModelingwithBidirectionalRecurrentNeuralNetworks--2014.pdf)

1. [Contextual LSTM (CLSTM) models for Large scale NLP tasks](https://arxiv.org/abs/1602.06291)

1. [LSTM Neural Networks for Language Modeling](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.248.4448&rep=rep1&type=pdf)

1. [Exploring the Limits of Language Modeling](http://arxiv.org/pdf/1602.02410.pdf)

1. [Conversational Contextual Cues](https://arxiv.org/abs/1606.00372) - Models context and participants in conversations.

1. [Sequence to sequence learning with neural networks](http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf)

1. [Efficient Estimation of Word Representations in Vector Space](http://arxiv.org/pdf/1301.3781.pdf)

1. [Learning Character-level Representations for Part-of-Speech Tagging](http://jmlr.org/proceedings/papers/v32/santos14.pdf)

1. [Representation Learning for Text-level Discourse Parsing](http://www.cc.gatech.edu/~jeisenst/papers/ji-acl-2014.pdf)

1. [Fast and Robust Neural Network Joint Models for Statistical Machine Translation](http://acl2014.org/acl2014/P14-1/pdf/P14-1129.pdf)

1. [Parsing With Compositional Vector Grammars](http://www.socher.org/index.php/Main/ParsingWithCompositionalVectorGrammars)

1. [Smart Reply: Automated Response Suggestion for Email](https://arxiv.org/abs/1606.04870)

1. [Neural Architectures for Named Entity Recognition](https://arxiv.org/abs/1603.01360) - State-of-the-art performance in NER with bidirectional LSTM with a sequential conditional random layer and transition-based parsing with stack LSTMs.

1. [Grammar as a Foreign Language](https://arxiv.org/abs/1412.7449) - State-of-the-art syntactic constituency parsing using generic sequence-to-sequence approach.



Blog Posts

----



1. [Natural Language Processing (NLP) progress](https://nlpprogress.com/) Tracking the most common NLP tasks, including the datasets and the current state-of-the-art 

1. [A Review of the Recent History of Natural Language Processing](http://blog.aylien.com/a-review-of-the-recent-history-of-natural-language-processing/)

1. [Deep Learning, NLP, and Representations](http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/)

1. [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)

1. [Neural Language Modeling From Scratch](http://ofir.io/Neural-Language-Modeling-From-Scratch/?a=1)

1. [Machine Learning for Emoji Trends](http://instagram-engineering.tumblr.com/post/117889701472/emojineering-part-1-machine-learning-for-emoji)

1. [Teaching Robots to Feel: Emoji & Deep Learning](http://getdango.com/emoji-and-deep-learning.html)

1. [Computational Linguistics and Deep Learning](http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239) - Opinion piece on how Deep Learning fits into the broader picture of text processing.

1. [Deep Learning NLP Best Practices](http://ruder.io/deep-learning-nlp-best-practices/index.html)

1. [7 types of Artificial Neural Networks for Natural Language Processing](https://medium.com/@datamonsters/artificial-neural-networks-for-natural-language-processing-part-1-64ca9ebfa3b2)

1. [How to solve 90% of NLP problems: a step-by-step guide](https://blog.insightdatascience.com/how-to-solve-90-of-nlp-problems-a-step-by-step-guide-fda605278e4e)

2. [7 Applications of Deep Learning for Natural Language Processing](https://machinelearningmastery.com/applications-of-deep-learning-for-natural-language-processing/)



Datasets

----

1. [Dataset from "One Billion Word Language Modeling Benchmark"](http://www.statmt.org/lm-benchmark/1-billion-word-language-modeling-benchmark-r13output.tar.gz) - Almost 1B words, already pre-processed text.

1. [Stanford Sentiment Treebank](https://nlp.stanford.edu/sentiment/treebank.html) - Fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences.

1. [Chatbot data from Kaggle](https://www.kaggle.com/samdeeplearning/deepnlp)

1. [A list of text datasets that are free/public domain in alphabetical order](https://github.com/niderhoff/nlp-datasets)

1. [Another list of text datasets that are free/public domain in reverse chronological order](https://github.com/karthikncode/nlp-datasets)

1. Question Answering datasets

	1. [Quora's Question Pairs Dataset](https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs) - Identify question pairs that have the same intent.

	1. [CMU's Wikipedia Factoid Question Answers](https://www.cs.cmu.edu/~ark/QA-data/)

	1. [DeepMind's Algebra Question Answering](https://github.com/deepmind/AQuA)

	1. [DeepMind's from CNN & DailyMail Question Answering](https://github.com/deepmind/rc-data)

	1. [Microsoft's WikiQA Open Domain Question Answering](https://www.microsoft.com/en-us/research/publication/wikiqa-a-challenge-dataset-for-open-domain-question-answering/)

	1. [Stanford Question Answering Dataset (SQuAD)](https://rajpurkar.github.io/SQuAD-explorer/) - covering reading comprehension



Word Embeddings and friends

----

1. [The amazing power of word vectors](https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/) from The Morning Paper blog

1. [Distributed Representations of Words and Phrases and their Compositionality](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) - The original word2vec paper.

1. [word2vec Parameter Learning Explained](https://arxiv.org/abs/1411.2738) An elucidating explanation of word2vec training

1. [Word embeddings in 2017: Trends and future directions](http://ruder.io/word-embeddings-2017/)

1. [Learning Word Vectors for 157 Languages](https://arxiv.org/abs/1802.06893)

1. [GloVe: Global Vectors for Word Representation](http://www-nlp.stanford.edu/pubs/glove.pdf) - A "count-based"/co-occurrence model to learn word embeddings.

1.  Doc2Vec

	- [A gentle introduction to Doc2Vec](https://medium.com/scaleabout/a-gentle-introduction-to-doc2vec-db3e8c0cce5e)

	- [Distributed Representations of Sentences and Documents](https://cs.stanford.edu/~quocle/paragraph_vector.pdf)

1. [Dynamic word embeddings for evolving semantic discovery](https://blog.acolyer.org/2018/02/22/dynamic-word-embeddings-for-evolving-semantic-discovery/) from The Morning Paper blog

1. Ali Ghodsi's lecture on word2vec: 

	- [part 1](https://www.youtube.com/watch?v=TsEGsdVJjuA)

	- [part 2](https://www.youtube.com/watch?v=nuirUEmbaJU)

1. [word2vec analogy demo](http://deeplearner.fz-qqq.net/)

1. [TensorFlow Embedding Projector of word vectors](http://projector.tensorflow.org/)

1. Skip-Thought Vectors - "unsupervised learning of a generic, distributed sentence encoder"

    - [Paper](http://arxiv.org/abs/1506.06726)

    - [Code](https://github.com/ryankiros/skip-thoughts)



-----

Contributing

----

Have anything in mind that you think is awesome and would fit in this list? Feel free to send me a pull request!



-----

License

----



[![CC0](http://i.creativecommons.org/p/zero/1.0/88x31.png)](http://creativecommons.org/publicdomain/zero/1.0/)



To the extent possible under law, [Dr. Brian J. Spiering](http://www.linkedin.com/in/brianspiering/) has waived all copyright and related or neighboring rights to this work.