{"id":13427124,"url":"https://github.com/graykode/nlp-tutorial","last_synced_at":"2025-05-13T19:02:54.418Z","repository":{"id":37484219,"uuid":"164852596","full_name":"graykode/nlp-tutorial","owner":"graykode","description":"Natural Language Processing Tutorial for Deep Learning Researchers","archived":false,"fork":false,"pushed_at":"2024-02-21T13:49:10.000Z","size":361,"stargazers_count":14566,"open_issues_count":37,"forks_count":3966,"subscribers_count":287,"default_branch":"master","last_synced_at":"2025-04-27T08:33:18.685Z","etag":null,"topics":["attention","bert","natural-language-processing","nlp","paper","pytorch","tensorflow","transformer","tutorial"],"latest_commit_sha":null,"homepage":"https://www.reddit.com/r/MachineLearning/comments/amfinl/project_nlptutoral_repository_who_is_studying/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/graykode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-09T11:44:20.000Z","updated_at":"2025-04-27T02:46:22.000Z","dependencies_parsed_at":"2023-02-17T08:45:46.517Z","dependency_job_id":"5c6abd11-ebf1-4b5e-95da-6c43324b7038","html_url":"https://github.com/graykode/nlp-tutorial","commit_stats":{"total_commits":62,"total_committers":14,"mean_commits":4.428571428571429,"dds":"0.22580645161290325","last_synced_commit":"d05e31ec81d56d70c1db89b99ab07e948f7ebc11"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fnlp-tutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fnlp-tutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fnlp-tutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fnlp-tutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/graykode","download_url":"https://codeload.github.com/graykode/nlp-tutorial/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254010792,"owners_count":21998993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","bert","natural-language-processing","nlp","paper","pytorch","tensorflow","transformer","tutorial"],"created_at":"2024-07-31T00:01:54.429Z","updated_at":"2025-05-13T19:02:54.394Z","avatar_url":"https://github.com/graykode.png","language":"Jupyter Notebook","readme":"## nlp-tutorial\n\n\u003cp align=\"center\"\u003e\u003cimg width=\"100\" src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/1/11/TensorFlowLogo.svg/225px-TensorFlowLogo.svg.png\" /\u003e  \u003cimg width=\"100\" src=\"https://media-thumbs.golden.com/OLqzmrmwAzY1P7Sl29k2T9WjJdM=/200x200/smart/golden-storage-production.s3.amazonaws.com/topic_images/e08914afa10a4179893eeb07cb5e4713.png\" /\u003e\u003c/p\u003e\n\n`nlp-tutorial` is a tutorial for who is studying NLP(Natural Language Processing) using **Pytorch**. Most of the models in NLP were implemented with less than **100 lines** of code.(except comments or blank lines)\n\n- [08-14-2020] Old TensorFlow v1 code is archived in [the archive folder](archive). For beginner readability, only pytorch version 1.0 or higher is supported.\n\n\n## Curriculum - (Example Purpose)\n\n#### 1. Basic Embedding Model\n\n- 1-1. [NNLM(Neural Network Language Model)](1-1.NNLM) - **Predict Next Word**\n  - Paper -  [A Neural Probabilistic Language Model(2003)](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)\n  - Colab - [NNLM.ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/1-1.NNLM/NNLM.ipynb)\n- 1-2. [Word2Vec(Skip-gram)](1-2.Word2Vec) - **Embedding Words and Show Graph**\n  - Paper - [Distributed Representations of Words and Phrases\n    and their Compositionality(2013)](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)\n  - Colab - [Word2Vec.ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/1-2.Word2Vec/Word2Vec_Skipgram(Softmax).ipynb)\n- 1-3. [FastText(Application Level)](1-3.FastText) - **Sentence Classification**\n  - Paper - [Bag of Tricks for Efficient Text Classification(2016)](https://arxiv.org/pdf/1607.01759.pdf)\n  - Colab - [FastText.ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/1-3.FastText/FastText.ipynb)\n\n\n\n#### 2. CNN(Convolutional Neural Network)\n\n- 2-1. [TextCNN](2-1.TextCNN) - **Binary Sentiment Classification**\n  - Paper - [Convolutional Neural Networks for Sentence Classification(2014)](http://www.aclweb.org/anthology/D14-1181)\n  - [TextCNN.ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/2-1.TextCNN/TextCNN.ipynb)\n\n\n\n#### 3. RNN(Recurrent Neural Network)\n\n- 3-1. [TextRNN](3-1.TextRNN) - **Predict Next Step**\n  - Paper - [Finding Structure in Time(1990)](http://psych.colorado.edu/~kimlab/Elman1990.pdf)\n  - Colab - [TextRNN.ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/3-1.TextRNN/TextRNN.ipynb)\n- 3-2. [TextLSTM](https://github.com/graykode/nlp-tutorial/tree/master/3-2.TextLSTM) - **Autocomplete**\n  - Paper - [LONG SHORT-TERM MEMORY(1997)](https://www.bioinf.jku.at/publications/older/2604.pdf)\n  - Colab - [TextLSTM.ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/3-2.TextLSTM/TextLSTM.ipynb)\n- 3-3. [Bi-LSTM](3-3.Bi-LSTM) - **Predict Next Word in Long Sentence**\n  - Colab - [Bi_LSTM.ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/3-3.Bi-LSTM/Bi_LSTM.ipynb)\n\n\n\n#### 4. Attention Mechanism\n\n- 4-1. [Seq2Seq](4-1.Seq2Seq) - **Change Word**\n  - Paper - [Learning Phrase Representations using RNN Encoder–Decoder\n    for Statistical Machine Translation(2014)](https://arxiv.org/pdf/1406.1078.pdf)\n  - Colab - [Seq2Seq.ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/4-1.Seq2Seq/Seq2Seq.ipynb)\n- 4-2. [Seq2Seq with Attention](4-2.Seq2Seq(Attention)) - **Translate**\n  - Paper - [Neural Machine Translation by Jointly Learning to Align and Translate(2014)](https://arxiv.org/abs/1409.0473)\n  - Colab - [Seq2Seq(Attention).ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/4-2.Seq2Seq(Attention)/Seq2Seq(Attention).ipynb)\n- 4-3. [Bi-LSTM with Attention](4-3.Bi-LSTM(Attention)) - **Binary Sentiment Classification**\n  - Colab - [Bi_LSTM(Attention).ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/4-3.Bi-LSTM(Attention)/Bi_LSTM(Attention).ipynb)\n\n\n\n#### 5. Model based on Transformer\n\n- 5-1.  [The Transformer](5-1.Transformer) - **Translate**\n  - Paper - [Attention Is All You Need(2017)](https://arxiv.org/abs/1706.03762)\n  - Colab - [Transformer.ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/5-1.Transformer/Transformer.ipynb), [Transformer(Greedy_decoder).ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/5-1.Transformer/Transformer(Greedy_decoder).ipynb)\n- 5-2. [BERT](5-2.BERT) - **Classification Next Sentence \u0026 Predict Masked Tokens**\n  - Paper - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding(2018)](https://arxiv.org/abs/1810.04805)\n  - Colab - [BERT.ipynb](https://colab.research.google.com/github/graykode/nlp-tutorial/blob/master/5-2.BERT/BERT.ipynb)\n\n\n\n## Dependencies\n\n- Python 3.5+\n- Pytorch 1.0.0+\n\n\n\n## Author\n\n- Tae Hwan Jung(Jeff Jung) @graykode\n- Author Email : nlkey2022@gmail.com\n- Acknowledgements to [mojitok](http://mojitok.com/) as NLP Research Internship.\n","funding_links":[],"categories":["Jupyter Notebook","BERT tutorials:","NPL (Natural Language Processing)","NLP","Table of Contents","其他_NLP自然语言处理","Artificial Intelligence","Natural Language Processing ##","Blog Articles, Papers, Case Studies","Uncategorized","Course and Tutorial"],"sub_categories":["Course","其他_文本生成、文本对话","PHP","Deep Learning","Uncategorized"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraykode%2Fnlp-tutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgraykode%2Fnlp-tutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraykode%2Fnlp-tutorial/lists"}