Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/adamspannbauer/cs224n


https://github.com/adamspannbauer/cs224n

Last synced: about 1 month ago
JSON representation

Awesome Lists containing this project

README 

          
# Stanford CS224n: NLP with Deep Learning



## Lectures



[YouTube Playlist](https://www.youtube.com/playlist?list=PLoROMvodv4rOhcuXMZkNm7j3fVwBBY42z)



## Schedule



See schedule below for ordering, assignemnts, and additional links.  For context on pacing, the class met every Tuesday and Thursday.



Below table copied from [here](http://web.stanford.edu/class/cs224n/index.html#schedule).



  

    

    

    

    

  

  

  

    Description

    Course Materials

    Events

    Deadlines

  

  

  

  

    Word Vectors

      


      [slides]

      

      [notes]

      



      Gensim word vectors example:

      


      [code]

      [preview]

    

    

      Suggested Readings:

      
    

            
  1. 

    Efficient Estimation of Word Representations in Vector Space (original word2vec paper)
    2. 

            
  2. 

    Distributed Representations of Words and Phrases and their Compositionality (negative sampling paper)
    3. 

          


    

    

      Assignment 1 out

      


      [code]

      


      [preview]

    

    

  



  

    Word Vectors 2 and Word Window Classification

      


      [slides]

      

      [notes]

    

    

      Suggested Readings:

      
    

            
  1. 

    GloVe: Global Vectors for Word Representation (original GloVe paper)
    2. 

            
  2. Improving Distributional Similarity with Lessons Learned from Word Embeddings
    3. 

            
  3. Evaluation methods for unsupervised word embeddings
    4. 

          


      Additional Readings:

      
    

            
  1. A Latent Variable Model Approach to PMI-based Word Embeddings
    2. 

            
  2. Linear Algebraic Structure of Word Senses, with Applications to Polysemy
    3. 

            
  3. On the Dimensionality of Word Embedding
    4. 

          


    

    

    

  



  

    Python Review Session

      


      [code]

      [preview]

    

    

       10:00am - 11:20am

    

    

    

  



  

    Backprop and Neural Networks

      


      [slides]

      [notes]

    

    

      Suggested Readings:

      
    

            
  1. matrix calculus notes
    2. 

            
  2. Review of differential calculus
    3. 

            
  3. CS231n notes on network architectures
    4. 

            
  4. CS231n notes on backprop
    5. 

            
  5. Derivatives, Backpropagation, and Vectorization
    6. 

            
  6. 

    Learning Representations by Backpropagating Errors (seminal Rumelhart et al. backpropagation paper)
    7. 

          


      Additional Readings:

      
    

            
  1. Yes you should understand backprop
    2. 

            
  2. Natural Language Processing (Almost) from Scratch
    3. 

          


    

    

      Assignment 2 out

      


      [code]

      [handout]

    

    Assignment 1 due

  



  

    Dependency Parsing

      


      [slides]

      [notes]

      


      [slides (annotated)]

    

    

      Suggested Readings:

      
    

            
  1. Incrementality in Deterministic Dependency Parsing
    2. 

            
  2. A Fast and Accurate Dependency Parser using Neural Networks
    3. 

            
  3. Dependency Parsing
    4. 

            
  4. Globally Normalized Transition-Based Neural Networks
    5. 

            
  5. Universal Stanford Dependencies: A cross-linguistic typology
    6. 

    

            
  6. 

    Universal Dependencies website

    7. 

          


    

    

    

  



  

    PyTorch Tutorial Session

    


      [colab notebook]

      [preview]

    


      [jupyter notebook]

    

    

       10:00am - 11:20am

    

    

    

  



  

    Recurrent Neural Networks and Language Models

      


      [slides]

       [notes (lectures 5 and 6)]

    

    

      Suggested Readings:

      
    

            
  1. 

    N-gram Language Models (textbook chapter)
    2. 

            
  2. 

    The Unreasonable Effectiveness of Recurrent Neural Networks (blog post overview)
    3. 

            

            
  3. 

    Sequence Modeling: Recurrent and Recursive Neural Nets (Sections 10.1 and 10.2)
    4. 

           
  4. 

    On Chomsky and the Two Cultures of Statistical Learning

          
    5. 



    

    Assignment 3 out

        


        [code]

        [handout]

    

    Assignment 2 due

  



  

    Vanishing Gradients, Fancy RNNs, Seq2Seq

      


      [slides]

      [notes (lectures 5 and 6)]

    

    

      Suggested Readings:

      
    

            
  1. 

    Sequence Modeling: Recurrent and Recursive Neural Nets (Sections 10.3, 10.5, 10.7-10.12)
    2. 

            
  2. 

    Learning long-term dependencies with gradient descent is difficult (one of the original vanishing gradient papers)
    3. 

            
  3. 

    On the difficulty of training Recurrent Neural Networks (proof of vanishing gradient problem)
    4. 

            
  4. 

    Vanishing Gradients Jupyter Notebook (demo for feedforward networks)
    5. 

            
  5. 

    Understanding LSTM Networks (blog post overview)
    6. 

            

          


    

    

    

  



  

    Machine Translation, Attention, Subword Models

      


      [slides]

      [notes]

    

    

      Suggested Readings:

      
    

            
  1. 

    Statistical Machine Translation slides, CS224n 2015 (lectures 2/3/4)
    2. 

            
  2. 

    Statistical Machine Translation (book by Philipp Koehn)
    3. 

            
  3. 

    BLEU (original paper)
    4. 

            
  4. 

    Sequence to Sequence Learning with Neural Networks (original seq2seq NMT paper)
    5. 

            
  5. 

    Sequence Transduction with Recurrent Neural Networks (early seq2seq speech recognition paper)
    6. 

            
  6. 

    Neural Machine Translation by Jointly Learning to Align and Translate (original seq2seq+attention paper)
    7. 

            
  7. 

    Attention and Augmented Recurrent Neural Networks (blog post overview)
    8. 

            
  8. 

    Massive Exploration of Neural Machine Translation Architectures (practical advice for hyperparameter choices)
    9. 

            
  9. Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models
    10. 

            
  10. Revisiting Character-Based Neural Machine Translation with Capacity and Compression
    11. 

          


    

    Assignment 4 out

        


        [code]

        [handout]

        [Azure Guide]

        [Practical Guide to VMs]

    

    Assignment 3 due

  



  

    Final Projects: Custom and Default; Practical Tips

      


      [slides]

      [notes]

    

    

      Suggested Readings:

      
    

            
  1. 

    Practical Methodology (Deep Learning book chapter)
    2. 

          


    

    Project Proposal out

        


        [instructions]

        



        Default Final Project out

        


        [handout (IID SQuAD track)]

        


        [handout (Robust QA track)]

        

    

    

  



  

    Transformers (lecture by John Hewitt)

    


    [slides]

    [notes]

    

    

      Suggested Readings:

      
    

            
  1. 

    Project Handout (IID SQuAD track)

            
    2. 

            
  2. 

    Project Handout (Robust QA track)

            
    3. 

            
  3. 

    Attention Is All You Need

            
    4. 

            
  4. 

    The Illustrated Transformer

            
    5. 

            
  5. 

    Transformer (Google AI blog post)

            
    6. 

            
  6. Layer Normalization
    7. 

            
  7. Image Transformer
    8. 

            
  8. Music Transformer: Generating music with long-term structure
    9. 

          


    

    

    

  



  

    More about Transformers and Pretraining (lecture by John Hewitt)

      


      [slides]

      [notes]

    

    

    Suggested Readings:

      
    

            
  1. 

              BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

            
    2. 

            
  2. 

               Contextual Word Representations: A Contextual Introduction

            
    3. 

            
  3. The Illustrated BERT, ELMo, and co.
    4. 

          


    

    Assignment 5 out

      


      [code]

      [handout]

    

    Assignment 4 due

  



  

    Question Answering (guest lecture by Danqi Chen)

      


      [slides]

    

    

    Suggested Readings:

      
    

            
  1. 

              SQuAD: 100,000+ Questions for Machine Comprehension of Text

            
    2. 

            
  2. 

              Bidirectional Attention Flow for Machine Comprehension

            
    3. 

            
  3. 

              Reading Wikipedia to Answer Open-Domain Questions

            
    4. 

            
  4. 

              Latent Retrieval for Weakly Supervised Open Domain Question Answering

            
    5. 

            
  5. 

              Dense Passage Retrieval for Open-Domain Question Answering

            
    6. 

            
  6. 

              Learning Dense Representations of Phrases at Scale

            
    7. 

          


    

    

    Project Proposal due

  



  

    Natural Language Generation (lecture by Antoine Bosselut)

      


      [slides]

    

    

    Suggested readings:

      
    

            
  1. 

            The Curious Case of Neural Text Degeneration

            
    2. 

            
  2. 

            Get To The Point: Summarization with Pointer-Generator Networks

            
    3. 

            
  3. 

            Hierarchical Neural Story Generation

            
    4. 

            
  4. 

            How NOT To Evaluate Your Dialogue System

            
    5. 

          


    

    

    

  



  

    

    

    

    Project Milestone out

    [instructions]

    

    Assignment 5 due

  



  

      Reference in Language and Coreference Resolution

        


        [slides]

      

    

    Suggested readings:

      
    

            
  1. 

            Coreference Resolution chapter of Jurafsky and Martin

            
    2. 

            
  2. 

              End-to-end Neural Coreference Resolution.

            
    3. 

          


    

    

    

  



  

    T5 and large language models: The good, the bad, and the ugly (guest lecture by Colin Raffel)

    


    [slides]

    

    

    Suggested readings:

    
    

            
  1. 

            Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

            
    2. 

        


    

    

    

  



  

    Integrating knowledge in language models (lecture by Megan Leszczynski)

    


    [slides]

    

    

    Suggested readings:

    
    

            
  1. 

            ERNIE: Enhanced Language Representation with Informative Entities

            
    2. 

            
  2. 

            Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling

            
    3. 

            
  3. 

            Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

            
    4. 

            
  4. 

            Language Models as Knowledge Bases?

            
    5. 

        


    

    

    Project Milestone due

  



  

    Social & Ethical Considerations in NLP Systems (guest lecture by Yulia Tsvetkov)

    


    [slides]

    

    

    

    

  



  

    Model Analysis and Explanation (lecture by John Hewitt)

      


      [slides]

    

    

    

    

  



  

    Future of NLP + Deep Learning (lecture by Shikhar Murty)

      


      [slides]

    

    

    

    

  



  

    

    

    Project Summary Image and Paragraph out

    [instructions]

    

    

  



  

    Ask Me Anything / Final Project Assistance

    

    

    

      Project due

      [instructions]

    

  



  

    Final Project Emergency Assistance

    

    

    

    

  



  

    

    

    

    Project Summary Image and Paragraph due