{"id":13688505,"url":"https://github.com/vlgiitr/DL_Topics","last_synced_at":"2025-05-01T19:31:04.086Z","repository":{"id":53908904,"uuid":"158021710","full_name":"vlgiitr/DL_Topics","owner":"vlgiitr","description":"List of DL topics and resources essential for cracking interviews","archived":false,"fork":false,"pushed_at":"2023-01-14T08:48:31.000Z","size":163,"stargazers_count":443,"open_issues_count":2,"forks_count":59,"subscribers_count":20,"default_branch":"master","last_synced_at":"2024-11-12T12:47:38.007Z","etag":null,"topics":["computer-vision","deep-learning","generative-models","linear-algebra","natural-language-processing","probability-statistics"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vlgiitr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-17T20:24:03.000Z","updated_at":"2024-11-03T21:38:29.000Z","dependencies_parsed_at":"2023-02-09T19:00:29.929Z","dependency_job_id":null,"html_url":"https://github.com/vlgiitr/DL_Topics","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlgiitr%2FDL_Topics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlgiitr%2FDL_Topics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlgiitr%2FDL_Topics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlgiitr%2FDL_Topics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vlgiitr","download_url":"https://codeload.github.com/vlgiitr/DL_Topics/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251932643,"owners_count":21667186,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","generative-models","linear-algebra","natural-language-processing","probability-statistics"],"created_at":"2024-08-02T15:01:15.551Z","updated_at":"2025-05-01T19:31:03.613Z","avatar_url":"https://github.com/vlgiitr.png","language":null,"funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"# Deep Learning Topics and Resources\n\n\u003cdiv style=\"text-align: center;\"\u003e\n\u003cimg src=\"./description.png\" alt=\"description\" width=\"800\"\u003e\n\u003c/div\u003e\n\n## Resources for DL in General\n\n1. **Blogs**\n    - Lilian Weng’s Blog [[link](https://lilianweng.github.io/)]\n    - AI Summer Blog [[link](https://theaisummer.com/learn-ai/)]\n    - Colah’s Blog [[link](https://colah.github.io/)]\n2. **Books**\n    - Neural Networks and Deep Learning [[link](http://neuralnetworksanddeeplearning.com/)]\n    - Deep Learning Book [[link](https://www.deeplearningbook.org/)]\n    - Dive into Deep Learning [[link](https://d2l.ai/)]\n    - Reinforcement Learning: An Introduction | Sutton and Barto [[link](http://www.incompleteideas.net/book/the-book-2nd.html)]\n3. **Open Courses**\n    - CS-229 Machine Learning Stanford | Andrew Ng [[youtube](https://youtube.com/playlist?list=PLoROMvodv4rMiGQp3WXShtMGgzqpfVfbU)] [[website](https://cs229.stanford.edu/)]\n    - CS-231n Computer Vision Stanford [[youtube](https://youtube.com/playlist?list=PLkt2uSq6rBVctENoVBg1TpCC7OQi31AlC)] [[website](http://cs231n.stanford.edu/)]\n    - CS-224n Natural Language Processing [[youtube](https://youtube.com/playlist?list=PLoROMvodv4rOSH4v6133s9LFPRHjEmbmJ)] [[website](https://web.stanford.edu/class/cs224n/)]\n    - Introduction to Reinforcement Learning with David Silver [[youtube](https://youtube.com/playlist?list=PLqYmG7hTraZBKeNJ-JE_eyJHZ7XgBoAyb)] [[website](https://www.deepmind.com/learning-resources/introduction-to-reinforcement-learning-with-david-silver)]\n\n## Mathematics\n\n1. Linear Algebra ([[notes](http://cs229.stanford.edu/section/cs229-linalg.pdf)][[practice questions](https://www.geeksforgeeks.org/linear-algebra-gq/)])\n    - 3Blue1Brown essence of linear algebra [[youtube](https://youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab)]\n    - Gilbert Strang’s lectures on Linear Algebra [[link](https://ocw.mit.edu/courses/18-06-linear-algebra-spring-2010/)] [[youtube](https://youtube.com/playlist?list=PL49CF3715CB9EF31D)]\n    - Topics\n        - Linear Transformations\n        - Linear Dependence and Span\n        - Eigendecomposition - Eigenvalues and Eigenvectors\n        - Singular Value Decomposition [[blog](https://medium.com/vlgiitr/eli5-singular-value-decomposition-svd-955c151f9907)]\n        \n2. Probability and Statistics ([[notes](http://www.mxawng.com/stuff/notes/stat110.pdf)][[youtube series](https://www.youtube.com/user/joshstarmer)])\n    - Harvard Statistics 110: Probability [[link](https://projects.iq.harvard.edu/stat110/home)] [[youtube](https://youtube.com/playlist?list=PL2SOU6wwxB0uwwH80KTQ6ht66KWxbzTIo)]\n    - Topics\n        - Expectation, Variance, and Co-variance\n        - Distributions\n        - Random Walks\n        - Bias and Variance\n            - Bias Variance Trade-off\n        - Estimators\n            - Biased and Unbiased\n        - Maximum Likelihood Estimation [[blog](https://towardsdatascience.com/probability-concepts-explained-maximum-likelihood-estimation-c7b4342fdbb1)]\n        - Maximum A-Posteriori (MAP) Estimation [[blog](https://towardsdatascience.com/probability-concepts-explained-bayesian-inference-for-parameter-estimation-90e8930e5348)]\n        \n3. Information Theory [[youtube](https://www.youtube.com/watch?v=ErfnhcEV1O8)]\n    - (Shannon) Entropy [[blog](https://towardsdatascience.com/information-entropy-c037a90de58f)]\n    - Cross Entropy, KL Divergence [[blog](https://towardsdatascience.com/entropy-cross-entropy-and-kl-divergence-explained-b09cdae917a)]\n    - KL Divergence\n        - Not a distance metric (unsymmetric)\n        - Derivation from likelihood ratio ([Blog](https://medium.com/@cotra.marko/making-sense-of-the-kullback-leibler-kl-divergence-b0d57ee10e0a))\n        - Always greater than 0\n            - Proof by Jensen's inequality ([Stack Overflow Link](https://stats.stackexchange.com/a/335201))\n        - Relation with Entropy ([Explanation](https://stats.stackexchange.com/questions/265966/why-do-we-use-kullback-leibler-divergence-rather-than-cross-entropy-in-the-t-sne))\n\n## Basics\n\n1. Neural Networks Overview [[youtube](https://youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi)]\n2. Backpropogation\n    - Vanilla [[blog](http://cs231n.github.io/optimization-2/)]\n    - Backpropagation in CNNs [[blog](https://pavisj.medium.com/convolutions-and-backpropagations-46026a8f5d2c)]\n    - Backprop through time [[blog](https://towardsdatascience.com/backpropagation-in-rnn-explained-bdf853b4e1c2)]\n3. Loss Functions\n    - MSE Loss\n        - Derivation by MLE and MAP\n    - Cross Entropy Loss\n        - Binary Cross Entropy\n        - Categorical Cross Entropy\n4. Activation Functions (Sigmoid, Tanh, ReLU and variants) ([blog](https://mlfromscratch.com/activation-functions-explained/))\n5. Optimizers \n6. Regularization\n    - Early Stopping\n    - Noise Injection\n    - Dataset Augmentation\n    - Ensembling\n    - Parameter Norm Penalties\n        - L1 (sparsity)\n        - L2 (smaller parameter values)\n    - BatchNorm [[Paper](https://github.com/vlgiitr/DL_Topics/blob/master)]\n        - Internal Covariate Shift\n        - BatchNorm in CNNs [[Link](https://stackoverflow.com/questions/38553927/batch-normalization-in-convolutional-neural-network)]\n        - Backprop through BatchNorm Layer [[Explanation](https://kratzert.github.io/2016/02/12/understanding-the-gradient-flow-through-the-batch-normalization-layer.html)]\n    - Dropout Regularization [[Paper](https://github.com/vlgiitr/DL_Topics/blob/master)]\n\n## Computer Vision\n\n1. Convolution [[youtube](https://youtu.be/8rrHTtUzyZA)]\n    - Cross-correlation\n    - Pooling (Average, Max Pool)\n    - Strides and Padding\n    - Output volume dimension calculation\n    - Deconvolution (Transposed Convolution), Upsampling, Reverse Pooling [[Visualization](https://github.com/vdumoulin/conv_arithmetic#readme)]\n    - Types of convolution operation [[blog](https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d)]\n    \n2. ImageNet Classification\n    - AlexNet [[paper](https://paperswithcode.com/model/alexnet)] [[blog](https://medium.com/p/b93598314160)]\n    - ZFNet [[paper](https://paperswithcode.com/method/zfnet)] [[blog](https://medium.com/coinmonks/paper-review-of-zfnet-the-winner-of-ilsvlc-2013-image-classification-d1a5a0c45103)]\n    - VGGNet [[paper](https://paperswithcode.com/method/vgg)] [[blog](https://medium.com/coinmonks/paper-review-of-vggnet-1st-runner-up-of-ilsvlc-2014-image-classification-d02355543a11)]\n    - InceptionNet [[paper](https://paperswithcode.com/method/inception-v3)] [[blog](https://sh-tsang.medium.com/review-inception-v3-1st-runner-up-image-classification-in-ilsvrc-2015-17915421f77c)]\n    - ResNet [[paper](https://paperswithcode.com/lib/torchvision/resnet)] [[blog](https://towardsdatascience.com/review-resnet-winner-of-ilsvrc-2015-image-classification-localization-detection-e39402bfa5d8)]\n    - DenseNet [[paper](https://paperswithcode.com/lib/timm/densenet)] [[blog](https://towardsdatascience.com/review-densenet-image-classification-b6631a8ef803)]\n    - SENet [[paper](https://paperswithcode.com/paper/squeeze-and-excitation-networks)] [[blog](https://towardsdatascience.com/review-senet-squeeze-and-excitation-network-winner-of-ilsvrc-2017-image-classification-a887b98b2883)]\n    - ViT [[paper](https://arxiv.org/abs/2010.11929)] [[blog](https://ai.googleblog.com/2020/12/transformers-for-image-recognition-at.html)]\n    - Swin Transformer [[paper](https://arxiv.org/abs/2103.14030)] [[blog](https://sh-tsang.medium.com/review-swin-transformer-3438ea335585)]\n    - BEiT [[paper](https://openreview.net/forum?id=p-BhZSz59o4)] [[blog](https://sh-tsang.medium.com/review-beit-bert-pre-training-of-image-transformers-c14a7ef7e295)]\n    - ConvNext [[paper](https://arxiv.org/abs/2201.03545)] [[blog](https://medium.com/augmented-startups/convnext-the-return-of-convolution-networks-e70cbe8dabcc)]\n    \n3. Object Detection [[blog series](https://jonathan-hui.medium.com/object-detection-series-24d03a12f904)]\n    - RCNN [[paper](https://paperswithcode.com/method/r-cnn)]\n    - Fast RCNN [[paper](https://paperswithcode.com/paper/fast-r-cnn)]\n    - Faster RCNN [[paper](https://paperswithcode.com/paper/faster-r-cnn-towards-real-time-object)]\n    - Mask RCNN [[paper](https://paperswithcode.com/paper/mask-r-cnn)]\n    - YOLO (Real-time object recognition) [[blog](https://medium.com/deelvin-machine-learning/the-evolution-of-the-yolo-neural-networks-family-from-v1-to-v7-48dd98702a3d)]\n    - SSD (Single Shot Detection) [[paper](https://paperswithcode.com/method/ssd)]\n    - DETR [[project page](https://alcinos.github.io/detr_page/)] [[annotated DETR](https://amaarora.github.io/2021/07/26/annotateddetr.html)]\n    \n4. Semantic Segmentation\n    - UNet [[paper](https://paperswithcode.com/method/u-net)]\n    - DeepLab [[paper](https://paperswithcode.com/method/deeplab)]\n    - MaskFormer [[paper](https://paperswithcode.com/paper/per-pixel-classification-is-not-all-you-need)] [[project page](https://bowenc0221.github.io/maskformer/)]\n\n## Natural Language Processing\n\n1. Recurrent Neural Networks\n    - Architectures (Limitations and inspiration behind every model)\n        - Vanilla [[blog](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)]\n        - GRU, LSTMs [[blog_1](https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21)] [[blog_2](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)]\n        - Bidirectional\n    - Vanishing and Exploding Gradients\n    \n2. Word Embeddings [[blog_1](https://lilianweng.github.io/posts/2017-10-15-word-embedding/)] [[blog_2](https://jalammar.github.io/illustrated-bert/)]\n    - Word2Vec\n    - CBOW\n    - Glove\n    - SkipGram, NGram\n    - FastText\n    - ELMO\n    - BERT\n    \n3. Transformers [[blog posts](http://jalammar.github.io/)] [[youtube series](https://youtube.com/playlist?list=PLTx9yCaDlo1UlgZiSgEjq86Zvbo2yC87d)]\n    - Attention is All You Need [[blog](https://jalammar.github.io/illustrated-transformer/)] [[paper](https://arxiv.org/abs/1706.03762)] [[annotated transformer](https://nlp.seas.harvard.edu/2018/04/03/attention.html)]\n    - Query-Key-Value Attention Mechanism  (Quadratic Time)\n    - Position Embeddings [[blog](https://kazemnejad.com/blog/transformer_architecture_positional_encoding/)]\n    - BERT (Masked Language Modelling) [[blog](https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html)]\n    - Longe Range Sequence Modelling [[blog](https://huggingface.co/blog/long-range-transformers)]\n    - ELECTRA (Pretraining Transformers as Discriminators) [[blog](https://ai.googleblog.com/2020/03/more-efficient-nlp-model-pre-training.html)]\n    - GPT (Causal Language Modelling) [[blog](https://openai.com/blog/gpt-3-edit-insert/)]\n    - OpenAI ChatGPT [[blog](https://openai.com/blog/chatgpt/)]\n\n## Multimodal Learning\n\n- Vision Language Models | AI Summer [[blog](https://theaisummer.com/vision-language-models/)]\n- Open AI DALL-E [[blog](https://openai.com/blog/dall-e/)]\n- OpenAI CLIP [[blog](https://openai.com/blog/clip/)]\n- Flamingo [[blog](https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model)]\n- Gato [[blog](https://www.deepmind.com/blog/a-generalist-agent)]\n- data2vec [[blog](https://ai.facebook.com/blog/the-first-high-performance-self-supervised-algorithm-that-works-for-speech-vision-and-text/)]\n- OpenAI Whisper [[blog](https://openai.com/blog/whisper/)]\n\n## Generative Models\n\n1. Generative Adversarial Networks (GANs) [[blog series](https://jonathan-hui.medium.com/gan-gan-series-2d279f906e7b)]\n    - Basic Idea\n    - Variants\n        - Vanilla GAN [[paper](https://arxiv.org/abs/1406.2661)]\n        - DCGAN [[paper](https://arxiv.org/abs/1511.06434v2)]\n        - Wasserstein GAN [[paper](https://arxiv.org/abs/1701.07875)]\n        - Conditional GAN [[paper](https://arxiv.org/abs/1411.1784)]\n    - Mode Collapse\n    - GAN Hacks [[link](https://github.com/soumith/ganhacks)]\n2. Variational Autoencoders (VAEs)\n    - Variational Inference [[tutorial paper](https://arxiv.org/abs/1606.05908)]\n    - ELBO and Loss Function derivation\n3. Normalizing Flows\n    - Basic Idea and Applications [[link](https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html)]\n    \n\n## Stable Diffusion\n\n- Demos\n    - Lexica (Stable Diffusion search engine) [[link](https://lexica.art/)]\n    - Stability AI | Huggingface Spaces [[link](https://huggingface.co/spaces/stabilityai/stable-diffusion)]\n- Diffusion Models in general [[paper](https://ommer-lab.com/research/latent-diffusion-models/)]\n    - What are Diffusion Models? | Lil'Log [[link](https://lilianweng.github.io/posts/2021-07-11-diffusion-models/)]\n\n- Stable Diffusion | Stability AI [[blog](https://stability.ai/blog/stable-diffusion-v2-release)] [[annotated stable diffusion](https://huggingface.co/blog/annotated-diffusion)]\n- Illustrated Stable DIffusion | Jay Alammar [[blog](https://jalammar.github.io/illustrated-stable-diffusion/)]\n- Stable Diffusion in downstream Vision tasks\n    - DiffusionDet [[paper](https://arxiv.org/abs/2211.09788)]\n    \n\n## Keeping up with the developments in Deep Learning\n\n- **Youtube Channels**\n    - Yannic Kilcher [[link](https://www.youtube.com/@YannicKilcher)]\n    - Two Minute Papers [[link](https://www.youtube.com/@TwoMinutePapers)]\n- **Blogs**\n    - DeepMind Blog [[link](https://deepmind.com/blog)]\n    - OpenAI Blog [[link](https://openai.com/blog/tags/research/)]\n    - Google AI Blog [[link](https://ai.googleblog.com/)]\n    - Meta AI Blog [[link](https://ai.facebook.com/blog/)]\n    - Nvidia - Deep Learning Blog [[link](https://blogs.nvidia.com/blog/category/deep-learning/)]\n    - Microsoft Research Blog [[link](https://www.microsoft.com/en-us/research/blog/)]\n- **Trending Reseach Papers**\n    - labml [[link](https://papers.labml.ai/papers/recent/)]\n    - deep learning monitor [[link](https://deeplearn.org/)]\n\n## Contributing\nWe welcome contributions to add resources such as notes, blogs, or papers for a topic. Feel free to open a pull request for the same!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvlgiitr%2FDL_Topics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvlgiitr%2FDL_Topics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvlgiitr%2FDL_Topics/lists"}