{"id":18375235,"url":"https://github.com/berthubert/hello-dl","last_synced_at":"2025-04-06T20:30:57.542Z","repository":{"id":64996054,"uuid":"578530101","full_name":"berthubert/hello-dl","owner":"berthubert","description":null,"archived":false,"fork":false,"pushed_at":"2023-09-01T14:54:33.000Z","size":1909,"stargazers_count":74,"open_issues_count":3,"forks_count":9,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-02T04:38:20.656Z","etag":null,"topics":["deep-learning","machine-learning","neural-network"],"latest_commit_sha":null,"homepage":"https://berthub.eu/tmp/hello-dl","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/berthubert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-12-15T09:22:04.000Z","updated_at":"2025-03-31T03:16:55.000Z","dependencies_parsed_at":"2023-02-19T07:45:48.566Z","dependency_job_id":null,"html_url":"https://github.com/berthubert/hello-dl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/berthubert%2Fhello-dl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/berthubert%2Fhello-dl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/berthubert%2Fhello-dl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/berthubert%2Fhello-dl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/berthubert","download_url":"https://codeload.github.com/berthubert/hello-dl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247547171,"owners_count":20956507,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","machine-learning","neural-network"],"created_at":"2024-11-06T00:18:10.281Z","updated_at":"2025-04-06T20:30:57.139Z","avatar_url":"https://github.com/berthubert.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# hello-dl\nbert@hubertnet.nl - I post about this project on Mastodon as [@bert_hubert@fosstodon.org](https://fosstodon.org/@bert_hubert). A series of associated blog posts is [here](https://berthub.eu/articles/posts/hello-deep-learning/).\n\nA from scratch introduction to modern machine learning. Many tutorials exist\nalready of course, but this one aims to really explain what is going on. Also, we'll develop the demo until it is actually useful on real life data.\n\nOther documents start out from the (very impressive) PyTorch environment, or\nthey attempt to math it up from first principles. \n\n![](img/boxed.png)  \n*([animated version](img/learning.mp4))*\n\nTrying to understand deep learning via PyTorch is like trying to learn\naerodynamics from flying an Airbus A380. \n\nMeanwhile the pure maths approach (\"see it is easy, it is just a Jacobian\nmatrix\") is probably only suited for seasoned mathematicians.\n\nThe goal of this tutorial is to develop modern neural networks entirely from\nscratch, but where we still end up with really impressive results.\n\nTo do so, this project contains some minimalist tooling.  The software may\nbe minimal, but will showcase modern deep learning techniques that should\nwow you into believing that something very special is going on.\n\nIn quantum computing there is the concept of '[quantum supremacy](https://en.wikipedia.org/wiki/Quantum_supremacy)', which is achieved when a quantum computer does something that no ordinary computer could do in the lifetime of the universe. \n\nWith this 'hello deep learning' software I'm aiming for a slightly more modest goal: show you a comprehensible amount of neural networking source code that solves a problem you'd never be able to address in fewer lines of code otherwise.\n\nThe intended audience for this project is people comfortable with programming and who want to\n(re)join the artificial intelligence scene from the ground up.\n\nThis project was inspired by [Georgi Gerganov](https://ggerganov.com/)'s\nAWESOME [C++ implementation of OpenAI's Whisper speech/translation model](https://github.com/ggerganov/whisper.cpp).\n\n# Mission statement\n\n * Really make you feel what deep learning is all about\n   * Tools must do things you will be personally be impressed by\n * Not skimp over the details, how does it ACTUALLY work\n * Make reading the tool's source code a good learning experience\n   * More important that code is readable than that it is as fast as possible\n   * Also be fast enough that it is \"doable\" though\n * Cover all the techniques that form the backbone of modern DL successes\n * Use language and terms that are compatible with modern day usage\n   * So you can also understand PyTorch tutorials\n   * However, also not confuse you with weird historical words\n * Provide modules that mirror popular functionality in PyTorch\n\nNon-goals:\n\n * Turn you into a machine learning professional overnight\n * Teach you PyTorch or TensorFlow directly\n\nThe idea is that after you are done with the blog posts (yet to be written) and\nhave worked with the tools that it should all make sense to you. And *then*\nyou can get to work with professional tooling and get to work.\n\nAlthough this is an introduction using minimal dependencies and lines of\ncode, we'll be covering modern subjects like momentum, data augmentation,\n[weight decay](https://vitalflux.com/weight-decay-in-machine-learning-concepts/), batch normalization, dropout, GELU, [weights and bias\nlogging](https://wandb.ai/), [learning rate scheduling](https://d2l.ai/chapter_optimization/lr-scheduler.html) \u0026  more.\n\nAnd unlike most demos, this one includes crowd sourced validation data that\nhelps show why many deep learning projects are great in theory [but\ndisappoint in practice](https://www.technologyreview.com/2021/07/30/1030329/machine-learning-ai-failed-covid-hospital-diagnosis-pandemic/). \n\nIt appears machine learning projects tend to go through four phases:\n\n1. It works on the training data\n2. It also works on the validation data\n3. After a lot of disappointment, we get it to work on other people's real life data too\n4. Other people can get it to work on their own data as well\n\nAlmost all demos declare victory after phase 2. This tutorial aims to achieve the final phase.\n\nFinally, we'll also spend time on how to detect that a neural network is not delivering a confident result.\n\n# Acknowledgments \nDespite doing some neural network experiments in the 1990s, I'm extremely new to the modern stuff. The following projects/sites have been very useful in catching up:\n\n * NIST and Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre van Schaik\n   from The MARCS Institute for Brain, Behaviour and Development for creating and maintaining the MNIST [database of handwritten characters](https://arxiv.org/abs/1702.05373).\n * The PyTorch project, their documentation and samples are invaluable\n * Facebook's [Flashlight](https://github.com/flashlight/flashlight), a fast, flexible machine learning library written entirely in C++ from the Facebook AI Research and the creators of Torch, TensorFlow, Eigen and Deep Speech. Their source code helped me understand lots of things.\n * [Andrej Karpathy](https://twitter.com/karpathy)'s [micrograd](https://github.com/karpathy/micrograd) Python autogradient implementation is a tiny work of art\n * Andrej Karpathy's post [The Unreasonable Effectiveness of Recurrent Neural Networks](https://karpathy.github.io/2015/05/21/rnn-effectiveness/), and also [this post](https://karpathy.github.io/2019/04/25/recipe/)\n * [FastAI](https://fast.ai)'s Jupyter notebooks.\n \n\u003c!-- https://medium.com/geekculture/a-2021-guide-to-improving-cnns-optimizers-adam-vs-sgd-495848ac6008 --\u003e \n\u003c!-- GRU https://blog.floydhub.com/gru-with-pytorch/ --\u003e\n\u003c!-- GRU paper https://arxiv.org/pdf/1406.1078.pdf --\u003e\n\u003c!-- https://arxiv.org/pdf/2212.14034.pdf large language models on smaller systems --\u003e\n\u003c!-- https://github.com/coin-or/ADOL-C - automatic differentiation --\u003e\n\u003c!-- need to address \"bagging\" --\u003e\n\u003c!-- need to address \"batch normalization\" --\u003e\n\n\u003c!-- good further examples: https://keras.io/examples/ --\u003e \n# Status\nPretty rough, but showing signs of doing something real.\n\nThe code implements two small but pretty nice autograd systems, one simple and slow, the other slightly more complicated but fast. In `tensor-relu.cc` you can find a ~150 line total computer program that\nlearns to recognize handwritten digits in a few seconds (90% accuracy so far).\n\nWe do have a few dependencies:\n\n```bash\nsudo apt install cmake libeigen3-dev libsqlite3-dev g++ libz-dev\n```\nThis works for Debian derived systems, but cmake, Eigen and SQLite are\nuniversally available.\n\nHere is how to install and run the first sample: \n\n```bash\ngit clone https://github.com/berthubert/hello-dl.git\ncd hello-dl\ncmake .\nmake -j4\nwget http://www.itl.nist.gov/iaui/vip/cs_links/EMNIST/gzip.zip\nunzip gzip.zip\n./tensor-relu\n```\nThe result is nice:\n\n```\nStart!\nHave 240000 images\n...\nPercent batch correct: 32.8125%\nAverage loss: 2.1219. Predicted: 8, actual: 8: We got it right!\nLoss: 1.89751, -2.44847 -2.54296 -2.43445 -1.92324 -3.07031 -2.50595 -2.27749 -2.1481 -1.89751 -2.26366\n          .******           \n         *XXXXXXXXX.        \n        .XXXXX**XXXX        \n        XXXX.    XXX.       \n        XXX*     *XX*       \n       .XXX.     .XX*       \n        XXX.     *XX.       \n        XXX.     XXX        \n        *XXX.   XXXX        \n         *XXX. .XXX*        \n         .XXXXXXXXX*        \n           *XXXXXXX         \n            XXXXXX.         \n          .XXXXXXX.         \n         .XXXXXXXXX         \n        .XXX*   XXX*        \n        *XXX    XXX*        \n        XXX*    XXX*        \n        XXX.    *XX*        \n        XXX*    XXX*        \n        *XXX*.*XXXX*        \n        *XXXXXXXXXX.        \n         .XXXXXXX.          \n           *****            \n```\n\nThis software is meant to accompany a series of blog posts introducing deep\nlearning from the ground up. That series hasn't started yet as I am still\nfiguring out how this stuff works.\n\nA more modern convolutional network can be found in [tensor-convo.cc](./tensor-convo.cc) and after hours and hours of crunching it will produce a network that you can feed to [img-ocr.cc](./img-ocr.cc). If the stars align, and handwritten 'print' letters in an image will be recognized by the network.\n\nIf you want to see something cool already, take a look at\n[testrunner.cc](./testrunner.cc)\nwhich already shows some of the autogradient stuff.\n\n# Getting started\nCheckout the repository, run `cmake .` and then `make`.\nTo actually do something [download the\nEMNIST](http://www.itl.nist.gov/iaui/vip/cs_links/EMNIST/gzip.zip) dataset\nand unzip it. There's no need to gunzip the .gz files.\n\nNext up, run `./tensor` or `./first-convo` and wonder what you are seeing.\n\n# Data\nhttps://www.nist.gov/itl/products-and-services/emnist-dataset\n\nhttps://arxiv.org/pdf/1702.05373v1\nhttp://yann.lecun.com/exdb/mnist/\n\n# Inspiration \u0026 Similar projects\nhttps://github.com/fastai/fastbook/blob/master/04_mnist_basics.ipynb\n\nAlso really good:\nhttps://pytorch.org/tutorials/beginner/nn_tutorial.html\n\n[PicoGPT](https://jaykmody.com/blog/gpt-from-scratch/)\n\n[NanoGPT](https://github.com/karpathy/nanoGPT/)\n\n# Internals\n~~Use `TrackedNumber` to build networks. You can also run them there if you want, but a faster way is to export them as 'Work' objects, which are a solid slab of contiguous memory.~~\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fberthubert%2Fhello-dl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fberthubert%2Fhello-dl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fberthubert%2Fhello-dl/lists"}