{"id":18029298,"url":"https://github.com/gmontamat/poor-mans-transformers","last_synced_at":"2025-03-27T03:31:05.397Z","repository":{"id":88929606,"uuid":"402773322","full_name":"gmontamat/poor-mans-transformers","owner":"gmontamat","description":"Implement Transformers (and Deep Learning) from scratch in NumPy","archived":false,"fork":false,"pushed_at":"2023-10-03T19:53:36.000Z","size":180,"stargazers_count":26,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-23T00:31:55.291Z","etag":null,"topics":["deep-learning","from-scratch","machine-learning","ml-framework","neural-network","nlp","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gmontamat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-03T13:06:41.000Z","updated_at":"2025-02-19T10:16:22.000Z","dependencies_parsed_at":"2024-10-30T09:18:55.962Z","dependency_job_id":null,"html_url":"https://github.com/gmontamat/poor-mans-transformers","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gmontamat%2Fpoor-mans-transformers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gmontamat%2Fpoor-mans-transformers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gmontamat%2Fpoor-mans-transformers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gmontamat%2Fpoor-mans-transformers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gmontamat","download_url":"https://codeload.github.com/gmontamat/poor-mans-transformers/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245778433,"owners_count":20670682,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","from-scratch","machine-learning","ml-framework","neural-network","nlp","transformers"],"created_at":"2024-10-30T09:08:45.286Z","updated_at":"2025-03-27T03:31:05.043Z","avatar_url":"https://github.com/gmontamat.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Poor Man's Transformers\n\nAdvanced Deep Learning from the ground-up.\n\nThe idea of this repository is to implement the necessary framework and layers of a transformer using just `numpy` for\nlearning purposes. The end goal is to train a Transformer model on [QQP](https://www.kaggle.com/c/quora-question-pairs)\nor a model that performs Named Entity Recognition (NER) decently. I was inspired by\n[ML-From-Scratch](https://github.com/eriklindernoren/ML-From-Scratch),\nthe [Advanced Machine Learning Specialization](https://www.coursera.org/specializations/aml), and\nthe [Natural Language Processing Specialization](https://www.coursera.org/specializations/natural-language-processing).\n\nSelf-imposed rules for developing this toy framework:\n\n- Use only the python standard library and `numpy` as a (tensor algebra) dependency. I relaxed this rule a bit to\n  include some other useful features such as a progress bar for the training loop (using `tqdm`) and visualizations (\n  with\n  `matplotlib`).\n- Readability is more important than efficiency here. Code is not optimal but should be clear.\n- No scooping at PyTorch, TensorFlow, or [tinygrad](https://github.com/tinygrad/tinygrad) implementations.\n\n## :notebook: Development log\n\nI'm keeping track of my progress in this section, so it can be used for future reference when learning Deep Learning\nfrom the very basics.\n\n### Index\n\n* [First steps: basic layers and training framework for an MLP](#bookmark-first-steps-basic-layers-and-training-framework-for-an-mlp)\n* Convolutional Neural Networks (TODO)\n* [Word Embeddings](#construction-word-embeddings)\n* [Handling non-sequential architectures](#construction-handling-non-sequential-architectures)\n\n---\n\n### :bookmark: First steps: basic layers and training framework for an MLP\n\nFirst things first, I need to implement the basic structure of the framework and be able to train a Multilayer\nPerceptron (MLP) with it. I base my work\non [ML-From-Scratch Deep Learning implementation](https://github.com/eriklindernoren/ML-From-Scratch#deep-learning)\nand an assignment from the [Introduction to Deep Learning](https://www.coursera.org/learn/intro-to-deep-learning)\ncourse.\n\nEven though this was supposed to be an easy step, I ended up spending a lot of time on it trying to come up with the\nsimplest OOP architecture possible. New layers have to be easy to code, and I also want to experiment with different\noptimizers (SGD, Adam, RMSProp), learning rate schedules, activation functions (ReLU, tanh, sigmoid, Softmax,\nLogSoftmax), custom loss functions (sum binary cross-entropy and categorical cross-entropy as used for training BERT)\nand easily handle input \u0026 output flow (serial, parallel, concatenations, skip-connections). At the same time, I wouldn't\nlike to waste time building a flexible and feature-rich framework since we already have PyTorch, TensorFlow with Keras,\n[JAX](https://github.com/google/jax), and [Google Trax](https://github.com/google/trax) for that.\n\nTo keep this toy \"framework\" as simple as possible, I want to minimize the number of base classes. I ended up with:\n`Layer`, `Parameter` (used in a `Layer` with an associated `Optimizer`), `Trainer`, and `Loss`. `Activation` functions\nare a subclass of a `Layer` object. This simplification comes with its costs, of course, in terms of RAM usage: more\n\"intermediate\" tensors will be stored in memory. When training a `Dense` layer with a `ReLU` activation, for example,\nboth the linear combination and the rectified (`max(X, 0)`) tensors will be stored in memory. I do not intend to run\nthis framework on a GPU, so RAM usage is not a big concern right now. Each layer will implement its backpropagation\nstep, the derivatives with respect to each parameter (Jacobian matrix) have to be computed because I don't want to\nimplement a tool such as [Autograd](https://github.com/hips/autograd) to do this automatically.\n\nHere's the list of objects I implemented:\n\n#### :pushpin: Layer and Activation\n\nA layer performs two operations: forward propagation and backward propagation. For doing the forward pass, it receives\nan input batch `X` and uses its `Parameter`s to compute the output batch. And in the case of the backward pass, it\nreceives the accumulated gradient `grad` (which represents the derivatives `d_loss / d_layer` for each element in the\nbatch) to compute and propagate to the previous layer: `d_loss / d_input = d_loss / d_layer · d_layer / d_input`. It\nalso receives the input batch `X` used in the forward step to compute the gradients with respect to the\nparameters `d_loss / d_parameter = d_loss / d_layer · d_layer / d_parameter`. Next, it calls the `update` method on\nall `Parameter`s which use an `Optimizer` instance to update their weights. Finally, the accumulated\ngradient `d_loss / d_input` is returned to proceed with the network's backward propagation.\n\nWhen instantiated, an `input_shape` and `output_shape` could be set, or else they will be set by the `Trainer` during\nthe model's initialization step. The initial weights of each `Parameter` also need to be defined during this step.\n\nAn `Activation` is a special type of `Layer` whose `input_shape` and `output_shape` are the same.\n\n:heavy_check_mark: [Layer](poormanslayers/layers.py#L32-L79)\n:heavy_check_mark: [Activation](poormanslayers/layers.py#L82-L92)\n:white_check_mark: [Dense](poormanslayers/layers.py#L95-L133)\n:white_check_mark: [ReLU](poormanslayers/layers.py#L136-L142)\n:white_check_mark: [Softmax](poormanslayers/layers.py#L155-L173)\n:white_check_mark: [LogSoftmax](poormanslayers/layers.py#L176-L193)\n:white_check_mark: [Dropout](poormanslayers/layers.py#L196-L214)\n\n#### :pushpin: Parameter and Optimizer\n\nA `Parameter` is instantiated only by a `Layer`. Its weights can be accessed by calling the `Parameter` instance and are\nupdated during back-propagation using the `update` method. For `update` to be called, an `Optimizer` instance needs to\nbe set and its initial weights need to be defined during the `Layer`'s initialization.\n\nEach `Parameter` instantiated in the framework will have a copy of an `Optimizer` instance with the properties defined\nby the `Trainer` object. The `Optimizer` is in charge of updating the parameter's value and may store auxiliary\nvariables to do so, hence, each parameter has a unique copy of it. Again, it set by the `Trainer` during the model's\ninitialization.\n\n:heavy_check_mark: [Parameter](poormanslayers/layers.py#L8-L29)\n:heavy_check_mark: [Optimizer](poormanslayers/optimizers.py#L4-L11)\n:white_check_mark: [Adam](poormanslayers/optimizers.py#L14-L43)\n\n#### :pushpin: Loss and Metric\n\nThese classes are pretty straightforward: instances are called with the ground truth `y` and predictions `y_hat` (or\nprediction's probabilities *logits*) and return the calculated metric. The `Loss` class also returns the gradient\n`d_loss / d_yhat` to begin the backward propagation.\n\n:heavy_check_mark: [Loss](poormanslayers/losses.py#L6-L27)\n:heavy_check_mark: [Metric](poormanslayers/losses.py#L30-L41)\n:white_check_mark: [CategoricalCrossEntropy](poormanslayers/losses.py#L44-L59)\n:white_check_mark: [Accuracy](poormanslayers/losses.py#L76-L84)\n\n#### :pushpin: Model, Trainer, and DataGeneratorWrapper\n\nInstead of following Keras-style Sequential Model and the `model.compile()` method to define the optimizer, loss, and\nmetrics, a `Model` in this framework is just a list of `Layer` instances (I think this will help us handle complex flows\nwith stack operations). Hence, I defined the `Trainer` which receives a model, optimizer, loss, learning rate schedule,\nearly stopping, and metrics to run the supervised training with training and evaluation data generators. This approach\nresembles the Trax framework more than Keras or PyTorch.\n\nThe `fit` method in `Trainer` is the key function of this class. It prepares the model by setting and validating the\n`input_shape` and `output_shape` for every layer, and initializing the layer's weights. Training and evaluation data is\npassed via a generator function that has to be written for every particular dataset and needs to be wrapped using the\n`DataGeneratorWrapper` whose only purpose is to initialize the generator with all the arguments passed so that the data\ncould be \"rewound\" at the beginning of each epoch.\n\n:heavy_check_mark: [Trainer](poormanslayers/train.py#L33)\n:heavy_check_mark: [DataGeneratorWrapper](poormanslayers/train.py#L15-L30)\n\n#### :warning: Challenges\n\nThe most difficult part of this first step was to do the backwards propagation. I needed to compute Jacobian matrices of\nseveral vector functions. The following articles helped me clarify the math needed:\n\n* [Jacobian, Chain rule and backpropagation](https://suzyahyah.github.io/calculus/machine%20learning/2018/04/04/Jacobian-and-Backpropagation.html)\n* [The Softmax function and its derivative](https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/)\n\n#### :gem: Sample code\n\n:heavy_check_mark: [MLP for MNIST Digit recognition](./examples/mlp.py)\n\n```shell\ncd examples\n./download_mnist.sh\npython3 mlp.py\n```\n\n---\n\n### :construction: Word Embeddings\n\nMy next goal is to have an `Embedding` layer implemented and try it out by replicating\n[word2vec](https://code.google.com/archive/p/word2vec/) models using both the Continuous Bag of Words (CBOW) and\nSkip-Gram architectures. We should be able to generate word embeddings and compare their accuracy on the\nSemantic-Syntactic Word Relationship test set mentioned in [word2vec's paper](https://arxiv.org/pdf/1301.3781.pdf).\n\nWith the framework in place and validated with the Multilayer Perceptron trained on MNIST, this part should have been a\nmatter of adding some subclasses and helper functions... but it wasn't. A very basic example with the CBOW model was\ncreated by just adding the `Embedding` and `AxisMean` layers. There were two problems though: first, it's not an exact\nreplica of the model architecture since `Softmax` will propagate the gradients to all the words in the vocabulary\n(Hierarchical Softmax is used in word2vec which is faster). Second, the lack of a `Lambda` layer which computes and\npropagates the gradients of a user-defined forward function is difficult to code (not impossible, but we don't want\nautograd here).\n\nThe skip-gram implementation, with negative sampling, is more faithful to its original implementation. But another\nproblem arises here: a `Model`, defined as a list of `Layer` objects, doesn't support multiple input branches (we need\nboth the target word and the context/negative word to be passed through the same `Embedding` layer and then merge them\nwith a dot product). The quick and dirty fix here is to pass both target and context through the `Embedding` layer and\ncreate a `AxisDot` layer which computes the dot product along the axis whose shape is 2. A `Sigmoid` layer is used here,\nwhich is a simpler version of the `Softmax` layer.\n\n#### :pushpin: Embedding\n\nThe `Embedding` layer is equivalent to a `Dense` layer if we converted the word representations (numbers in the range\n`[0, vocab_size)`) to their one-hot representation and performed a matrix-matrix dot product between the input and\nweights. Here instead, the layer takes the word representation (integer between 0 and `vocab_size-1`) and use it to\nindex the weights' matrix. We avoid doing a matrix-matrix dot product which is more expensive.\n\n:white_check_mark: [Embedding](poormanslayers/layers.py#L217-L249)\n\n#### :pushpin: AxisMean\n\nThe CBOW model works by averaging the embeddings of a context window surrounding the target word. The dimension average\nis usually done by a `Lambda` layer which takes a lambda function and use it as the forward propagation step. Frameworks\nhave tools like [autograd](https://github.com/HIPS/autograd) to compute a gradient (formally, *jacobian*) given the\nforward function. For simplicity, I created the `AxisMean` layer instead of a `Lambda` layer which doesn't require the\naforementioned tool.\n\n:white_check_mark: [AxisMean](poormanslayers/layers.py#L252-L272)\n\n#### :construction: AxisDot\n\n:white_check_mark: [AxisDot](poormanslayers/layers.py#L275-L298)\n\n#### :construction: BinaryCrossEntropy\n\n:white_check_mark: [BinaryCrossEntropy](poormanslayers/losses.py#L62-L73)\n\n#### :pushpin: RMSProp\n\nImplementing this optimizer is straightforward. Just need to keep a moving average of the element-wise squared gradient\nand use its squared root when updating the weights.\n\n:white_check_mark: [RMSProp](poormanslayers/optimizers.py#L46-L65)\n\n#### :warning: Challenges\n\nImplementing backpropagation for the `Embedding` layer was a bit tricky but not as hard as the Softmax and LogSoftmax\nlayers. The following resources guided me through this step:\n\n* [What is the difference between an Embedding Layer and a Dense Layer?](https://stackoverflow.com/questions/47868265/what-is-the-difference-between-an-embedding-layer-and-a-dense-layer)\n* [Back propagation in an embedding layer](https://medium.com/@ilyarudyak/back-propagation-in-an-embedding-layer-30382fa7f023)\n\nAs mentioned above, implementing the CBOW and Skip-gram architectures wasn't simple. I followed these articles:\n\n* [Implementing Deep Learning Methods and Feature Engineering for Text Data: The Continuous Bag of Words (CBOW)](https://www.kdnuggets.com/2018/04/implementing-deep-learning-methods-feature-engineering-text-data-cbow.html)\n* [word2vec](https://code.google.com/archive/p/word2vec/)\n* [A word2vec Keras Tutorial (via web.archive.org)](https://web.archive.org/web/20210212061052/https://adventuresinmachinelearning.com/word2vec-keras-tutorial/)\n\nSubsampling and negative sampling formulas used are explained in the following sources:\n\n* [Word2Vec Tutorial Part 2 - Negative Sampling](https://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/)\n\nTraining word embeddings like those they released by word2vec is painfully slow and difficult. It's also very hard to\ndebug since we didn't follow the code they've released in C but copied the architecture they describe in their papers.\nTo validate that our network is working I created [this toy example](./examples/validate_word2vec.py) that trains a very\nbasic embedding of 2 dimensions with a vocabulary of words \"Paris\", \"France\", \"Berlin\", and \"Germany\". I got promising\nresults like the following:\n\n![Basic word2vec embedding](./assets/word2vec_sample.png)\n\nThe vector \"Paris\" -\u003e \"France\" is almost the same as \"Berlin\" -\u003e \"Germany\" indicating that this direction in the\nembedding represents \"is the capital of\".\n\n#### :construction: Sample code\n\n:construction: [Continuous Bag of Words (CBOW) with Text8](./examples/cbow.py)\n\n```shell\ncd examples\n./download_text8.sh\npython3 cbow.py\n```\n\n:construction: [Skip-gram with Text8](./examples/skipgram.py)\n\n```shell\ncd examples\n./download_text8.sh\npython3 skipgram.py\n```\n\n---\n\n### :construction: Handling non-sequential architectures\n\nSo far we created very simple model architectures that can be implemented as a sequence of layers. We could define the\nmulti-layer perceptron, convolutional neural networks, and, with some tricks (like `AxisDot`), the skip-gram model\nin the previous section. But looking at more complex architectures, like the encoder-decoder transformer, we see that\ntensors don't flow in a sequential manner. More generally, deep neural networks can be seen as directed acyclic\nmulti-graphs with several inputs and outputs, and residual operations. I tried to extend the original `Layer` class\nto support networks but then realized that the framework is inflexible. I also began using PyTorch and found its\n`nn.Module` component practical for defining any kind of architecture and connections (no need to\nuse [combinator layers](https://trax-ml.readthedocs.io/en/latest/trax.layers.html#module-trax.layers.combinators) for\nexample). Coding in a Deep Neural network on PyTorch is more natural. I therefore replicate this component from scratch,\nwith another caveat: since anyone can extend this module to implement any type of operation, automatic gradient\ncomputation is a must.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgmontamat%2Fpoor-mans-transformers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgmontamat%2Fpoor-mans-transformers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgmontamat%2Fpoor-mans-transformers/lists"}