https://github.com/jaykef/ai-algorithms

First-principle implementations of various AI algorithms using a wide range of deep learning frameworks, accompanied by relevant research papers
https://github.com/jaykef/ai-algorithms

ai algorithms nn papers

Last synced: 7 days ago
JSON representation

First-principle implementations of various AI algorithms using a wide range of deep learning frameworks, accompanied by relevant research papers

• Host: GitHub
• URL: https://github.com/jaykef/ai-algorithms
• Owner: Jaykef
• Created: 2024-08-08T02:56:20.000Z (3 months ago)
• Default Branch: main
• Last Pushed: 2024-09-15T11:46:34.000Z (about 2 months ago)
• Last Synced: 2024-09-15T13:33:54.279Z (about 2 months ago)
• Topics: ai, algorithms, nn, papers
• Language: Jupyter Notebook
• Homepage:
• Size: 3.16 MB
• Stars: 22
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# AI Algorithms

This repo is a work in progress containing first-principle implementations of groundbreaking AI algorithms using a wide range of deep learning frameworks. Each implementation is accompanied by its supporting research paper(s). The goal is to provide comprehensive educational resources for understanding and implementing foundational AI algorithms from scratch.

## Implementations

- [mnist_self_compressing_nns](https://github.com/Jaykef/ai-algorithms/blob/main/mnist_self_compression.ipynb) - Pytorch implementation of ["Self-compressing Neural Networks"](https://arxiv.org/pdf/2301.13142). The paper shows dynamic neural network compression during training - reduced size of weight, activation tensors and bits required to represent weights.

- [mnist_ijepa](https://github.com/Jaykef/ai-algorithms/blob/main/mnist_ijepa.ipynb) - Simplified image-based implementation of JEPA (Joint-Embedding Predictive Architecture) - an alternative to auto-regressive LLM architectures pioneered by Prof. Yann LeCun. [I-JEPA](https://arxiv.org/pdf/2301.08243.pdf) predicts image segment representations (Target) based on representations of other segments within the same image (Context).

- [nns_are_decision_trees](https://github.com/Jaykef/ai-algorithms/blob/main/nns_are%20decision_trees.ipynb) - Simplified implementation of “Neural Networks are Decision Trees”. Showing that any neural network with any activation function can be represented as a decision tree. Since decision trees are inherently interpretable, their equivalence helps us understand how the network makes decisions.

- [mnist_the_forward_forward_algorithm](https://github.com/Jaykef/ai-algorithms/blob/main/mnist_the_forward_forward_algorithm.ipynb) - Implements the [Forward-Forward Algorithm](https://arxiv.org/abs/2212.13345) proposed by AI godfather Geoffrey Hinton. The algorithm replaces the forward and backward passes in backpropagation with two forward passes on different data with opposite objectives. The positive pass uses real data and adjusts weights to increase goodness in every hidden layer. The negative pass uses "negative data" and decreases goodness.

- [sigmoid_attention](https://github.com/Jaykef/ai-algorithms/blob/main/sigmoid_attn.ipynb) - Implements newly introduced [Sigmoid Self-Attention](https://arxiv.org/abs/2409.04431) by Apple.

- [DIFF_Transformer](https://github.com/Jaykef/ai-algorithms/blob/main/DIFF_Transformer.ipynb) - Lightweight implementation of newly introduced “Differential Transformer”: Proposes differential attention mechanism which computes attention scores as a difference between two separate softmax attention maps thereby reducing noise in attention blocks. [Paper](https://arxiv.org/pdf/2410.05258) by microsoft.

- [triton_nanoGPT.ipynb](https://github.com/Jaykef/ai-algorithms/blob/main/triton_nanoGPT.ipynb) - Implements custom triton kernels for training Karpthy's nanoGPT (more improvements needed).