{"id":20324339,"url":"https://github.com/oniani/ai","last_synced_at":"2025-08-12T06:34:18.702Z","repository":{"id":60944027,"uuid":"529740606","full_name":"oniani/ai","owner":"oniani","description":"From-scratch impls of AI models, approaches, tricks, and more!","archived":false,"fork":false,"pushed_at":"2024-04-09T05:45:05.000Z","size":1989,"stargazers_count":27,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-07-31T16:52:07.016Z","etag":null,"topics":["ai","cs","dl","ml","software"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oniani.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-28T02:11:29.000Z","updated_at":"2025-07-01T07:02:00.000Z","dependencies_parsed_at":"2024-04-09T04:27:14.456Z","dependency_job_id":"7ff3f105-338a-4089-abca-912839971ff5","html_url":"https://github.com/oniani/ai","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oniani/ai","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oniani%2Fai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oniani%2Fai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oniani%2Fai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oniani%2Fai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oniani","download_url":"https://codeload.github.com/oniani/ai/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oniani%2Fai/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270014431,"owners_count":24512653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","cs","dl","ml","software"],"created_at":"2024-11-14T19:33:37.959Z","updated_at":"2025-08-12T06:34:18.681Z","avatar_url":"https://github.com/oniani.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ai\n\nFrom-scratch impls of AI models, approaches, tricks, and more!\n\n## Setup\n\n```console\npython -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n```\n\n## Contents\n\n- [Activation Functions and Their Derivative Functions from Scratch Using Pytorch](#activation-functions-and-their-derivative-functions-from-scratch-using-pytorch)\n- [Functions](#functions)\n- [Gradient Descent](#gradient-descent)\n- [Deep Learning](#deep-learning)\n- [Machine Learning Models from Scratch Using NumPy](#machine-learning-models-from-scratch-using-numpy)\n- [Representation](#representation)\n\n### Activation Functions and Their Derivative Functions from Scratch Using Pytorch\n\n- Sigmoid\n  - [:rocket: Implementation][sigmoid]\n  - [:orange_book: Theory][sigmoid_theory]\n  - [:chart_with_upwards_trend: Plot][sigmoid_plot]\n  - [:tv: YouTube Video: Implementing Sigmoid and Its Derivative from Scratch][sigmoid_youtube]\n- ReLU\n  - [:rocket: Implementation][relu]\n  - [:orange_book: Theory][relu_theory]\n  - [:chart_with_upwards_trend: Plot][relu_plot]\n  - [:tv: YouTube Video: Implementing ReLU and Its Derivative from Scratch][relu_youtube]\n- Leaky ReLU\n  - [:rocket: Implementation][leaky_relu]\n  - [:orange_book: Theory][leaky_relu_theory]\n  - [:chart_with_upwards_trend: Plot][leaky_relu_plot]\n  - [:tv: YouTube Video: Implementing Leaky ReLU and Its Derivative from Scratch][leaky_relu_youtube]\n- GELU\n  - [:rocket: Implementation][gelu]\n  - [:orange_book: Theory][gelu_theory]\n  - [:chart_with_upwards_trend: Plot][gelu_plot]\n  - [:tv: YouTube Video: Implementing GELU and Its Derivative from Scratch][gelu_youtube]\n- Swish\n  - [:rocket: Implementation][swish]\n  - [:orange_book: Theory][swish_theory]\n  - [:chart_with_upwards_trend: Plot][swish_plot]\n  - [:tv: YouTube Video: Implementing Swish and Its Derivative from Scratch][swish_youtube]\n- SERF\n  - [:rocket: Implementation][serf]\n  - [:orange_book: Theory][serf_theory]\n  - [:chart_with_upwards_trend: Plot][serf_plot]\n  - [:tv: YouTube Video: Implementing SERF and Its Derivative from Scratch (r/MachineLearning special)][serf_youtube]\n  - [:newspaper: r/MachineLearning reddit post by u/Shronnin: \\[R\\] \\[D\\] SERF activation function - improving Swish][serf_reddit]\n- Tanh\n  - [:rocket: Implementation][tanh]\n  - [:orange_book: Theory][tanh_theory]\n  - [:chart_with_upwards_trend: Plot][tanh_plot]\n  - [:tv: YouTube Video: Implementing Tanh and Its Derivative from Scratch][tanh_youtube]\n\n### Deep Learning\n\n- Image Classification: CNN for MNIST (+Deep Learning Project Setup)\n  - [:orange_book: The Reproducible MNIST][the_reproducible_mnist]\n  - [:rocket: MNIST CNN Model][mnist_cnn]\n  - [:tv: YouTube Video: Image Classification: CNN for MNIST (+Deep Learning Project Setup)][deep_learning_setup_youtube]\n- Image Classification: VGG Model from Scratch\n  - [:rocket: Implementation][vgg]\n  - [:tv: YouTube Video: Image Classification: VGG Models from Scratch][vgg_youtube]\n- Recommender Systems: Generalized Matrix Factorization from Scratch\n  - [:rocket: Implementation][gmf]\n  - [:tv: YouTube Video: Recommender Systems: Generalized Matrix Factorization from Scratch][gmf_youtube]\n\n### Functions\n\n- Distance Functions\n  - [:rocket: Implementation][distance_functions]\n  - [:orange_book: Theory][distance_functions_theory]\n  - [:tv: YouTube Video: Implementing Distance Functions from Scratch][distance_functions_youtube]\n- Convolution\n  - [:rocket: Implementation][convolution]\n  - [:orange_book: Theory][convolution_theory]\n  - [:tv: YouTube Video: Implementing a Convolution from Scratch (+ Baby Yoda)][convolution_youtube]\n\n### Gradient Descent\n\n- Discussing Batch, Stochastic, and Mini-Batch Gradient Descent\n  - [:orange_book: Theory][gradient_descent_theory]\n  - [:chart_with_upwards_trend: Convex and Non-Convex Functions][gradient_descent_plot]\n  - [:tv: YouTube Video: Discussing Batch, Stochastic, and Mini-Batch Gradient Descent][gradient_descent_youtube]\n\n### Machine Learning Models From Scratch Using NumPy\n\n- Gaussian Naive Bayes\n  - [:rocket: Implementation][gaussian_naive_bayes]\n  - [:orange_book: Theory][gaussian_naive_bayes_theory]\n  - [:tv: YouTube Video: Implementing Gaussian Naive Bayes From Scratch][gaussian_naive_bayes_youtube]\n  - [:tv: YouTube Video: AI/ML Model API Design and Numerical Stability (follow-up)][api_design_and_numerical_stability_youtube]\n- K-Nearest Neighbors (k-NN)\n  - [:rocket: Implementation][k_nearest_neighbors]\n  - [:orange_book: Theory][k_nearest_neighbors_theory]\n  - [:tv: YouTube Video: Implementing K-Nearest Neighbors From Scratch][k_nearest_neighbors_youtube]\n  - [:tv: YouTube Video: AI/ML Model API Design and Numerical Stability (follow-up)][api_design_and_numerical_stability_youtube]\n- Linear Regression\n  - [:rocket: Implementation][linear_regression]\n  - [:orange_book: Theory][linear_regression_theory]\n  - [:tv: YouTube Video: Implementing Linear Regression from Scratch][linear_regression_youtube]\n  - [:tv: YouTube Video: AI/ML Model API Design and Numerical Stability (follow-up)][api_design_and_numerical_stability_youtube]\n- Logistic Regression\n  - [:rocket: Implementation][logistic_regression]\n  - [:orange_book: Theory][logistic_regression_theory]\n  - [:orange_book: Computing Gradients][logistic_regression_computing_gradients]\n  - [:tv: YouTube Video: Implementing Linear Regression from Scratch][logistic_regression_youtube]\n- k-Means Clustering\n  - [:rocket: Implementation][k_means_clustering]\n  - [:orange_book: Theory][k_means_clustering_theory]\n  - [:tv: YouTube Video: Implementing k-Means Clustering from Scratch][k_means_clustering_youtube]\n\n### Representation\n\n- Principal Component Analysis (PCA) from Scratch\n  - [:rocket: Implementation][pca]\n  - [:tv: YouTube Video: Implementing Principal Component Analysis (PCA) from Scratch][pca_youtube]\n- TF-IDF from Scratch\n  - [:rocket: Implementation][tfidf]\n  - [:tv: YouTube Video: Implementing TF-IDF from Scratch][tfidf_youtube]\n\n## License\n\n[MIT License][license]\n\n[license]: LICENSE\n[sigmoid]: activation/sigmoid.py\n[sigmoid_theory]: https://en.wikipedia.org/wiki/Sigmoid_function\n[sigmoid_plot]: activation/plots/sigmoid.png\n[sigmoid_youtube]: https://www.youtube.com/watch?v=oxC3T_-_Amw\n[relu]: activation/relu.py\n[relu_theory]: https://en.wikipedia.org/wiki/Rectifier_(neural_networks)\n[relu_plot]: activation/plots/relu.png\n[relu_youtube]: https://www.youtube.com/watch?v=93qjwrP7PfE\n[leaky_relu]: activation/leaky_relu.py\n[leaky_relu_theory]: https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#Leaky_ReLU\n[leaky_relu_plot]: activation/plots/leaky_relu.png\n[leaky_relu_youtube]: https://www.youtube.com/watch?v=1HLKeWG0qnE\n[gelu]: activation/gelu.py\n[gelu_theory]: https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#Gaussian-error_linear_unit_(GELU)\n[gelu_plot]: activation/plots/gelu.png\n[gelu_youtube]: https://www.youtube.com/watch?v=1HLKeWG0qnE\n[swish]: activation/swish.py\n[swish_theory]: https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#SiLU\n[swish_plot]: activation/plots/swish.png\n[swish_youtube]: https://www.youtube.com/watch?v=1HLKeWG0qnE\n[serf]: activation/serf.py\n[serf_theory]: https://arxiv.org/abs/2108.09598\n[serf_plot]: activation/plots/serf.png\n[serf_youtube]: https://www.youtube.com/watch?v=CLjmEuCxuT4\n[serf_reddit]: https://www.reddit.com/r/MachineLearning/comments/uhgupq/r_d_serf_activation_function_improving_swish/\n[tanh]: activation/tanh.py\n[tanh_theory]: https://en.wikipedia.org/wiki/Hyperbolic_functions#Exponential_definitions\n[tanh_plot]: activation/plots/tanh.png\n[tanh_youtube]: https://www.youtube.com/watch?v=MSi1tobj-jg\n[distance_functions]: function/distance.py\n[distance_functions_theory]: https://en.wikipedia.org/wiki/Similarity_measure\n[distance_functions_youtube]: https://www.youtube.com/watch?v=50G47n42-9o\n[convolution]: function/convolution.py\n[convolution_theory]: https://en.wikipedia.org/wiki/Convolution\n[convolution_youtube]: https://www.youtube.com/watch?v=pmyulQwV62k\n[gradient_descent_theory]: theory/gradient_descent/gradient_descent.pdf\n[gradient_descent_plot]: theory/gradient_descent/convex_and_non_convex_plot.png\n[gradient_descent_youtube]: https://www.youtube.com/watch?v=mV247Fe1DJc\n[gaussian_naive_bayes]: model/ml/gaussian_naive_bayes.py\n[gaussian_naive_bayes_theory]: https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Gaussian_naive_Bayes\n[gaussian_naive_bayes_youtube]: https://www.youtube.com/watch?v=maJIRFeQBVI\n[k_nearest_neighbors]: model/ml/k_nearest_neighbors.py\n[k_nearest_neighbors_theory]: https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm\n[k_nearest_neighbors_youtube]: https://www.youtube.com/watch?v=8SFTAcZb9i4\n[linear_regression]: model/ml/linear_regression.py\n[linear_regression_theory]: https://en.wikipedia.org/wiki/Linear_regression\n[linear_regression_youtube]: https://www.youtube.com/watch?v=7FdQZ9r41LU\n[logistic_regression]: model/ml/logistic_regression.py\n[logistic_regression_theory]: https://en.wikipedia.org/wiki/Logistic_regression\n[logistic_regression_computing_gradients]: theory/gradients/logistic_regression/logistic_regression.pdf\n[logistic_regression_youtube]: https://www.youtube.com/watch?v=YDa3rX9yLCE\n[k_means_clustering]: model/ml/k_means_clustering.py\n[k_means_clustering_theory]: https://en.wikipedia.org/wiki/K-means_clustering\n[k_means_clustering_youtube]: https://www.youtube.com/watch?v=NfPGFSUM-nI\n[pca]: representation/pca.py\n[pca_youtube]: https://www.youtube.com/watch?v=Gx7MmwmBFaM\n[tfidf]: representation/tfidf.py\n[tfidf_youtube]: https://www.youtube.com/watch?v=otgVLfBabKk\n[implement]: https://www.youtube.com/watch?v=maJIRFeQBVI\u0026list=PLG8XxYPkVOUvVzz1ZKcGAJpIBK7GRrFYR\n[api_design_and_numerical_stability_youtube]: https://www.youtube.com/watch?v=BOoTX0hkO6k\n[the_reproducible_mnist]: https://github.com/davidoniani/mnist\n[mnist_cnn]: model/dl/mnist_cnn.py\n[deep_learning_setup_youtube]: https://www.youtube.com/watch?v=2JkJZQP9dHg\n[vgg]: model/dl/vgg.py\n[vgg_youtube]: https://www.youtube.com/watch?v=0Ak4i2j_diM\n[gmf]: model/dl/gmf.py\n[gmf_youtube]: https://www.youtube.com/watch?v=gZgftF5hZOs\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foniani%2Fai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foniani%2Fai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foniani%2Fai/lists"}