{"id":13425058,"url":"https://github.com/gokadin/ai-simplest-network","last_synced_at":"2025-03-15T19:32:31.087Z","repository":{"id":102446061,"uuid":"199751413","full_name":"gokadin/ai-simplest-network","owner":"gokadin","description":"The simplest form of an artificial neural network explained and demonstrated. ","archived":false,"fork":false,"pushed_at":"2020-05-31T02:49:54.000Z","size":265,"stargazers_count":397,"open_issues_count":0,"forks_count":30,"subscribers_count":23,"default_branch":"master","last_synced_at":"2024-10-28T03:39:09.911Z","etag":null,"topics":["artificial-intelligence","artificial-neural-networks","golang","gradient-descent","machine-learning","tutorial"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gokadin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-31T01:05:31.000Z","updated_at":"2024-10-22T07:07:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"f1480193-9ed0-4b2f-bff0-a0c4a6cbb852","html_url":"https://github.com/gokadin/ai-simplest-network","commit_stats":null,"previous_names":["gokadin/ai-linear-associative-network"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gokadin%2Fai-simplest-network","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gokadin%2Fai-simplest-network/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gokadin%2Fai-simplest-network/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gokadin%2Fai-simplest-network/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gokadin","download_url":"https://codeload.github.com/gokadin/ai-simplest-network/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243781790,"owners_count":20347146,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","artificial-neural-networks","golang","gradient-descent","machine-learning","tutorial"],"created_at":"2024-07-31T00:01:03.546Z","updated_at":"2025-03-15T19:32:30.738Z","avatar_url":"https://github.com/gokadin.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Simplest artificial neural network\n\nThis is the simplest artificial neural network possible explained and demonstrated. \n\n## This is part 1 of a series of github repos on neural networks\n\n- part 1 - simplest network (**you are here**)\n- [part 2 - backpropagation](https://github.com/gokadin/ai-backpropagation)\n- [part 3 - backpropagation-continued](https://github.com/gokadin/ai-backpropagation-continued)\n\n## Table of Contents\n\n- [Theory](#theory)  \n  - [Mimicking neurons](#mimicking-neurons)\n  - [A simple example](#a-simple-example)\n  - [The error](#the-error)\n  - [Gradient descent](#gradient-descent)\n- [Code example](#code-example)\n- [References](#references)\n\n## Theory\n\n### Mimicking neurons\n\nArtificial neural networks are inspired by the brain by having interconnected artificial neurons store patterns and communicate with each other. \nThe simplest form of an artificial neuron has one or multiple inputs \u003cimg src=\"/tex/9fc20fb1d3825674c6a279cb0d5ca636.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=14.045887349999989pt height=14.15524440000002pt/\u003e each having a specific weight \u003cimg src=\"/tex/c2a29561d89e139b3c7bffe51570c3ce.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=16.41940739999999pt height=14.15524440000002pt/\u003e and one output \u003cimg src=\"/tex/deceeaf6940a8c7a5a02373728002b0f.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=8.649225749999989pt height=14.15524440000002pt/\u003e. \n\n![alt text](readme-images/perceptron.jpg)\n\nAt the simplest level, the output is the sum of its inputs times its weights. \n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/c2d2775d67e954682fac686e557baed2.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=88.33802834999999pt height=44.89738935pt/\u003e\u003c/p\u003e\n\n### A simple example\n\nThe purpose of a network is to learn a certain output \u003cimg src=\"/tex/deceeaf6940a8c7a5a02373728002b0f.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=8.649225749999989pt height=14.15524440000002pt/\u003e given certain input(s) \u003cimg src=\"/tex/332cc365a4987aacce0ead01b8bdcc0b.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=9.39498779999999pt height=14.15524440000002pt/\u003e by approximating a complex function with many parameters \u003cimg src=\"/tex/31fae8b8b78ebe01cbfbe2fe53832624.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=12.210846449999991pt height=14.15524440000002pt/\u003e that we couldn't come up with ourselves. \n\nSay we have a network with two inputs \u003cimg src=\"/tex/f9b6dcc9279f659321ac3e1098b0ba4f.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=59.69172164999999pt height=21.18721440000001pt/\u003e and \u003cimg src=\"/tex/bf84a893effff44b6d014b2b60460585.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=59.69172164999999pt height=21.18721440000001pt/\u003e and two weights \u003cimg src=\"/tex/4b4518f1b7f0fb1347fa21506ebafb19.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=18.32105549999999pt height=14.15524440000002pt/\u003e and \u003cimg src=\"/tex/f7eb0e840408d84a0c156d6efb611f3e.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=18.32105549999999pt height=14.15524440000002pt/\u003e.  \n\nThe idea is to adjust the weights in such a way that the given inputs produce the desired output. \n\nWeights are normally initialized randomly since we can't know their optimal value ahead of time, however for simplicity we will initialize them both to \u003cimg src=\"/tex/034d0a6be0424bffe9a6e7ac9236c0f5.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=8.219209349999991pt height=21.18721440000001pt/\u003e. \n\n![alt text](readme-images/perceptron-example.jpg)\n\nIf we calculate the output of this network, we will get \n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/48c4f6073c4655b74cebf396493c9228.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=322.4824614pt height=13.789957499999998pt/\u003e\u003c/p\u003e\n\n### The error\n\nIf the output \u003cimg src=\"/tex/deceeaf6940a8c7a5a02373728002b0f.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=8.649225749999989pt height=14.15524440000002pt/\u003e doesn't match the expected target value, then we have an error.  \nFor example, if we wanted to get a target value of \u003cimg src=\"/tex/72dd6f18a88876a3e9697d3cd247d0b8.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=48.858371099999985pt height=21.18721440000001pt/\u003e then we would have a difference of \n\n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/3a5765ad1562dc76491f974c309877cd.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=161.6168763pt height=13.789957499999998pt/\u003e\u003c/p\u003e\n\nOne common way to measure the error (also referred to as the cost function) is to use the mean squared error:\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/f9a1a3500d0047642b9e0aa44e315d61.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=101.17868145pt height=32.990165999999995pt/\u003e\u003c/p\u003e\n\nIf we had multiple associations of inputs and target values, then the error becomes the average sum of each association. \n\n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/0b27b4f92bb0aa649d3980861c413f8b.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=147.88537499999998pt height=44.89738935pt/\u003e\u003c/p\u003e\n\nWe use the mean squared error to measure how far away the results are from our desired target. The squaring removes negative signs and gives more weight to bigger differences between output and target. \n\nTo rectify the error, we would need to adjust the weights in a way that the output matches our target. In our example, lowering \u003cimg src=\"/tex/4b4518f1b7f0fb1347fa21506ebafb19.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=18.32105549999999pt height=14.15524440000002pt/\u003e from \u003cimg src=\"/tex/f58ed17486d1735419372f2b7d091779.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=21.00464354999999pt height=21.18721440000001pt/\u003e to \u003cimg src=\"/tex/cde2d598001a947a6afd044a43d15629.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=21.00464354999999pt height=21.18721440000001pt/\u003e would do the trick, since \n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/385570808d7e0408dd491b7647e205de.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=236.5025355pt height=13.789957499999998pt/\u003e\u003c/p\u003e\n\nHowever, in order to adjust the weights of our neural networks for many different inputs and target values, we need a *learning algorithm* to do this for us automatically. \n\n### Gradient descent\n\nThe idea is to use the error to understand how each weight should be adjusted so that the error is minimized, but first, we need to learn about gradients. \n\n##### What is a gradient?\n\nIt's essentially a vector pointing to the direction of the steepest ascent of a function. The gradient is denoted with \u003cimg src=\"/tex/47c28f1929c18f887420345e9225e08b.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=13.69867124999999pt height=22.465723500000017pt/\u003e and is simply the partial derivative of each variable of a function expressed as a vector. \n\nIt looks like this for a two variable function:\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/b142e84f3f77e6dc3144eb723cd4510d.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=303.75993285pt height=37.9216761pt/\u003e\u003c/p\u003e\n\nLet's inject some numbers and calculate the gradient with a simple example. \nSay we have a function \u003cimg src=\"/tex/8f41fe11e0e63d98f2fe17a78df752c8.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=138.37136445pt height=26.76175259999998pt/\u003e, then the gradient would be\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/ecdfccd2a7dad070b8c82c8e0f741e6d.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=148.2573411pt height=19.9563243pt/\u003e\u003c/p\u003e\n\n##### What is gradient descent?\n\nThe *descent* part simply means using the gradient to find the direction of steepest ascent of our function and then going in the opposite direction by a *small* amount many times to find the function *global (or sometimes local) minimum*.  \n\nWe use a constant called the **learning rate**, denoted with \u003cimg src=\"/tex/7ccca27b5ccc533a2dd72dc6fa28ed84.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=6.672392099999992pt height=14.15524440000002pt/\u003e to define how small of a step to take in that direction.  \n\nIf \u003cimg src=\"/tex/7ccca27b5ccc533a2dd72dc6fa28ed84.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=6.672392099999992pt height=14.15524440000002pt/\u003e is too large, then we risk overshooting the function minimum, but if it's too low then the network will take longer to learn and we risk getting stuck in a shallow local minimum. \n\n![alt text](readme-images/gradient-descent.jpg)\n\n##### Gradient descent applied to our example network\n\nFor our two weights \u003cimg src=\"/tex/4b4518f1b7f0fb1347fa21506ebafb19.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=18.32105549999999pt height=14.15524440000002pt/\u003e and \u003cimg src=\"/tex/f7eb0e840408d84a0c156d6efb611f3e.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=18.32105549999999pt height=14.15524440000002pt/\u003e we need to find the gradient of those weights with respect to the error function \u003cimg src=\"/tex/84df98c65d88c6adf15d4645ffa25e47.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=13.08219659999999pt height=22.465723500000017pt/\u003e  \n\n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/912be46ac0db99c8544f0800527d4b9f.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=147.62782815pt height=36.2778141pt/\u003e\u003c/p\u003e\n\nFor both \u003cimg src=\"/tex/4b4518f1b7f0fb1347fa21506ebafb19.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=18.32105549999999pt height=14.15524440000002pt/\u003e and \u003cimg src=\"/tex/f7eb0e840408d84a0c156d6efb611f3e.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=18.32105549999999pt height=14.15524440000002pt/\u003e, we can find the gradient by using the chain rule\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/8d7dec910e58764e57aa27f328a7396c.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=570.4098938999999pt height=39.452455349999994pt/\u003e\u003c/p\u003e\n\nFrom now on we will denote the \u003cimg src=\"/tex/053259646a84530b56faa65084783f24.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=76.5775791pt height=28.92634470000001pt/\u003e as the \u003cimg src=\"/tex/38f1e2a089e53d5c990a82f284948953.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=7.928075099999989pt height=22.831056599999986pt/\u003e term for simplicity. \n\nOnce we have the gradient, we can update our weights by subtracting the calculated gradient times the learning rate. \n\n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/f46613c78403dce8eed0b6093ff36d28.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=222.19498950000002pt height=15.52509255pt/\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/223d39de0113b6136f26962ed907c8aa.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=222.19498950000002pt height=15.52509255pt/\u003e\u003c/p\u003e\n\nAnd we repeat this process until the error is minimized and is close enough to zero. \n\n## Code example\n\nThe included example teaches the following dataset to a neural network with two inputs and one output using gradient descent:\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"/tex/0cdd43e831c22b1560861b7a3e660010.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=233.52364695pt height=39.452455349999994pt/\u003e\u003c/p\u003e\n\nOnce learned, the network should output ~0 when given two \u003cimg src=\"/tex/034d0a6be0424bffe9a6e7ac9236c0f5.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=8.219209349999991pt height=21.18721440000001pt/\u003es and ~\u003cimg src=\"/tex/034d0a6be0424bffe9a6e7ac9236c0f5.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=8.219209349999991pt height=21.18721440000001pt/\u003e when given a \u003cimg src=\"/tex/034d0a6be0424bffe9a6e7ac9236c0f5.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=8.219209349999991pt height=21.18721440000001pt/\u003e and a \u003cimg src=\"/tex/29632a9bf827ce0200454dd32fc3be82.svg?invert_in_darkmode\u0026sanitize=true\" align=middle width=8.219209349999991pt height=21.18721440000001pt/\u003e. \n\n### How to run\n\n#### Online on repl.it\n\n[![Run on Repl.it](https://repl.it/badge/github/gokadin/ai-simplest-network)](https://repl.it/github/gokadin/ai-simplest-network)\n\n#### Docker\n\n``` bash\ndocker build -t simplest-network .\ndocker run --rm simplest-network\n```\n\n## References\n\n1. Artificial intelligence engines by James V Stone (2019)\n2. Complete guide on deep learning: http://neuralnetworksanddeeplearning.com/chap2.html","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgokadin%2Fai-simplest-network","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgokadin%2Fai-simplest-network","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgokadin%2Fai-simplest-network/lists"}