{"id":18674053,"url":"https://github.com/jmaczan/ffnn","last_synced_at":"2025-07-04T23:06:49.164Z","repository":{"id":205474051,"uuid":"711123612","full_name":"jmaczan/ffnn","owner":"jmaczan","description":"Feedforward Neural Network from scratch - backpropagation, gradient descent, activation functions","archived":false,"fork":false,"pushed_at":"2023-11-24T12:19:39.000Z","size":59,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-18T11:07:09.484Z","etag":null,"topics":["activation-functions","backpropagation","educational","feedforward-neural-network","from-scratch","gradient-descent","machine-learning","neural-network","nn","python","rectifier","relu","softmax"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jmaczan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-28T09:34:28.000Z","updated_at":"2024-05-29T06:22:39.000Z","dependencies_parsed_at":"2023-11-11T21:23:32.435Z","dependency_job_id":"7fa8b2e4-98b3-4c7b-b089-c2040e195b05","html_url":"https://github.com/jmaczan/ffnn","commit_stats":null,"previous_names":["jmaczan/neural-network","jmaczan/ffnn"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jmaczan/ffnn","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmaczan%2Fffnn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmaczan%2Fffnn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmaczan%2Fffnn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmaczan%2Fffnn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jmaczan","download_url":"https://codeload.github.com/jmaczan/ffnn/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmaczan%2Fffnn/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263632060,"owners_count":23491530,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activation-functions","backpropagation","educational","feedforward-neural-network","from-scratch","gradient-descent","machine-learning","neural-network","nn","python","rectifier","relu","softmax"],"created_at":"2024-11-07T09:17:21.749Z","updated_at":"2025-07-04T23:06:49.144Z","avatar_url":"https://github.com/jmaczan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# neural-network\n\n🐱 Neural Network from a very scratch - backpropagation, gradient descent, activation functions\n\n## prerequisities\n\ninstall anaconda, so you have `conda` available in shell\n\n## development\n\n```\n# To activate this environment, use\n#\n#     $ conda activate nn\n#\n# To deactivate an active environment, use\n#\n#     $ conda deactivate\n```\n\nexport env settings to .yml:\n\n```\nconda env export --from-history \u003e environment.yml\n```\n\n## notes\n\nsoftmax derivative:\n\n$$\nsoftmax(z_i) = \\frac{e^{z_i}}{\\sum_{j=1}^K{e^{z_j}}}\n$$\n\nwe need two derivatives of softmax - with respect to $z_i$ and with respect to $z_k$ when $k \\ne i$\n\nlet's say $softmax(z_i) = S_i$. derivative of $S_i$ w.r.t $z_i$:\n\n$$\n\\frac{\\partial{S_i}}{\\partial{z_i}}=\\frac{\\partial}{\\partial{z_i}}(\\frac{e^{z_i}}{\\sum_{j=1}^K{e^{z_j}}})\n$$\n\napply the quotient rule of differentiation\n\n$$\n\\frac{\\partial}{\\partial {x}}(\\frac{f}{g})=\\frac{f'g - fg'}{g^2}\n$$\n\nso then\n\n$$\n\\frac{\\partial}{\\partial{z_i}}(\\frac{e^{z_i}}{\\sum_{j=1}^K{e^{z_j}}})=\\frac{e^{z_i}\\sum_{j=1}^K{e^{z_j}} - e^{z_i}\\sum_{j=1}^K{e^{z_j}}'}{(\\sum_{j=1}^K{e^{z_j}})^2}\n$$\n\nderivative of sum ${\\sum_{j=1}^K{e^{z_j}}}$ is ${e^{z_i}}$, because all other derivatives of $e^(z_k)$ w.r.t $e^{z_i}$ are $0$\n\n$$\n\\frac{e^{z_i}\\sum_{j=1}^K{e^{z_j}} - e^{z_i}e^{z_i}}{(\\sum_{j=1}^K{e^{z_j}})^2} = \\frac{e^{z_i}(\\sum_{j=1}^K{e^{z_j}} - e^{z_i})}{(\\sum_{j=1}^K{e^{z_j}})^2}=\\frac{e^{z_i}}{\\sum_{j=1}^K{e^{z_j}}}\\frac{(\\sum_{j=1}^K{e^{z_j}})-e^{z_i}}{\\sum_{j=1}^K{e^{z_j}}}=S_i(1-S_i)\n$$\n\nnow another derivative of $S_i$ w.r.t $z_k$ when $k \\ne i$:\n\n$$\n\\frac{\\partial{S_i}}{\\partial{z_k}}=\\frac{\\partial}{\\partial{z_k}}(\\frac{e^{z_i}}{\\sum_{j=1}^K{e^{z_j}}})\n$$\n\nonce again let's apply the quotient rule of differentiation\n\n$$\n\\frac{\\partial}{\\partial{z_k}}(\\frac{e^{z_i}}{\\sum_{j=1}^K{e^{z_j}}})=\\frac{\\frac{\\partial{e^{z_i}}}{\\partial{z_k}}\\sum_{j=1}^K{e^{z_j}}-e^{z_i}\\frac{\\partial \\sum_{j=1}^K{e^{z_j}}}{\\partial{z_k}}}{(\\sum_{j=1}^K{e^{z_j}})^2}\n$$\n\nfirst term in numerator is $0$, because from the basic principles of partial differentiation, where the derivative of a function with respect to a variable that does not appear in the function is $0$\n\nthen there's a sum, from which all terms are $0$ except $e^{z_k}$. so then:\n\n$$\n\\frac{-e^{z_i}e^{z_k}}{(\\sum_{j=1}^K{e^{z_j}})^2}=S_i\\frac{-e^{z_k}}{\\sum_{j=1}^K{e^{z_j}}}\n$$\n\nSo $\\frac{-e^{z_k}}{\\sum_{j=1}^K{e^{z_j}}}$ is softmax but with ${z_k}$ as a parameter,so then:\n\n$$\n\\frac{\\partial{S_i}}{\\partial{z_k}}=-S_i \\cdot S_k\n$$\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmaczan%2Fffnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjmaczan%2Fffnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmaczan%2Fffnn/lists"}