{"id":22451463,"url":"https://github.com/chanlumerico/autoprop","last_synced_at":"2025-03-27T12:26:52.076Z","repository":{"id":257336639,"uuid":"857960999","full_name":"ChanLumerico/autoprop","owner":"ChanLumerico","description":"Auto Propagation System for Complex Neural Networks of Luma Python library","archived":false,"fork":false,"pushed_at":"2024-10-28T11:19:20.000Z","size":38,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-01T17:11:25.196Z","etag":null,"topics":["autograd-engine","backpropagation-algorithm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ChanLumerico.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-16T03:23:22.000Z","updated_at":"2024-10-28T11:19:24.000Z","dependencies_parsed_at":"2024-10-28T12:46:29.327Z","dependency_job_id":"bcf76982-3977-4038-98e2-c9940b1b1430","html_url":"https://github.com/ChanLumerico/autoprop","commit_stats":null,"previous_names":["chanlumerico/autoprop"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChanLumerico%2Fautoprop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChanLumerico%2Fautoprop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChanLumerico%2Fautoprop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChanLumerico%2Fautoprop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ChanLumerico","download_url":"https://codeload.github.com/ChanLumerico/autoprop/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245843533,"owners_count":20681529,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autograd-engine","backpropagation-algorithm"],"created_at":"2024-12-06T06:08:05.770Z","updated_at":"2025-03-27T12:26:52.052Z","avatar_url":"https://github.com/ChanLumerico.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003cimg src=\"https://raw.githubusercontent.com/ChanLumerico/luma/main/img/title/ap_dark.png\" alt=\"logo\" height=\"50%\" width=\"50%\"\u003e\n\nAuto propagation system for complex neural networks of Luma Python library\n\n## What is AutoProp?\n\nAutoProp is a specialized package designed for automated dynamic graph construction within `Luma`'s neural components. \n\nIt facilitates seamless automatic feed-forward and backpropagation processes, streamlining neural network operations and enhancing flexibility in model building.\n\n## Why is Automated Propagation Needed?\n\n- **Dynamic Graph Construction**: Automatically builds the computational graph, allowing flexible and non-linear architectures without manual intervention.\n- **Automatic Feed-forwarding**: Ensures correct data flow between layers, handling dependencies and the order of operations during forward passes.\n- **Backpropagation \u0026 Gradient Calculation**: Automates gradient computation using the chain rule, ensuring efficient and accurate backpropagation through the graph.\n- **Error Handling \u0026 Optimization**: Detects errors early and applies optimizations (e.g., gradient clipping, batch normalization) automatically, reducing manual adjustments.\n\n---\n\n## How to Use `LayerNode`\n\n`LayerNode` is an abstraction representing nodes in a neural computational graph, encapsulating individual neural components such as layers or blocks.\n\n### 1️⃣ Insert a Neural Component\n\n`LayerNode` has three arguments:\n\n- **layer**: `LayerLike`\n- **merge_mode**: `MergeMode`, default=*MergeMode.SUM*\n- **name**: `str`, optional, default=*None*\n\n```python\nnode = LayerNode(\n    layer=Conv2D(3, 6, 3),\n    ...,\n)\n```\n\nPass `LayerLike` instance to the argument `layer` to encapsulate the neural component as a new neural node. \n\n### 2️⃣ Set Merge Mode\n\nA neural node can have multiple incoming branches from previous nodes, requiring it to resolve how to merge the incoming tensors during the propagation process. \n\nThis involves selecting or applying a specific merging strategy, such as concatenation, addition, or averaging, to combine the inputs into a single tensor before passing it through the node for further computation.\n\nFor this purpose, `AutoProp` has a dedicated class `MergeMode` that can specify the merging strategy.\n\n**Enum Class `MergeMode`**\n\n| Mode | Operation |\n| --- | --- |\n| `SUM` *(Default)* | Sum the tensors |\n| `CHCAT` | Concatenate over the channels of the tensors |\n| `HADAMARD` | Perform Hadamard(element-wise) product over the tensors |\n| `AVG` | Perform Element-wise averaging over the tensors |\n| `MAX` | Select the maximum elements among those of the tensors |\n| `MIN` | Select the minimum elements among those of the tensors |\n| `DOT` | Perform dot products over the tensors sequentially |\n| `SUB` | Subtract the tensors |\n\n```python\nnode = LayerNode(\n    layer=Conv2D(3, 6, 3),\n    merge_mode=MergeMode.CHCAT,\n    ...,\n)\n```\n\n### 3️⃣ Set Nickname\n\n`LayerNode` allows setting a nickname for each node to distinguish it from other potentially existing nodes. \n\nThis feature helps in identifying and referencing specific nodes within a neural computational graph, especially in complex architectures where multiple nodes of the same type or function may exist.\n\n```python\nnode = LayerNode(\n    layer=Conv2D(3, 6, 3),\n    merge_mode=MergeMode.CHCAT,\n    name=\"my_node\",\n)\n```\n\nIf not set, ‘LayerNode’ is automatically assigned as a nickname in default.\n\n---\n\n## How to use `LayerGraph`\n\n`LayerGraph` is a graph representation of neural nodes, providing support for dynamic structural modifications. \n\nIt facilitates automatic feed-forward and backpropagation processes, making it a central component of the `AutoProp` system. \n\nThis allows flexible, adaptive network design and efficient computational graph management.\n\n### 1️⃣ Instantiation\n\n`LayerGraph` has three arguments:\n\n- **graph**: `Dict[LayerNode, List[LayerNode]]`\n- **root**: `LayerNode`\n- **term**: `LayerNode`\n\n```python\nA = LayerNode(..., name=\"A\")\nB = LayerNode(..., name=\"B\")\nC = LayerNode(..., name=\"C\")\nD = LayerNode(..., name=\"D\")\n\nmodule = LayerGraph(\n    graph={\n        A: [B, C],\n        B: [D],\n        C: [D],\n    },\n    root=A,\n    term=D,\n)\n```\n\nThe argument `graph` requires a dictionary of `LayerNode`, formatted as an adjacency list. \n\nIn this structure, each dictionary key represents a node, and its corresponding value contains a list of nodes that follow it. \n\nAdditionally, when constructing the graph, you must specify the root node and the terminal node using the arguments `root` and `term`, respectively.\n\n### 2️⃣ Build the Graph\n\nCall the `build` method to finalize the graph using the defined nodes and the specified root and terminal nodes.\n\n```python\nmodule.build()\n```\n\n### 3️⃣ Perform Forward and Backward Propagations\n\nUse the `forward` and `backward` methods to perform feed-forwarding and backpropagation, respectively, on the neural graph you’ve constructed.\n\n## Examples\n\n**Linearly Connected Graph**\n\nStructure:\n\n```\nConv2D --\u003e BatchNorm2D --\u003e ReLU\n```\n\nImplementation:\n\n```python\nconv = LayerNode(Conv2D(3, 6, 3))\nbn = LayerNode(BatchNorm2D(6))\nrelu = LayerNode(Activation.ReLU())\n\nmodule = LayerGraph(\n    graph={conv: [bn], bn: [relu]},\n    root=conv,\n    term=relu,\n)\nmodule.build()\n```\n\n**Shortcut Connection**\n\nStructure:\n\n```\nRoot -+-\u003e Conv2D --\u003e ReLU -+-\u003e Term\n      |                    |\n      +--------------------+\n```\n\nImplementation:\n\n```python\nroot = LayerNode(Identity())\nconv = LayerNode(Conv2D(3, 6, 3))\nrelu = LayerNode(Activation.ReLU())\nterm = LayerNode(\n    Identity(), MergeMode.SUM,\n)\n\nmodule = LayerGraph(\n    graph={\n        conv: [relu, term], \n        relu: [term],\n    },\n    root=conv,\n    term=term,\n)\nmodule.build()\n```\n\n**Channel Concatenation**\n\nStructure:\n\n```\n      +-\u003e Conv2D --\u003e ReLU -+\n      |                    |\nRoot -+-\u003e Conv2D --\u003e ReLU -+-\u003e Term\n      |                    |\n      +-\u003e Conv2D --\u003e ReLU -+\n```\n\nImplementation:\n\n```python\nroot = LayerNode(Identity())\nbranch_1 = LayerNode(\n    Sequential(\n        Conv2D(3, 6, 3),\n        Activation.ReLU(),\n    ),\n)\nbranch_2 = LayerNode(\n    Sequential(\n        Conv2D(3, 12, 3),\n        Activation.ReLU(),\n    ),\n)\nbranch_3 = LayerNode(\n    Sequential(\n        Conv2D(3, 18, 3),\n        Activation.ReLU(),\n    ),\n)\nterm = LayerNode(\n    Identity(),\n    MergeMode.CHCAT,\n)\n\nmodule = LayerGraph(\n    graph={\n        root: [branch_1, branch_2, branch_3],\n        branch_1: [term],\n        branch_2: [term],\n        branch_3: [term],\n    },\n    root=root,\n    term=term,\n)\nmodule.build()\n```\n\n---\n\n## Summary\n\n`AutoProp` is a core component of the Luma framework, designed to automate dynamic graph construction for neural network components. \n\nIt simplifies model building by managing the creation and modification of neural graphs, supporting automatic feed-forwarding and backpropagation. \n\nKey features include:\n\n- **LayerNode**: Represents individual neural layers or blocks in the graph.\n- **LayerGraph**: Manages the neural graph structure, allowing dynamic changes and defining node relationships through adjacency lists.\n- **Graph Construction**: Requires setting `root` and `term` nodes, with the `build` method finalizing the structure.\n- **Automated Propagation**: Handles feed-forward and backpropagation with `forward` and `backward` methods, ensuring efficient data flow and gradient calculation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchanlumerico%2Fautoprop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchanlumerico%2Fautoprop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchanlumerico%2Fautoprop/lists"}