{"id":23516863,"url":"https://github.com/18520339/nn-auto-differentiation-engine","last_synced_at":"2026-05-13T20:34:13.426Z","repository":{"id":265911954,"uuid":"896841240","full_name":"18520339/nn-auto-differentiation-engine","owner":"18520339","description":"C++ version of Andrej Karpathy's micrograd with Keras-like Sequential API and auto differentiation engine, facilitating Tensor arithmetic using chain-rule \u0026 topological sort, mini-batch Gradient Descent, and learning rate scheduling","archived":false,"fork":false,"pushed_at":"2025-01-12T21:43:13.000Z","size":202,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-27T16:37:11.891Z","etag":null,"topics":["autograd","backpropagation","chain-rule","cplusplus-17","gradient-descent","neural-network","tensorflow"],"latest_commit_sha":null,"homepage":"https://youtu.be/tH6AvNnQnLQ","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/18520339.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-01T12:43:25.000Z","updated_at":"2025-12-01T21:27:38.000Z","dependencies_parsed_at":"2025-01-12T22:25:24.324Z","dependency_job_id":"466c2c74-9446-4323-8666-ca09e9bf646f","html_url":"https://github.com/18520339/nn-auto-differentiation-engine","commit_stats":null,"previous_names":["18520339/nn-autograd-engine"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/18520339/nn-auto-differentiation-engine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/18520339%2Fnn-auto-differentiation-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/18520339%2Fnn-auto-differentiation-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/18520339%2Fnn-auto-differentiation-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/18520339%2Fnn-auto-differentiation-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/18520339","download_url":"https://codeload.github.com/18520339/nn-auto-differentiation-engine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/18520339%2Fnn-auto-differentiation-engine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32999490,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T13:14:54.681Z","status":"ssl_error","status_checked_at":"2026-05-13T13:14:51.610Z","response_time":115,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autograd","backpropagation","chain-rule","cplusplus-17","gradient-descent","neural-network","tensorflow"],"created_at":"2024-12-25T15:15:21.242Z","updated_at":"2026-05-13T20:34:13.411Z","avatar_url":"https://github.com/18520339.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# **C++17** Autograd Neural Network Framework\n\n![image](https://github.com/user-attachments/assets/3879a2b3-a978-4b32-b4ac-b3907377d000)\n\n\u003e Demo: https://youtu.be/tH6AvNnQnLQ\n\nA flexible and extensible framework in pure **C++17** designed to facilitate the construction, training, and evaluation of Neural Networks. Inspired by modern Deep Learning frameworks like [PyTorch](https://pytorch.org) and [TensorFlow](https://www.tensorflow.org), this project provides:\n- A collection of modular components with an **automatic differentiation engine** as essential building blocks for experimenting with custom **model architectures**.\n- A foundational understanding of how Neural Network and its **computational graph** can be implemented from scratch, offering insights into the underlying mechanics of **forward** and **backward** propagation, gradient computation using **chain rule**, and its optimization using **Gradient Descent**.\n\nThis project serves both educational purposes for those interested in understanding the internals of Neural Networks and practical applications where a lightweight, efficient, and customizable framework is needed.\n\n# Key Features\n\n- **Pure C++17 Implementation**: No external dependencies, leveraging modern C++ features for efficient. Memory management is handled using smart pointers (`std::shared_ptr`), minimizing the risk of memory leaks.\n- **Tensor Operations**: Support for tensor arithmetic with automatic gradient tracking, which performs and manipulates mathematical operations with tensors, like `+`, `*`, or activation functions, etc.\n  - **Automatic Differentiation**: Automatically compute gradients efficiently during **backpropagation**.\n  - **Activation Functions**: Include common activation functions like `sigmoid`, `tanh`, `relu`, and `softmax`.\n- **Sequential Model**: A high-level API similar to [TensorFlow](https://www.tensorflow.org/guide/keras/sequential_model) for building and training Neural Network models using a sequential stack of layers.\n  - **Batch Processing**: Support for training models with **Mini-batch Gradient Descent**.\n  - **Loss Functions**: Implementations of standard loss functions like **Mean Squared Error** or **Binary/Categorical Cross-Entropy** Loss.\n  - **Evaluation Metrics**: Functions to evaluate the performance of models using metrics like `accuracy`.\n  - **Learning Rate Scheduler**: Offer schedulers for dynamic learning rate adjustment during training.\n  - **Logging**: Tools for model summarizing, monitoring, and logging training progress like [TensorFlow's Model.summary()](https://www.tensorflow.org/api_docs/python/tf/keras/Model#summary)\n- **Data Preprocessing**: Utilities for loading, shuffling, splitting, scaling, and encoding datasets like in [scikit-learn](https://scikit-learn.org/stable).\n\n# Quick Started\n\nI have included some example scripts demonstrating how to use the engine. Compile any of these following and run the executable:\n\n\u003ctable\u003e\n\u003ctr\u003e\n  \u003cth\u003eExample\u003c/th\u003e\n  \u003cth\u003eDescription\u003c/th\u003e\n  \u003cth\u003eCompile and Run\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003ca href=\"./backward_test.cpp\"\u003ebackward_test.cpp\u003c/a\u003e\u003c/td\u003e\n  \u003ctd\u003eDemonstrate/verify the correctness of auto-differentiation by calculating the gradients of a simple computation graph.\u003c/td\u003e\n  \u003ctd\u003e\n  \n  ```bash\n  g++ backward_test.cpp -o verify\n  ./verify\n  ```\n  \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003ca href=\"./train_cubic.cpp\"\u003etrain_cubic.cpp\u003c/a\u003e\u003c/td\u003e\n  \u003ctd\u003eTrain a Neural Network to approximate a cubic function y = 2x³ + 3x² - 3x, demonstrating regression capabilities.\u003c/td\u003e\n  \u003ctd\u003e\n  \n  ```bash\n  g++ train_cubic.cpp -o train_cubic\n  ./train_cubic\n  ```\n  \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003ca href=\"./train_iris.cpp\"\u003etrain_iris.cpp\u003c/a\u003e\u003c/td\u003e\n  \u003ctd\u003eLoad, preprocess, and train a Neural Network for multi-class classification on the \u003ca href=\"https://www.kaggle.com/datasets/arshid/iris-flower-dataset\"\u003eIris\u003c/a\u003e dataset using one-hot encoding.\u003c/td\u003e\n  \u003ctd\u003e\n  \n  ```bash\n  g++ train_iris.cpp -o train_iris\n  ./train_iris\n  ```\n  \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003ca href=\"./train_mnist.cpp\"\u003etrain_mnist.cpp\u003c/a\u003e\u003c/td\u003e\n  \u003ctd\u003eSimilar to \u003ca href=\"./train_iris.cpp\"\u003etrain_iris.cpp\u003c/a\u003e, but train a model on the \u003ca href=\"https://www.kaggle.com/datasets/oddrationale/mnist-in-csv\"\u003eMNIST\u003c/a\u003e dataset for digit recognition with pixels as input features.\u003c/td\u003e\n  \u003ctd\u003e\n  \n```bash\ng++ train_mnist.cpp -o train_mnist\n./train_mnist\n```\n  \u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n***[Note]** Before running, ensure that:\n- You have a **C++17** (or higher) compatible compiler.\n- The **Iris** and **MNIST** datasets are available in the specified `data` directory. Here, I just simply use their `.csv` files directly from **Kaggle**.\n\n# Core Components\n\nThe engine is organized into several header files (`.hpp`) located in the [n2n_autograd](./n2n_autograd/) folder, each including classes and functions responsible for different aspects of the Neural Network and **auto-differentiation** operations.\n\n**Table of Contents**:\n- [I. `tensor.hpp` | Tensor Class and Auto-differentiation Engine](#i-tensorhpp--tensor-class-and-auto-differentiation-engine-)\n  - [1.1. Internal pointers and functions for tracking operations and dependencies](#11-internal-pointers-and-functions-for-tracking-operations-and-dependencies-)\n  - [1.2. Public Members](#12-public-members-)\n  - [1.3. Usage Example](#13-usage-example-)\n- [II. `layers.hpp` | Neural Network Layers](#ii-layershpp--neural-network-layers-)\n  - [2.1. class `Initializers`](#21-class-initializers-)\n  - [2.2. class `Neuron`](#22-class-neuron-)\n  - [2.3. class `Dense`](#23-class-dense-)\n  - [2.4. Usage Example](#24-usage-example-)\n- [III. `models.hpp` | Neural Network Model and Training Utilities](#iii-modelshpp--neural-network-model-and-training-utilities-)\n  - [3.1. class `Sequential`](#31-class-sequentialoutputtype-like-tensorflow-)\n  - [3.2. Usage Example](#32-usage-example-)\n- [IV. `losses.hpp` | Loss Functions for guiding optimization process](#iv-losseshpp--loss-functions-for-guiding-optimization-process-)\n  - [4.1. class `Loss`](#41-class-loss-)\n  - [4.2. Usage Example](#42-usage-example-)\n- [V. `metrics.hpp` | Metric Functions for Model Evaluation](#v-metricshpp--metric-functions-for-model-evaluation-)\n  - [5.1. class `Metrics`](#51-class-metrics-)\n  - [5.2. Usage Example](#52-usage-example-)\n- [VI. `optim.hpp` | Optimizers and Learning Rate Schedulers](#vi-optimhpp--optimizers-and-learning-rate-schedulers-)\n  - [6.1. class `LearningRateScheduler`](#61-class-learningratescheduler-)\n  - [6.2. class `WarmUpAndDecayScheduler`](#62-class-warmupanddecayscheduler-)\n  - [6.3. Usage Example](#63-usage-example-)\n- [VII. `preprocess.hpp` | Data Preprocessing Utilities](#vii-preprocesshpp--data-preprocessing-utilities-)\n  - [7.1. Data Loading](#71-data-loading-)\n  - [7.2. Data Shuffling and Splitting](#72-data-shuffling-and-splitting-)\n  - [7.3. class `StandardScaler`](#73-class-standardscaler-)\n  - [7.4. Usage Example](#74-usage-example-)\n- [VIII. `converters.hpp` | Data Conversion Utilities](#viii-convertershpp--data-conversion-utilities-)\n  - [8.1. *any* Conversion](#81-any-conversion-)\n  - [8.2. One-Hot Encoding](#82-one-hot-encoding-)\n  - [8.3. `Tensor` Conversion](#83-tensor-conversion-)\n\n## I. [`tensor.hpp`](./n2n_autograd/tensor.hpp) | Tensor Class and Auto-differentiation Engine [🔝](#core-components)\n\nContain the **`Tensor`** class, a core data structure of the **autograd** engine, representing a node or scalar value in the computation graph. It supports **automatic differentiation** by recording/maintaining references to its child **tensors** and the operations that produced them. \n\nWhen operations are performed on **tensors** (e.g., `addition`, `multiplication`), new **tensors** are created, and the graph is dynamically built. During **backpropagation**, the `gradients` with respect to each **`Tensor`** are computed by traversing this graph in **reverse topological order**. \n\nHere, the `local_backward` function will specify how the `gradient` is computed locally for the **`Tensor`** based on its `children`. For example, in `multiplication`, the `gradient` with respect to each operand is the product of the other operand and the upstream `gradient`.\n\n### 1.1. Internal pointers and functions for tracking operations and dependencies [🔝](#core-components)\n\n- *function\u003cvoid(const `Tensor`\\*)\u003e* `local_backward`: Lambda function to compute local `gradients` with respect to the **`Tensor`**.\n- *set\u003c**`TensorPtr`**\u003e* `children`: A set of child nodes (**tensors**) in the computational graph that are inputs to the operation producing this **`Tensor`**.\n- *string* `operation`: The operation that produced this **`Tensor`**.\n\n👉 *vector\u003c**`Tensor`**\\*\u003e* **topological_sort**(): Performs a topological sort of the computation graph, returning nodes in the order they should be processed during **backpropagation**.\n\n### 1.2. Public Members [🔝](#core-components)\n\n- *double* `data`: The numerical value of the **`Tensor`**.\n- *double* `gradient`: The accumulated gradient computed during **backpropagation**.\n- *string* `label`: An optional label for identification.\n\n👉 **Constructors**:\n- **`Tensor`**(*double* `_data`, *const string* `_label = \"\"`): Initializes a **`Tensor`** with a value and an optional label.\n- **`Tensor`**(*double* `_data`, *const set\u003c**`TensorPtr`**\u003e\u0026* `_children`, *const string\u0026* `_operation`): Initializes a **`Tensor`** resulting from an operation.\n\n👉 **Operator Supports with Automatic Gradient Tracking**:\n- Arithmetic operations: `+`, `-`, `*`, `/`, `pow`.\n- Unary operations: `-`, `exp`, `log`.\n- Activation Functions: `sigmoid`, `tanh`, `relu`.\n  - *static **`TensorPtr`*** **create**(*double* `_data`, *string* `_label = \"\"`): Static Factory method to create a shared pointer to a **`Tensor`**.\n  - *void* **backward**(): Performs **backpropagation** to compute `gradients` for all **tensors** throughout the computation graph using **reverse-mode automatic differentiation**.\n\n### 1.3. Usage Example [🔝](#core-components)\n\nCreate **tensors** and perform operations as you would with scalar values. Call the `backward()` method on the final output **`Tensor`** to compute `gradients`.\n\n```cpp\nauto x = Tensor::create(2.0, \"x\");\nauto w = Tensor::create(3.0, \"w\");\nauto y = x * w;\ny-\u003ebackward();\n\ncout \u003c\u003c \"Gradient of x: \" \u003c\u003c x-\u003egradient \u003c\u003c endl; // Outputs 3.0\ncout \u003c\u003c \"Gradient of w: \" \u003c\u003c w-\u003egradient \u003c\u003c endl; // Outputs 2.0\n```\n\n## II. [`layers.hpp`](./n2n_autograd/layers.hpp) | Neural Network Layers [🔝](#core-components)\n\n\u003e Defines classes for Neural Network layers and neurons, including parameter initialization.\n\n### 2.1. class `Initializers` [🔝](#core-components)\n\nWeight initialization is critical in Neural Network training. Proper initialization helps in preventing issues like **vanishing/exploding gradients**, ensuring that the network learns effectively from the beginning of training.\n- *static double* **random_uniform**(*double* `low`, *double* `high`): Generate a random number uniformly distributed between [`low`; `high`].\n- *static double* **he_uniform**(*int* `fan_in`, *int* `fan_out`): Initialize weights using the **He** initialization method, suitable for layers with `relu` activation functions. It calculates the limit using sqrt(6 / `fan_in`).\n- *static double* **glorot_uniform**(*int* `fan_in`, *int* `fan_out`): Initialize weights using the **Glorot (Xavier)** initialization method, suitable for layers with `sigmoid` or `tanh` activation functions. It calculates the limit using sqrt(6 / (`fan_in` + `fan_out`)).\n\n**he_uniform** and **glorot_uniform** are commonly used initialization methods in practice. They generate a random number between [`-limit`; `limit`] with `fan_in` as the number of input units and `fan_out` as the number of output units.\n\n### 2.2. class `Neuron` [🔝](#core-components)\n\nRepresent a single neuron within a layer with `weights` and `bias`. It also supports custom initialization and `activation` functions, allowing for flexibility in defining the neuron's behavior.\n\n- *function\u003cdouble()\u003e* `initializer`: Function used to initialize `weights` and `bias`.\n- *vector\u003c**`TensorPtr`**\u003e* `weights`: Weights associated with the neuron's inputs.\n- ***`TensorPtr`*** `bias`: Bias term for the neuron.\n- *string* `activation`: Activation function.\n- *string* `name`: Name identifier for the neuron.\n- *vector\u003c**`TensorPtr`**\u003e* `parameters`: Collection of the neuron's parameters (`weights` and `bias`).\n\n👉 **Constructor**:\n\n```cpp\nNeuron(int input_size, const string \u0026_activation = \"\", function\u003cdouble()\u003e init_func = nullptr, const string \u0026_name = \"Neuron\");\n```\n- `input_size`: Number of inputs to the neuron.\n- `_activation`: Activation function to apply (`sigmoid`, `tanh`, `relu`, `linear`, or `softmax`).\n- `init_func`: Function to initialize `weights` and `bias`.\n- `_name`: A name for the neuron.\n\nIt will create a `weight` and a `bias` **`Tensor`** for each input, initializing them using the [initializer](#21-class-initializers-). If no [initializer](#21-class-initializers-) is provided, a default uniform **random initializer** between **[-1; 1]** is used. These `weights` and `bias` are stored in the `parameters` *vector*.\n\n👉 ***`TensorPtr`*** **forward**(*const vector\u003c**`TensorPtr`**\u003e\u0026* `inputs`):\n\n- Compute the neuron's output given the **`TensorPtr`** inputs.\n- Calculate the weighted sum of `inputs` and adds the bias.\n- Apply the `activation` function if specified in the **Constructor**.\n\n👉 *vector\u003c**`TensorPtr`**\u003e\u0026* **get_parameters**(): Return references to the neuron's `parameters` (`weights` and `bias`).\n\n### 2.3. class `Dense` [🔝](#core-components)\n\nModel a fully connected (**`Dense`**) layer in a Neural Network. It manages a collection of `Neurons`, their `parameters`, and the **forward** pass computation. By specifying the `activation` function and [initializer](#21-class-initializers-), you can customize the behavior of the layer to match your network architecture.\n\n- *function\u003cdouble()\u003e* `initializer`: Function used to initialize `weights` and `biases` of the [neurons](#22-class-neuron-).\n- *vector\u003c[Neuron](#22-class-neuron-)\u003e* `neurons`: Collection of [neurons](#22-class-neuron-) within the layer.\n- *vector\u003c**`TensorPtr`**\u003e* `parameters`: Collection of the layer's `parameters`.\n- *int* `input_size`: Number of inputs to the layer.\n- *int* `output_size`: Number of outputs ([neurons](#22-class-neuron-)) in the layer.\n- *string* `activation`: Activation function used by the [neurons](#22-class-neuron-).\n- *string* `name`: Name identifier for the layer.\n\n👉 **Constructor**:\n\n```cpp\nDense(int _input_size, int _output_size, const string \u0026_activation = \"\", function\u003cdouble(int, int)\u003e init_func = nullptr, const string \u0026_name = \"Dense\");\n```\n\n- `input_size`: Number of input features to the layer.\n- `output_size`: Number of [neurons](#22-class-neuron-) (outputs) in the layer.\n- `_activation`: Activation function name for the [neurons](#22-class-neuron-) (`sigmoid`, `tanh`, `relu`, `linear`, or `softmax`).\n- `init_func`: Function to initialize `weights`.\n- `_name`: Name identifier for the layer.\n\nIt creates the specified number of [neurons](#22-class-neuron-), each initialized accordingly:\n- If an [initializer](#21-class-initializers-) is provided, use it to initialize the [neurons](#22-class-neuron-).\n- Collect all `parameters` from the [neurons](#22-class-neuron-) into the `parameters` *vector*.\n\n👉 *vector\u003c**`TensorPtr`**\u003e* **forward**(*const vector\u003c**`TensorPtr`**\u003e\u0026* `inputs`): \n\nCompute layer's outputs for all [neurons](#22-class-neuron-) in the layer given input **tensors**:\n- For each [neurons](#22-class-neuron-), calls its **forward** method with the `inputs`.\n- Collects the outputs from all [neurons](#22-class-neuron-) into a *vector*.\n- Handles special cases like `softmax`, where `activation` is applied across the entire layer.\n\n👉 *vector\u003c**`TensorPtr`**\u003e\u0026* **get_parameters**(): Return a reference to the layer's `parameters` for all [neurons](#22-class-neuron-) in the layer.\n\n👉 **Getter Methods**:\n\n- *const int\u0026* **get_input_size**(): Return the `input_size` of the layer.\n- *const int\u0026* **get_output_size**(): Return the `output_size` of the layer.\n- *const string\u0026* **get_name**(): Return the layer's `name`.\n- *const string\u0026* **get_activation**(): Return the `activation` function's name used by the layer.\n\n### 2.4. Usage Example [🔝](#core-components)\n\nCreate a **`Dense`** layer with 2 input features and 3 output neurons using the `relu` activation function. The `weights` and `bias` of the [neurons](#22-class-neuron-) are initialized using the **He initializer**.\n\n```cpp\nDense layer(2, 3, \"relu\", Initializers::he_uniform, \"HiddenLayer\");\nauto inputs = vector\u003cTensorPtr\u003e{Tensor::create(1.0), Tensor::create(2.0)};\nauto outputs = layer.forward(inputs);\n```\n\n### III. [`models.hpp`](./n2n_autograd/models.hpp) | Neural Network Model and Training Utilities [🔝](#core-components)\n\n\u003e Combine all components to define, train, and evaluate a Neural Network model. It handles data flow through the [layers](#ii-layershpp--neural-network-layers-), computation of [loss](#41-class-loss-) and [metrics](#51-class-metrics-), `parameter` updates with forward/backward passes, and provides utilities for monitoring training progress.\n\n![image](https://miro.medium.com/v2/resize:fit:1400/1*J-v2B6T9RKxdvwThtQ1NVg.png)\n\n### 3.1. class `Sequential`\u003c`OutputType`\u003e (like [TensorFlow](https://www.tensorflow.org)) [🔝](#core-components)\n\nA **templated** class representing a sequential Neural Network model composed of [layers](#ii-layershpp--neural-network-layers-), where the type of the model's output (`OutputType`) is either ***`TensorPtr`*** (for **regression** or **binary classification**) or *vector\u003c**`TensorPtr`**\u003e* (for **multi-class classification**).\n- *static constexpr int* `output_index`: Determines the index for accessing outputs based on `OutputType`.\n- *vector\u003c**`TensorPtr`**\u003e* `parameters`: Collection of all trainable `parameters` in the model.\n- *vector\u003c[Dense](#23-class-dense)\u003e* `layers`: [Layers](#ii-layershpp--neural-network-layers-) constituting the model.\n- *function\u003c**`TensorPtr`**(*const vector\u003c**`OutputType`**\u003e\u0026*, *const vector\u003c**`OutputType`**\u003e\u0026*)\u003e* `loss_func`: The [loss](#41-class-loss-) function used during training.\n- *unordered_map\u003cstring, function\u003cdouble(const **`YTruesVariant`**\u0026, const **`YPredsVariant`**\u0026)\u003e\u003e* `metric_funcs`: Map of [metric](#51-class-metrics-) functions for evaluation.\n- *unordered_map\u003cstring, vector\u003cany\u003e\u003e* `history`: Records of training history, including [loss](#41-class-loss-) and [metrics](#51-class-metrics-).\n\n👉 **Constructor**:\n\n```cpp\nSequential(const vector\u003cDense\u003e \u0026_layers, function\u003cTensorPtr(const vector\u003cOutputType\u003e \u0026, const vector\u003cOutputType\u003e \u0026)\u003e _loss_func, unordered_map\u003cstring, function\u003cdouble(const YTruesVariant \u0026, const YPredsVariant \u0026)\u003e\u003e _metric_funcs = {});\n```\n- `_layers`: A vector of [Dense](#23-class-dense) layers defining the model architecture.\n- `_loss_func`: [Loss](#41-class-loss-) function to use.\n- `metric_funcs`: An optional map of [metric](#51-class-metrics-) functions for evaluation.\n\nInitialize the sequential model with specified layers (collect all `parameters` from the provided layers), [loss](#41-class-loss-) function, and optional [metrics](#51-class-metrics-) associated with a training `history`.\n\n👉 *void* **train**(*const vector\u003cvector\u003cdouble\u003e\u003e\u0026* `X_train`, *const **`YTruesVariant`**\u0026* `y_train`, *const int\u0026* `epochs = 100`, *const variant\u003c[LearningRateScheduler](#61-class-learningratescheduler-)*, double\u003e\u0026* `learning_rate = 0.01`, *const int\u0026* `batch_size = 1`, *const double\u0026* `clip_value = 0.0`): \n- `X_train`: Training data features.\n- `y_train`: Training data labels.\n- `epochs`: Number of training epochs.\n- `learning_rate`: Learning rate or [scheduler](#61-class-learningratescheduler-) used for optimization.\n- `batch_size`: Number of samples per training batch.\n- `clip_value`: Gradient clipping threshold.\n\n| | |\n| --- | --- |\n| ![image](https://media.licdn.com/dms/image/D4D12AQElGrpg2NiisQ/article-cover_image-shrink_600_2000/0/1707688084849?e=2147483647\u0026v=beta\u0026t=iBiIxGUrle6a1mlTadU-0vWvyVjCxW7DBa5qXqK_Qa4) | ![image](https://raw.githubusercontent.com/greyhatguy007/Machine-Learning-Specialization-Coursera/1a6b6fc2851e6ab2d44f86b84db316a82a70e494/C1%20-%20Supervised%20Machine%20Learning%20-%20Regression%20and%20Classification/week1/Optional%20Labs/images/C1_W1_Lab03_lecture_slopes.PNG) |\n\n👉 *vector\u003c**`PredDataType`**\u003e* **predict**(*const vector\u003cvector\u003cdouble\u003e\u003e\u0026* `X`): Perform forward passes through the model for each input sample. Then, collect and return the predictions for the given input data.\n\n👉 *void* **summary**(): Print a summary of model architecture, including layers, output shapes, and parameter counts.\n\n👉 *vector\u003c**`TensorPtr`**\u003e\u0026* **get_parameters**(): Accesses all trainable `parameters` in the model.\n\n👉 *unordered_map\u003cstring, vector\u003cany\u003e\u003e\u0026* **get_history**(): Retrieve the training `history` including [loss](#41-class-loss-) and [metrics](#51-class-metrics-).\n\n### 3.2. Usage Example [🔝](#core-components)\n\nInstantiate a **`Sequential`** model with a desired architecture, [loss](#41-class-loss-) function, and [metrics](#51-class-metrics-). Train the model on the provided data and evaluate its performance.\n\n```cpp\nSequential\u003cvector\u003cTensorPtr\u003e\u003e model(\n  { // input_size, output_size, activation, initializer, name\n    Dense(input_size, 8, \"relu\", Initializers::he_uniform, \"Dense0\"), \n    Dense(8, 4, \"relu\", Initializers::he_uniform, \"Dense1\"),\n    Dense(4, num_classes, \"softmax\", Initializers::he_uniform, \"Dense2\")\n  },\n  Loss::categorical_crossentropy,   // Loss function for multi-class classification\n  {{\"accuracy\", Metrics::accuracy}} // Metric dictionary for evaluation\n);\nmodel.summary(); // Print model summary\nmodel.train(X_train, y_train, epochs, learning_rate, batch_size);\nauto predictions = model.predict(X_test);\n```\n\n## IV. [`losses.hpp`](./n2n_autograd/losses.hpp) | Loss Functions for guiding optimization process [🔝](#core-components)\n\n\u003e Define common loss functions used during the training of Neural Networks, essential for quantifying the difference or calculating the error between the model's predictions (`y_preds`) and actual values (`y_trues`).\n\n### 4.1. class `Loss` [🔝](#core-components)\n\n- *static **`TensorPtr`*** **mean_squared_error**(*const vector\u003c**`TensorPtr`**\u003e\u0026* `y_trues`, *const vector\u003c**`TensorPtr`**\u003e\u0026* `y_preds`): Compute the **Mean Squared Error (MSE)** between predicted and true values.\n    - `y_trues`: *vector* of true values as **`TensorPtr`**.\n    - `y_preds`: *vector* of predicted values as **`TensorPtr`**.\n- *static **`TensorPtr`*** **binary_crossentropy**(*const vector\u003c**`TensorPtr`**\u003e\u0026* `y_trues`, *const vector\u003c**`TensorPtr`**\u003e\u0026* `y_preds`): Compute the **Binary Cross-Entropy** loss for **binary classification** tasks with `sigmoid` activation.\n    - `y_trues`: *vector* of true binary labels as **`TensorPtr`**.\n    - `y_preds`: *vector* of predicted probabilities as **`TensorPtr`**.\n    - It uses the formula `-(y_true * log(y_pred) + (1 - y_true) * log(1 - y_pred))` and averages the loss over all samples.\n\n- *static **`TensorPtr`*** **categorical_crossentropy**(*const vector\u003cvector\u003c**`TensorPtr`**\u003e\u003e\u0026* `y_trues`, *const vector\u003cvector\u003c**`TensorPtr`**\u003e\u003e\u0026* `y_preds`): Computes the **Categorical Cross-Entropy** loss for **multi-class classification** tasks with one-hot encoded labels and `softmax` activation.\n    - y_trues: *vector* of *vectors* representing true one-hot encoded labels as **`TensorPtr`**.\n    - y_preds: *vector* of *vectors* representing predicted probabilities as **`TensorPtr`**.\n    - It uses the formula `-sum(y_true * log(y_pred))` and averages the loss over all samples.\n\n### 4.2. Usage Example [🔝](#core-components)\n\nChoose an appropriate loss function based on the problem type (**regression** or **binary/multi-class classification**).\n\n```cpp\nSequential\u003cvector\u003cTensorPtr\u003e\u003e model(layers, Loss::categorical_crossentropy);\nTensorPtr loss = Loss::mean_squared_error(y_true_tensors, y_pred_tensors);\nloss-\u003ebackward();\n```\n\n## V. [`metrics.hpp`](./n2n_autograd/metrics.hpp) | Metric Functions for Model Evaluation [🔝](#core-components)\n\n\u003e Provides functions to calculate evaluation metrics for model performance, such as accuracy.\n\n### 5.1. class `Metrics` [🔝](#core-components)\n\n- *double* **accuracy**(*const **`YTruesVariant`**\u0026* `y_trues`, *const **`YPredsVariant`**\u0026* `y_preds`)\n  - `y_trues`: `variant` type holding true labels, either as a *vector\u003cdouble\u003e* (for **binary classification**) or a *vector\u003cvector\u003cint\u003e\u003e* (*vector* of one-hot encoded *vectors* for **multi-class classification**).\n  - `y_preds`: `variant` type holding predicted labels in matching format.\n  - It calculates the proportion of correct predictions for both **binary** and **multi-class classification**:\n    - For **binary classification**, thresholds predicted probabilities at **0.5** to determine predicted classes.\n    - For **multi-class classification** with one-hot encoded labels, it compares the index of the maximum value in the predicted probabilities with the true label index.\n\n### 5.2. Usage Example [🔝](#core-components)\n\nEvaluate the model's performance using the `accuracy` metric.\n\n```cpp\ndouble acc = Metrics::accuracy(y_true, y_pred);\ncout \u003c\u003c \"Accuracy: \" \u003c\u003c acc \u003c\u003c endl;\n```\n\n## VI. [`optim.hpp`](./n2n_autograd/optim.hpp) | Optimizers and Learning Rate Schedulers [🔝](#core-components)\n\n\u003e Define components for optimizing the model's parameters during training\n\n### 6.1. class `LearningRateScheduler` [🔝](#core-components)\n\nLearning rate scheduling is an effective technique to improve training convergence and performance. This is an **abstract** base class for defining custom learning rate schedulers.\n- *double* `initial_learning_rate`: The starting learning rate.\n- *string* `name`: Name of the scheduler.\n\n👉 **Constructor**: `LearningRateScheduler`(*double* `initial_lr`, *string* `name`);\n\n👉 **Pure Virtual Method** to compute lr at a given training step: *virtual double* **operator**()(*int* `step`) = 0.\n\n### 6.2. class `WarmUpAndDecayScheduler` [🔝](#core-components)\n\nA concrete implementation of [LearningRateScheduler](#61-class-learningratescheduler-) that warms up the learning rate and then decays it exponentially.\n- *int* `warmup_steps`: Number of steps to warm up.\n- *int* `decay_steps`: Number of steps over which to decay the learning rate.\n- *double* `decay_rate`: Rate at which the learning rate decays.\n\n👉 **Constructor**:\n\n```cpp\nWarmUpAndDecayScheduler(double initial_lr, int warmup_steps, int decay_steps, double decay_rate, string name = \"WarmUpAndDecayScheduler\");\n```\n- `initial_lr`: Initial learning rate.\n- `warmup_steps`: Number of steps to linearly increase the learning rate.\n- `decay_steps`: Number of steps to decay the learning rate exponentially.\n- `decay_rate`: Rate at which the learning rate decays.\n- `name`: Name of the scheduler.\n\n👉 *double* **operator**()(*int* `step`): \n\nComputes the learning rate at a specific step.\n- If the current step is within the warm-up phase, increase the learning rate linearly.\n- After the warm-up, decay the learning rate exponentially based on the decay rate and number of decay steps.\n\n### 6.3. Usage Example [🔝](#core-components)\n\nUse a [scheduler](#61-class-learningratescheduler-) to adjust the learning rate during training dynamically.\n\n```cpp\nLearningRateScheduler *lr_scheduler = new WarmUpAndDecayScheduler(0.1, 5, 10, 0.9);\nmodel.train(X_train, y_train, epochs, lr_scheduler, batch_size);\ndelete lr_scheduler;\n```\n\n### VII. [`preprocess.hpp`](./n2n_autograd/preprocess.hpp) | Data Preprocessing Utilities [🔝](#core-components)\n\n\u003e Provides functions and classes for data loading and preprocessing.\n\n### 7.1. Data Loading [🔝](#core-components)\n\n👉 *pair\u003cvector\u003cvector\u003c*any*\u003e\u003e, vector\u003c*any*\u003e\u003e* **Xy_from_csv**(*const string\u0026* `file_path`, *int* `y_idx = -1`, *bool* `header = false`): \n- `file_path`: Path to the `CSV` file.\n- `y_idx`: Index of the target column (default is the last column). Negative values count from the end.\n- `header`: Indicate if the `CSV` has a header row.\n\nIt will read the `CSV` file line by line, parse each cell and convert them to `any` using [str_to_any](#81-any-conversion-). Then, it separates features and target variables, handling both numerical and categorical data (encode *string* class labels to *int* indices).\n\n### 7.2. Data Shuffling and Splitting [🔝](#core-components)\n\n👉 *pair\u003cvector\u003cvector\u003c*any*\u003e\u003e, vector\u003c*any*\u003e\u003e* **shuffle_data**(*const vector\u003cvector\u003cany\u003e\u003e\u0026* `X`, *const vector\u003cany\u003e\u0026* `y`): Generate a random permutation of indices, then reorder `X` and `y` according to the shuffled indices.\n\n👉 *tuple\u003cvector\u003cvector\u003c*any*\u003e\u003e, vector\u003cvector\u003c*any*\u003e\u003e, vector\u003c*any*\u003e, vector\u003c*any*\u003e\u003e* **train_test_split**(*const vector\u003cvector\u003c*any*\u003e\u003e\u0026* `X`, *const vector\u003c*any*\u003e\u0026* `y`, *float* `test_size = 0.2`):\n- Shuffle the data, then split `X` and `y` into training and testing sets based on the specified `test_size` as the proportion of the dataset to include in the test split.\n- **return**: `X_train`, `X_test`, `y_train`, `y_test`.\n\n### 7.3. class `StandardScaler` [🔝](#core-components)\n\nStandardizes input features by removing the mean and scaling to unit variance:\n- *vector\u003cdouble\u003e* `means`: Mean values of each feature calculated from the training data.\n- *vector\u003cdouble\u003e* `stds`: Standard deviations for each feature.\n\n👉 *vector\u003cvector\u003cdouble\u003e\u003e* **fit_transform**(*const vector\u003cvector\u003cany\u003e\u003e\u0026* `X`): Computes the `means` and `stds` for each feature, and scales the data.\n\n👉 *vector\u003cvector\u003cdouble\u003e\u003e* **transform**(*const vector\u003cvector\u003cany\u003e\u003e\u0026* `X`): Scales new data using previously computed `means` and `stds` from the training data.\n\n### 7.4. Usage Example [🔝](#core-components)\n\nPrepare data before training to improve model performance.\n\n```cpp\nauto [X_raw, y_raw] = Xy_from_csv(\"data.csv\");\nauto [X_train, X_test, y_train, y_test] = train_test_split(X_raw, y_raw, 0.2);\n\nStandardScaler scaler;\nauto X_train_scaled = scaler.fit_transform(X_train);\nauto X_test_scaled = scaler.transform(X_test);\n```\n\n## VIII. [`converters.hpp`](./n2n_autograd/converters.hpp) | Data Conversion Utilities [🔝](#core-components)\n\n\u003e Contain utility functions to facilitate the conversion of different data types commonly used in preprocessing, particularly when preparing data for training as well as inference by converting raw data into tensors suitable for model input.\n\n### 8.1. *any* Conversion [🔝](#core-components)\n\n- *any* **str_to_any**(*const string\u0026* `str`): Converts a string to an *any* type, attempting to parse it as an *int*, *double*, or leaving it as a string based on its content.\n- *double* **any_to_double**(*const any\u0026* `input`): Safely converts an *any* type (expected to hold an *int* or *double*) to a *double*, supporting both *int* and *double* internally.\n- *vector\u003c*int*\u003e* **anys_to_ints**(*const vector\u003cany\u003e\u0026* `inputs`): Converts a *vector of any* types (each expected to hold an *int*) to a *vector of ints*.\n- *vector\u003c*double*\u003e* **anys_to_doubles**(*const vector\u003cany\u003e\u0026* `inputs`): Converts a *vector of any* types (each expected to hold an *int* or *double*) to a *vector of doubles*.\n\n### 8.2. One-Hot Encoding [🔝](#core-components)\n\n- *vector\u003cvector\u003c*int*\u003e\u003e* **anys_to_1hots**(*const vector\u003cany\u003e\u0026* `y_raw`, *int* `num_classes`): Converts a *vector* of class labels to one-hot encoded *vectors*. It will create a *vector of vectors* (`n_samples` x `num_classes`), initialize all elements to `0`, and set the index corresponding to each class label to `1` in the one-hot *vector*.\n- *vector\u003cvector\u003c**`TensorPtr`**\u003e\u003e* **anys_to_1hot_tensors**(*const vector\u003c*any*\u003e\u0026* `y_raw`, *int* `num_classes`): Similar to above but return a *vector* of *vectors* (instead of *int*) containing **`TensorPtr`** representing one-hot encodings.\n\n### 8.3. `Tensor` Conversion [🔝](#core-components)\n\n- *vector\u003c**`TensorPtr`**\u003e* **doubles_to_1d_tensors**(*const vector\u003c*double*\u003e\u0026* `data`): Converts a *vector* of *doubles* to a *vector* of **1D** [Tensor](#i-tensorhpp--tensor-class-and-auto-differentiation-engine-) pointers by iterating over the data and creating a **`TensorPtr`** for each value.\n- *vector\u003c**`TensorPtr`**\u003e* **doubles_to_2d_tensors**(*const vector\u003cvector\u003c*double*\u003e\u003e\u0026* `data`): Converts a **2D** *vector* of doubles to a **2D** *vector* of **`TensorPtr`**.\n\n## Potential Improvements\n\n- [ ] **Extend Tensor Support**: Implement support for multi-dimensional [Tensor](#i-tensorhpp--tensor-class-and-auto-differentiation-engine-) ([Tensor](#i-tensorhpp--tensor-class-and-auto-differentiation-engine-) with \u003e 1 dimension).\n- [ ] **Additional Layers**: Add more types of layers such as convolutional layers and recurrent layers.\n- [ ] **Optimizers**: Implement more sophisticated optimization algorithms like Adam or RMSProp.\n- [ ] **Concurrency**: The code currently runs on a single thread. Multi-threading or GPU acceleration can be explored for more computational efficiency or performance improvements on large datasets.\n- [ ] **Model Serialization**: Add functionality to save and load trained models.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F18520339%2Fnn-auto-differentiation-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F18520339%2Fnn-auto-differentiation-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F18520339%2Fnn-auto-differentiation-engine/lists"}