{"id":16572574,"url":"https://github.com/jamesyang007/fastad","last_synced_at":"2025-07-13T07:33:30.333Z","repository":{"id":45229569,"uuid":"181966964","full_name":"JamesYang007/FastAD","owner":"JamesYang007","description":"FastAD is a C++ implementation of automatic differentiation both forward and reverse mode.","archived":false,"fork":false,"pushed_at":"2023-09-27T18:46:10.000Z","size":1186,"stargazers_count":108,"open_issues_count":10,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-02T00:41:41.628Z","etag":null,"topics":["auto-differentiation","autodiff","autodifferentiation","automatic-differentiation","cpp17","derivatives","differentiation","linux","macos","math"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JamesYang007.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-17T20:56:03.000Z","updated_at":"2025-02-12T22:15:48.000Z","dependencies_parsed_at":"2024-10-27T11:19:21.571Z","dependency_job_id":"2ae6518b-de06-4d2b-9a4f-451cac2bcad5","html_url":"https://github.com/JamesYang007/FastAD","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/JamesYang007/FastAD","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JamesYang007%2FFastAD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JamesYang007%2FFastAD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JamesYang007%2FFastAD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JamesYang007%2FFastAD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JamesYang007","download_url":"https://codeload.github.com/JamesYang007/FastAD/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JamesYang007%2FFastAD/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265105737,"owners_count":23712194,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["auto-differentiation","autodiff","autodifferentiation","automatic-differentiation","cpp17","derivatives","differentiation","linux","macos","math"],"created_at":"2024-10-11T21:28:00.115Z","updated_at":"2025-07-13T07:33:30.295Z","avatar_url":"https://github.com/JamesYang007.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FastAD \r\n\r\n[![Build Status](https://travis-ci.org/JamesYang007/FastAD.svg?branch=master)](https://travis-ci.org/JamesYang007/FastAD) \r\n[![CircleCI](https://circleci.com/gh/JamesYang007/FastAD/tree/master.svg?style=svg)](https://circleci.com/gh/JamesYang007/FastAD/tree/master)\r\n[![Coverage Status](https://coveralls.io/repos/github/JamesYang007/FastAD/badge.svg?branch=master\u0026service=github)](https://coveralls.io/github/JamesYang007/FastAD?branch=master)\r\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/5fe0893b770643e7bd9d4c9ad6ab189b)](https://www.codacy.com/manual/JamesYang007/FastAD?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=JamesYang007/FastAD\u0026amp;utm_campaign=Badge_Grade)\r\n[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/3532/badge)](https://bestpractices.coreinfrastructure.org/projects/3532)\r\n[![GitHub tag (latest by date)](https://img.shields.io/github/v/tag/JamesYang007/FastAD)](https://github.com/JamesYang007/FastAD/tags)\r\n[![GitHub release (latest by date)](https://img.shields.io/github/v/release/JamesYang007/FastAD?include_prereleases)](https://github.com/JamesYang007/FastAD/releases)\r\n[![GitHub issue (latest by date)](https://img.shields.io/github/issues/JamesYang007/FastAD)](https://github.com/JamesYang007/FastAD/issues)\r\n[![License](https://img.shields.io/github/license/JamesYang007/FastAD?color=blue)](http://badges.mit-license.org)\r\n\r\n## Table of contents\r\n\r\n- [FastAD](#fastad)\r\n  - [Table of contents](#table-of-contents)\r\n  - [Overview](#overview)\r\n    - [Intuitive syntax](#intuitive-syntax)\r\n    - [Robustness](#robustness)\r\n    - [Memory Efficiency](#memory-efficiency)\r\n    - [Speed](#speed)\r\n  - [Installation](#installation)\r\n    - [General Users](#general-users)\r\n    - [Developers](#developers)\r\n  - [Integration](#integration)\r\n    - [CMake](#cmake)\r\n    - [Others](#others)\r\n  - [User Guide](#user-guide)\r\n    - [Forward Mode](#forward-mode)\r\n    - [Reverse Mode](#reverse-mode)\r\n      - [Basic Usage](#basic-usage)\r\n      - [Placeholder](#placeholder)\r\n      - [Advanced Usage](#advanced-usage)\r\n  - [Applications](#applications)\r\n    - [Black-Scholes Put-Call Option Pricing](#black-scholes-put-call-option-pricing)\r\n    - [Quadratic Expression Differential](#quadratic-expression-differential)\r\n    - [Simple Linear Regression Model](#simple-linear-regression-model)\r\n    - [3-layer Neural Network with Jump Connection](#3-layer-neural-network-with-jump-connection)\r\n  - [Quick Reference](#quick-reference)\r\n    - [Forward](#forward)\r\n    - [Reverse](#reverse)\r\n  - [Contact](#contact)\r\n  - [Contributors](#contributors)\r\n  - [Third Party Tools](#third-party-tools)\r\n  - [License](#license)\r\n\r\n## Overview\r\n\r\nFastAD is a header-only C++ template library for automatic differentiation supporting both forward and reverse mode. \r\nIt utilizes the latest features in C++17 and expression templates for efficient computation.\r\nFastAD is unique for the following:\r\n\r\n### Intuitive syntax\r\n\r\nSyntax choice is very important for C++ developers. \r\nOur philosophy is that syntax should be as similar as possible to mathematical notation.\r\nThis makes FastAD easy to use and allow users to write readable, intuitive, and simple code.\r\nSee [User Guide](#user-guide) for more details.\r\n\r\n### Robustness\r\n\r\nFastAD has been heavily unit-tested with high test coverage followed by a few integration tests.\r\nA variety of functions have been tested against analytical solutions;\r\nat machine-level precision, the derivatives coincide.\r\n\r\n### Memory Efficiency\r\n\r\nFastAD is written to be incredibly efficient with memory usage and cache hits.\r\nThe main overhead of most AD libraries is the tape, which stores adjoints.\r\nUsing expression template techniques, and smarter memory management,\r\nwe can significantly reduce this overhead.\r\n\r\n### Speed\r\n\r\nSpeed is the utmost critical aspect of any AD library.\r\nFastAD has been proven to be extremely fast, which inspired the name of this library.\r\nBenchmark shows over orders of magnitude improvement from existing libraries \r\nsuch as Adept, Stan Math Library, ADOL-C, CppAD, and Sacado\r\n(see our separate benchmark repositor [ADBenchmark](https://github.com/JamesYang007/ADBenchmark)).\r\nMoreover, it also shows 10x improvement from the naive (and often inaccurate) finite-difference method.\r\n\r\n## Installation\r\n\r\nFirst, clone the repo:\r\n```\r\ngit clone https://github.com/JamesYang007/FastAD.git ~/FastAD\r\n```\r\n\r\nFrom here on, we will refer to the cloned directory as `workspace_dir`\r\n(in the example above, `workspace_dir` is `~/FastAD`).\r\n\r\nThe library has the following dependencies:\r\n- Eigen3.3\r\n- GoogleTest (dev only)\r\n- Google Benchmark (dev only)\r\n\r\n### General Users\r\n\r\nIf the user already has Eigen3.3 installed in their system, they can omit the following step.\r\nFor general users, if they wish to install Eigen locally, they can run \r\n```bash\r\n./setup.sh\r\n``` \r\nfrom `workspace_dir`. \r\nThis will install Eigen3.3 into `workspace_dir/libs/eigen-3.3.7/build`.\r\n\r\nFor those who want to install `FastAD` globally into the system, simply run:\r\n```bash\r\n./install.sh\r\n```\r\nThis will build and install the header files into the system.\r\n\r\nFor users who want to install `FastAD` locally, run the following from `workspace_dir`:\r\n```bash\r\nmkdir -p build \u0026\u0026 cd build\r\ncmake -DCMAKE_INSTALL_PREFIX=. ..\r\nmake install\r\n```\r\nOne can set the `CMAKE_INSTALL_PREFIX` to anything.\r\nThis example will install the library in `workspace_dir/build`.  \r\n\r\nFor users that want to integrate `FastAD` in their own CMakeLists, use FetchContent (CMake \u003e= 3.11)\r\n```CMake\r\ninclude(FetchContent)\r\nFetchContent_Declare(\r\n        FastAD\r\n        GIT_REPOSITORY https://github.com/JamesYang007/FastAD\r\n        GIT_TAG v3.2.1\r\n        GIT_SHALLOW TRUE\r\n        GIT_PROGRESS TRUE)\r\nFetchContent_MakeAvailable(FastAD)\r\n# Further link target 'FastAD'\r\n```\r\n### Developers\r\n\r\nRun the following to install all of the dependencies locally:\r\n```bash\r\n./setup.sh dev\r\n```\r\n\r\nTo build the library, run the following:\r\n```bash\r\n./clean-build.sh \u003cdebug/release\u003e [other CMake flags...]\r\n```\r\nHere are the following options one can specify as a CMake flag `-D...=ON`\r\n(replace `...` with any of the following):\r\n- FASTAD_ENABLE_TEST        (builds tests)\r\n- FASTAD_ENABLE_BENCHMARK   (builds benchmarks)\r\n- FASTAD_ENABLE_EXAMPLE     (builds examples)\r\n\r\nBy default, the flags are `OFF`.\r\nNote that this only builds and does not install the library.\r\n\r\nTo run tests, execute the following:\r\n```bash\r\ncd build/\u003cdebug/release\u003e\r\nctest -j6\r\n```\r\n\r\nTo run benchmarks, change directory to \r\n`build/\u003cdebug/release\u003e/benchmark` and run any one of the executables.\r\n\r\n## Integration\r\n\r\n### CMake\r\n\r\nIf your project is built using CMake, add the following to CMakeLists.txt in the root directory:\r\n```cmake\r\nfind_package(FastAD CONFIG REQUIRED)\r\n```\r\n\r\nIf you installed the library locally, say `path_to_install`, then add the following:\r\n```cmake\r\nfind_package(FastAD CONFIG REQUIRED HINTS path_to_install/share)\r\n```\r\n\r\nFor any program that requires `FastAD`, \r\nuse `target_link_libraries` to link with `FastAD::FastAD`.\r\n\r\nAn example project that uses FastAD as a dependency may have a CMakeLists.txt that looks like this:\r\n```cmake\r\nproject(\"MyProject\")\r\nfind_package(FastAD CONFIG REQUIRED)\r\nadd_executable(main src/main.cpp)\r\ntarget_link_libraries(main FastAD::FastAD)\r\n```\r\n\r\n### Others\r\n\r\nSimply add the following flag when compiling your program:\r\n```\r\n-Ipath_to_install/include\r\n```\r\n\r\nAn example build command would be:\r\n```\r\ng++ main.cpp -Ipath_to_install/include\r\n```\r\n\r\nIf Eigen3.3 was installed locally, you must provide its path as well.\r\n\r\n## User Guide\r\n\r\nThe only header the user needs to include is `fastad`.\r\n\r\n### Forward Mode\r\n\r\nForward mode is extremely simple to use.\r\n\r\nThe only class a user will need to deal with is `ForwardVar\u003cT\u003e`,\r\nwhere `T` is the underlying data type (usually `double`).\r\nThe API only exposes getters and setters:\r\n```cpp\r\nForwardVar\u003cdouble\u003e v;       // initialize value and adjoint to 0\r\nv.set_value(1.);            // value is now 1.\r\ndouble r = v.get_value();   // r is now 1.\r\nv.set_adjoint(1.);          // adjoint is now 1.\r\ndouble s = v.get_adjoint(); // s is now 1.\r\n```\r\n\r\nThe rest of the work has already been done by the library\r\nwith operator overloading.\r\n\r\nHere is an example program that differentiates a complicated function:\r\n```cpp\r\n#include \u003cfastad\u003e\r\n#include \u003ciostream\u003e\r\n\r\nint main()\r\n{\r\n    using namespace ad;\r\n\r\n    ForwardVar\u003cdouble\u003e w1(0.), w2(1.);\r\n    w1.set_adjoint(1.); // differentiate w.r.t. w1\r\n    ForwardVar\u003cdouble\u003e w3 = w1 * sin(w2);\r\n    ForwardVar\u003cdouble\u003e w4 = w3 + w1 * w2;\r\n    ForwardVar\u003cdouble\u003e w5 = exp(w4 * w3);\r\n\r\n    std::cout \u003c\u003c \"f(x, y) = exp((x * sin(y) + x * y) * x * sin(y))\\n\"\r\n              \u003c\u003c \"df/dx = \" \u003c\u003c w5.get_adjoint() \u003c\u003c std::endl;\r\n    return 0;\r\n}\r\n```\r\n\r\nWe initialize `w1` and `w2` with values `0.` and `1.`, respectively.\r\nWe set the adjoint of `w1` to `1.` to indicate that we are differentiating w.r.t. `w1`.\r\nBy default, all adjoints of `ForwardVar` are set to `0.`.\r\nThis indicates that we will be differentiating in the direction of `(1.,0.)`, i.e. partial derivative w.r.t. `w1`.\r\nNote that user could also set the adjoint for `w2`, \r\n__but this will compute the directional derivative multiplied by the norm of (w1, w2)__.\r\nAfter computing the desired expression, we get the directional derivative \r\nby calling `get_adjoint()` on the final `ForwardVar` object.\r\n\r\n### Reverse Mode\r\n\r\n#### Basic Usage\r\n\r\nThe most basic usage simply requires users to create `Var\u003cT, ShapeType\u003e` objects.\r\n`T` denotes the underlying value type (usually `double`).\r\n`ShapeType` denotes the general shape of the variable.\r\nIt must be one of `ad::scl, ad::vec, ad::mat` corresponding to\r\nscalar, (column) vector, and matrix, respectively.\r\n\r\n```cpp\r\nVar\u003cdouble, scl\u003e x;\r\nVar\u003cdouble, vec\u003e v(5);    // set size to 5\r\nVar\u003cdouble, mat\u003e m(2, 3); // set shape to 2x3\r\n```\r\n\r\nFrom here, one can create complicated expressions \r\nby invoking a wide range of functions \r\n(see [Quick Reference](#quick-reference) for a full list of expression builders).\r\n\r\nAs an example, here is an expression to differentiate `sin(x) + cos(v)`:\r\n```cpp\r\nauto expr = (sin(x) + cos(v));\r\n```\r\nNote that this represents a vector expression, since `sin(x)` is a scalar expression\r\nbut `cos(v)` is a vectorized function on a vector, which is again a vector expression.\r\n\r\nBefore we differentiate, the expression is required to\r\n\"bind\" to a storage for the values and adjoints of intermediate expression nodes.\r\nThe reason for this design is for speed purposes and cache hits.\r\nIf the user wishes to manage this storage, they can do this:\r\n```cpp\r\nauto size_pack = expr.bind_cache_size();\r\nstd::vector\u003cdouble\u003e val_buf(size_pack(0));\r\nstd::vector\u003cdouble\u003e adj_buf(size_pack(1));\r\nexpr.bind_cache({val_buf.data(), adj_buf.data()});\r\n```\r\n\r\nThe `bind_cache_size()` will return exactly how many doubles are needed\r\nfor values and adjoints, respectively, of type `util::SizePack`, which is an alias for `Eigen::Array\u003csize_t, 2, 1\u003e`.\r\nand `bind(util::PtrPack\u003cdouble\u003e)` will bind itself to that region of memory.\r\nIt is encouraged to create the pointer pack object using initializer list as shown above.\r\nThis pattern occurs so often that if the user does not care about managing this,\r\nthey should use the following helper function:\r\n```cpp\r\nauto expr_bound = ad::bind(sin(x) + cos(v));\r\n```\r\n\r\n`ad::bind` will return a wrapper class that wraps the expression\r\nand at construction binds it to a privately owned storage \r\nin the same way described above.\r\n\r\n_If the expression is not bound to any storage, it will lead to segfault_!\r\n\r\nTo differentiate the expression, simply call the following:\r\n```cpp\r\nauto f = ad::autodiff(expr_bound, seed);\r\n\r\n// or if the raw expression is manually bound,\r\nauto f = ad::autodiff(expr, seed);\r\n```\r\nwhere `seed` is the initial adjoint for the root of the expression.\r\nIf the expression is scalar, seed is a literal (`double`)\r\nand the default value is `1`, so the user does not have to input anything.\r\nIf the expression is multi-dimensional, \r\nseed does not have a default value,\r\nmust be of type `Eigen::Array`,\r\nand must have the same dimensions as the expression.\r\n`autodiff` will return the evaluated function value.\r\nThis return value is `T` if it is a scalar expression, and otherwise,\r\n`Eigen::Map\u003cEigen::Matrix\u003cT, Eigen::Dynamic, ...\u003e\u003e` where `...` depends on\r\nthe shape of the expression (`1` if vector, `Eigen::Dynamic` if matrix).\r\n\r\nYou can retrieve the adjoints by calling `get_adj(i,j)` or \r\n`get_adj()` (with no arguments) from `x, v` like so:\r\n```cpp\r\nx.get_adj(0,0); // (1) get adjoint for x \r\nv.get_adj(2,0); // (2) get adjoint for v at index 2\r\nx.get_adj();    // (3) get full adjoint (same as (1))\r\nv.get_adj();    // (4) get full adjoint\r\n```\r\n\r\nThe full code for this example is the following:\r\n```cpp\r\n#include \u003cfastad\u003e\r\n#include \u003ciostream\u003e\r\n\r\nint main()\r\n{\r\n    using namespace ad;\r\n\r\n    Var\u003cdouble, scl\u003e x(2);\r\n    Var\u003cdouble, vec\u003e v(5);\r\n    \r\n    // randomly generate values for v\r\n    v.get().setRandom();\r\n\r\n    // create AD expression bound to storage\r\n    auto expr_bound = bind(sin(x) + cos(v));\r\n    \r\n    // seed to get gradient of function at index 2\r\n    Eigen::Array\u003cdouble, Eigen::Dynamic, 1\u003e seed(v.size());\r\n    seed.setZero();\r\n    seed[2] = 1;\r\n\r\n    // differentiate\r\n    auto f = autodiff(expr_bound, seed);\r\n\r\n    std::cout \u003c\u003c x.get_adj() \u003c\u003c std::endl;\r\n    std::cout \u003c\u003c v.get_adj(2,0) \u003c\u003c std::endl;\r\n\r\n    return 0;\r\n}\r\n```\r\n\r\n_Note: once you have differentiated an expression, \r\nyou must reset the adjoints of all variables to 0 before differentiating again.\r\nThis includes placeholder variables (see below)._\r\nTo that end, we provide a member function for `Var` called `reset_adj()`.\r\n\r\nHere is a more complicated example:\r\n\r\n```cpp\r\n#include \u003cfastad\u003e\r\n\r\nint main()\r\n{\r\n    Var\u003cdouble, vec\u003e v1(6);\r\n    Var\u003cdouble, vec\u003e v2(5);\r\n    Var\u003cdouble, mat\u003e M(5, 6);\r\n    Var\u003cdouble, vec\u003e w(5);\r\n    Var\u003cdouble, scl\u003e r;\r\n\r\n    auto\u0026 v1_raw = v1.get();  // Eigen::Map\r\n    auto\u0026 v2_raw = v2.get();  // Eigen::Map\r\n    auto\u0026 M_raw = M.get();  // Eigen::Map\r\n\r\n    // initialize...\r\n\r\n    auto expr = bind((\r\n        w = ad::dot(M, v1) + v2,\r\n        r = sum(w) * sum(w * v2)\r\n    ));\r\n\r\n    autodiff(expr);\r\n\r\n    std::cout \u003c\u003c v1.get_adj(0,0) \u003c\u003c std::endl;  // adjoint of v1 at index 0\r\n    std::cout \u003c\u003c v2.get_adj(1,0) \u003c\u003c std::endl;  // adjoint of v2 at index 1\r\n    std::cout \u003c\u003c M.get_adj(1,2) \u003c\u003c std::endl;   // adjoint of M at index (1,2)\r\n\r\n    return 0;\r\n}\r\n```\r\n\r\n#### Placeholder\r\n\r\nIn the previous example, we used a placeholder expression,\r\nwhich is of the form `v = expr`.\r\nYou can use placeholders to greatly speed up the performance\r\nand also save a lot of memory.\r\n\r\nConsider the following expression:\r\n```cpp\r\nauto expr = (sin(x) + cos(v) + sum(cos(v)));\r\n```\r\nWhen there are common expressions (like `cos(v)`),\r\nthey will be evaluated multiple times unnecessarily.\r\n\r\nPlaceholder expressions are created by using `operator=` with\r\na `Var` and an expression:\r\n```cpp\r\nVar\u003cdouble, scl\u003e x;\r\nVar\u003cdouble, vec\u003e v(5);\r\nVar\u003cdouble, vec\u003e w(v.size());\r\nauto expr = (\r\n    w = cos(v),\r\n    sin(x) + w + sum(w)\r\n);\r\n```\r\nThis will only evaluate `cos(v)` once, and reuse the results\r\nfor the subsequent expressions by using `w`.\r\n\r\nWhile this is not specific to placeholder expressions,\r\n`operator,` is usually invoked to \"glue\" many placeholder expressions.\r\nHowever, one can certainly glue any kinds of expressions, if they wish.\r\n\r\n#### Advanced Usage\r\n\r\nFor advanced users who need to get more low-level control over \r\nthe memory for values and adjoints for all variables, they can use\r\n`VarView\u003cT, ShapeType\u003e`.\r\nAll of the discussion above holds for `VarView` objects.\r\nIn fact, when we build an expression out of `Var` of `VarView`,\r\nwe convert all of them to `VarView`s so that the expression is solely a viewer.\r\n\r\n`VarView` objects __do not__ own the values and adjoints, but views them.\r\nHere is an example program that binds the viewers to a contiguous chunk of memory:\r\n```cpp\r\nVarView\u003cdouble, scl\u003e x;\r\nVarView\u003cdouble, vec\u003e v(3);\r\nVarView\u003cdouble, vec\u003e w(3);\r\n\r\nstd::vector\u003cdouble\u003e vals(x.size() + v.size());\r\nstd::vector\u003cdouble\u003e adjs(x.size() + v.size());\r\nstd::vector\u003cdouble\u003e w_vals(w.size());\r\nstd::vector\u003cdouble\u003e w_adjs(w.size());\r\n\r\n// x binds to the first element of storages\r\ndouble* val_next = x.bind(vals.data());\r\ndouble* adj_next = x.bind_adj(adjs.data());\r\n\r\n// v binds starting from 2nd element of storages\r\nv.bind(val_next);\r\nv.bind_adj(adj_next);\r\n\r\n// bind placeholders to a separate storage region\r\nw.bind(w_vals.data());\r\nw.bind_adj(w_adjs.data());\r\n\r\nauto expr = (\r\n    w = cos(v),\r\n    sin(x) + w + sum(w)\r\n);\r\n```\r\n\r\n## Applications\r\n\r\n### Black-Scholes Put-Call Option Pricing\r\n\r\nThe following is an example of computing deltas in Black-Scholes model using FastAD.\r\nThis example was taken from the autodiff library in \r\n[boost](https://www.boost.org/doc/libs/master/libs/math/doc/html/math_toolkit/autodiff.html#math_toolkit.autodiff.example-black_scholes).\r\n\r\n```cpp\r\n#include \u003cfastad\u003e\r\n#include \u003ciostream\u003e\r\n\r\nenum class option_type {\r\n    call, put\r\n};\r\n\r\n// Standard Normal CDF\r\ntemplate \u003cclass T\u003e\r\ninline auto Phi(const T\u0026 x)\r\n{\r\n    return 0.5 * (ad::erf(x / std::sqrt(2.)) + 1.);\r\n}\r\n\r\n// Generates expression that computes Black-Scholes option price\r\ntemplate \u003coption_type cp, class Price, class Cache\u003e\r\nauto black_scholes_option_price(const Price\u0026 S,\r\n                                double K,\r\n                                double sigma,\r\n                                double tau,\r\n                                double r,\r\n                                Cache\u0026 cache)\r\n{\r\n    cache.resize(3);\r\n    double PV = K * std::exp(-r * tau);\r\n    auto common_expr = (\r\n            cache[0] = ad::log(S / K),\r\n            cache[1] = (cache[0] + ((r + sigma * sigma / 2.) * tau)) / \r\n                            (sigma * std::sqrt(tau)),\r\n            cache[2] = cache[1] - (sigma * std::sqrt(tau))\r\n    );\r\n    if constexpr (cp == option_type::call) {\r\n        return (common_expr,\r\n                Phi(cache[1]) * S - Phi(cache[2]) * PV);\r\n    } else {\r\n        return (common_expr,\r\n                Phi(-cache[2]) * PV - Phi(-cache[1]) * S);\r\n    }\r\n}\r\n\r\nint main()\r\n{\r\n    double K = 100.0;        \r\n    double sigma = 5;        \r\n    double tau = 30.0 / 365; \r\n    double r = 1.25 / 100;   \r\n    ad::Var\u003cdouble\u003e S(105);  \r\n    std::vector\u003cad::Var\u003cdouble\u003e\u003e cache;\r\n\r\n    auto call_expr = ad::bind(\r\n            black_scholes_option_price\u003coption_type::call\u003e(\r\n                S, K, sigma, tau, r, cache));\r\n\r\n    double call_price = ad::autodiff(call_expr);\r\n\r\n    std::cout \u003c\u003c call_price \u003c\u003c std::endl;\r\n    std::cout \u003c\u003c S.get_adj() \u003c\u003c std::endl;\r\n\r\n    // reset adjoints before differentiating again\r\n    S.reset_adj();\r\n    for (auto\u0026 c : cache) c.reset_adj();\r\n\r\n    auto put_expr = ad::bind(\r\n            black_scholes_option_price\u003coption_type::put\u003e(\r\n                S, K, sigma, tau, r, cache));\r\n\r\n    double put_price = ad::autodiff(put_expr);\r\n\r\n    std::cout \u003c\u003c put_price \u003c\u003c std::endl;\r\n    std::cout \u003c\u003c S.get_adj() \u003c\u003c std::endl;\r\n\r\n    return 0;\r\n}\r\n```\r\n\r\nWe observed the same output as the one shown in \r\n[boost](https://www.boost.org/doc/libs/master/libs/math/doc/html/math_toolkit/autodiff.html#math_toolkit.autodiff.example-black_scholes)\r\nfor the prices and deltas (S's adjoints).\r\n\r\n### Quadratic Expression Differential\r\n\r\nIn ML applications, user usually provide a full parameter vector to an optimizer and write an objective function to calculate objective value and gradient. The gradient pointer is provided by the optimizer and user fill values. Then it will be more convinent to use `VarView` to bind the gradient pointer as buffer.\r\n\r\nHere is an example of differentiating a quadratic expression `x^T*Sigma*x` using `VarView`.\r\n\r\n```cpp\r\n#include \u003ciostream\u003e\r\n#include \"fastad\"\r\n#include \u003cEigen/src/Core/Matrix.h\u003e\r\n\r\nint main() {\r\n    using namespace ad;\r\n\r\n    // Generating buffer.\r\n    Eigen::MatrixXd x_data(2, 1);\r\n    x_data \u003c\u003c 0.5, 0.6;\r\n    Eigen::MatrixXd x_adj(2, 1);\r\n    x_adj.setZero(); // Set adjoints to zeros.\r\n\r\n    // Initialize variable.\r\n    VarView\u003cdouble, mat\u003e x(x_data.data(), x_adj.data(), 2, 1);\r\n\r\n    // Initialize matrix.\r\n    Eigen::MatrixXd _Sigma(2, 2);\r\n    _Sigma \u003c\u003c 2, 3, 3, 6;\r\n    std::cout \u003c\u003c _Sigma \u003c\u003c std::endl;\r\n    auto Sigma = constant(_Sigma);\r\n\r\n    // Quadratic expression: x^T*Sigma*x\r\n    auto expr = bind(dot(dot(transpose(x), Sigma), x));\r\n    // Seed\r\n    Eigen::MatrixXd seed(1, 1);\r\n    seed.setOnes(); // Usually seed is 1. DONT'T FORGET!\r\n    // Auto differential.\r\n    auto f = autodiff(expr, seed.array());\r\n\r\n    // Print results.\r\n    std::cout \u003c\u003c \"f: \" \u003c\u003c f \u003c\u003c std::endl;\r\n    std::cout \u003c\u003c x.get() \u003c\u003c std::endl;     //[0.5, 0.6]\r\n    std::cout \u003c\u003c x.get_adj() \u003c\u003c std::endl; //[5.6, 10.2]\r\n\r\n    return 0;\r\n}\r\n```\r\n\r\n### Simple Linear Regression Model\r\nIn a regression model, one has many rows of data. A loop is needed to calculate loss of each row.\r\n\r\n```cpp\r\n#include \"fastad\"\r\n#include \u003ciostream\u003e\r\n\r\nint main() {\r\n    using namespace ad;\r\n    // Create data matrix.\r\n    Eigen::MatrixXd X(5, 2);\r\n    X \u003c\u003c 1, 10, 2, 20, 3, 30, 4, 40, 5, 50;\r\n    Eigen::VectorXd y(5);\r\n    y \u003c\u003c 32, 64, 96, 128, 160; // y=2*x1+3*x2\r\n\r\n    // Generating buffer.\r\n    Eigen::MatrixXd theta_data(2, 1);\r\n    theta_data \u003c\u003c 1, 2;\r\n    Eigen::MatrixXd theta_adj(2, 1);\r\n    theta_adj.setZero(); // Set adjoints to zeros.\r\n\r\n    // Initialize variable.\r\n    VarView\u003cdouble, mat\u003e theta(theta_data.data(), theta_adj.data(), 2, 1);\r\n\r\n    // Create expr. Use a row buffer to store data. Then we only need to manipulate data when\r\n    // looping.\r\n    Eigen::MatrixXd x_row_buffer = X.row(0);\r\n    auto xi = constant_view(x_row_buffer.data(), 1, X.cols());\r\n    Eigen::MatrixXd y_row_buffer = y.row(0);\r\n    auto yi = constant_view(y_row_buffer.data(), 1, y.cols());\r\n    auto expr = bind(pow\u003c2\u003e(yi - dot(xi, theta)));\r\n\r\n    // Seed\r\n    Eigen::MatrixXd seed(1, 1);\r\n    seed.setOnes(); // Usually seed is 1. DONT'T FORGET!\r\n\r\n    // Loop over each row to calulate loss.\r\n    double loss = 0;\r\n    for (int i = 0; i \u003c X.rows(); ++i) {\r\n        x_row_buffer = X.row(i);\r\n        y_row_buffer = y.row(i);\r\n\r\n        auto f = autodiff(expr, seed.array());\r\n        loss += f.coeff(0);\r\n    }\r\n\r\n    // Print results.\r\n    std::cout \u003c\u003c \"loss: \" \u003c\u003c loss \u003c\u003c std::endl; // 6655\r\n    std::cout \u003c\u003c theta.get() \u003c\u003c std::endl;      //[1, 2]\r\n    std::cout \u003c\u003c theta.get_adj() \u003c\u003c std::endl;  //[-1210, -12100]\r\n\r\n    theta_adj.setZero(); // Reset differential to zero after one full pass.\r\n\r\n    return 0;\r\n}\r\n```\r\n\r\n### 3-layer Neural Network with Jump Connection\r\n\r\nHere is an example of 3-layer neural network. The result has been confirmed by symbolic differential using Mathematica. See `test/reverse/util/GenTestData.nb` for Mathematica code used.\r\n\r\nThere is also a full independent example that using `ceres` to train this network in `examples/ceres_3layer_neural_net`. `ceres` has it's own automatic differential tool for non-linear least square problem which is unavailiable for the more versatile `GradientProbelm`. That's why we need `FastAD`. To run this example, you need to install and `Eigen` and `ceres`.\r\n\r\n```cpp\r\n#include \"fastad\"\r\n#include \u003ciostream\u003e\r\n\r\nusing namespace ad;\r\n// A3*s(A2*s(A1*x+b1)+b2+(A1*x+b1))+b3\r\nstruct NN {\r\n    Eigen::MatrixXd X, y;\r\n    NN(const Eigen::MatrixXd \u0026X, const Eigen::VectorXd \u0026y)\r\n        : X(X), y(y){\r\n\r\n                };\r\n    double loss(const double *parm, double *grad) {\r\n        Eigen::Map\u003cEigen::VectorXd\u003e g(grad, 25);\r\n        g.setZero();\r\n\r\n\t\t//Create variables.\r\n        VarView\u003cdouble, mat\u003e A1(const_cast\u003cdouble *\u003e(parm), grad, 3, 2);\r\n        VarView\u003cdouble, mat\u003e b1(const_cast\u003cdouble *\u003e(parm + 6), grad + 6, 3, 1);\r\n        VarView\u003cdouble, mat\u003e A2(const_cast\u003cdouble *\u003e(parm + 9), grad + 9, 3, 3);\r\n        VarView\u003cdouble, mat\u003e b2(const_cast\u003cdouble *\u003e(parm + 18), grad + 18, 3, 1);\r\n        VarView\u003cdouble, mat\u003e A3(const_cast\u003cdouble *\u003e(parm + 21), grad + 21, 1, 3);\r\n        VarView\u003cdouble, mat\u003e b3(const_cast\u003cdouble *\u003e(parm + 24), grad + 24, 1, 1);\r\n\r\n        // Data buffer\r\n        Eigen::VectorXd x_row_buffer = X.row(0);\r\n        auto xi = constant_view(x_row_buffer.data(), X.cols(), 1);\r\n        Eigen::VectorXd y_row_buffer = y.row(0);\r\n        auto yi = constant_view(y_row_buffer.data(), y.cols(), 1);\r\n        \r\n        // Expression\r\n        auto x1 = dot(A1, xi) + b1;\r\n        auto y1 = sigmoid(x1);\r\n        auto y2 = sigmoid(dot(A2, y1) + b2) + x1;\r\n        auto y3 = dot(A3, y2) + b3;\r\n        auto residual_norm2 = pow\u003c2\u003e(yi - y3);\r\n        auto expr = bind(residual_norm2);\r\n\r\n        // Seed\r\n        Eigen::MatrixXd seed(1, 1);\r\n        seed.setOnes(); // Usually seed is 1. DONT'T FORGET!\r\n\r\n        // Loop over each row to calulate loss.\r\n        double loss = 0;\r\n        for (int i = 0; i \u003c X.rows(); ++i) {\r\n            x_row_buffer = X.row(i);\r\n            y_row_buffer = y.row(i);\r\n\r\n            auto f = autodiff(expr, seed.array());\r\n            loss += f.coeff(0);\r\n        }\r\n        return loss;\r\n    };\r\n};\r\n\r\nint main() {\r\n\r\n    // Create data matrix.\r\n    Eigen::MatrixXd X(5, 2);\r\n    X \u003c\u003c 1, 10, 2, 20, 3, 30, 4, 40, 5, 50;\r\n    Eigen::VectorXd y(5);\r\n    y \u003c\u003c 32, 64, 96, 128, 160; // y=2*x1+3*x2\r\n\r\n    // Generating parameter buffer and NN.\r\n    Eigen::VectorXd parm_data(25);\r\n    parm_data \u003c\u003c 0.043984, 0.960126, -0.520941, -0.800526, -0.0287914, 0.635809, 0.584603,\r\n        -0.443382, 0.224304, 0.97505, -0.824084, 0.2363, 0.666392, -0.498828, -0.781428, -0.911053,\r\n        -0.230156, -0.136367, 0.263425, 0.841535, 0.920342, 0.65629, 0.848248, -0.748697, 0.21522;\r\n\r\n    Eigen::VectorXd grad_data(25);\r\n    grad_data.setZero(); // Set adjoints to zeros.\r\n    NN net(X, y);\r\n\r\n    auto loss = net.loss(parm_data.data(), grad_data.data());\r\n\r\n    // Print results.\r\n    std::cout \u003c\u003c \"loss: \" \u003c\u003c loss \u003c\u003c std::endl;      // 6655\r\n    std::cout \u003c\u003c \"parm: \" \u003c\u003c parm_data \u003c\u003c std::endl; //[1, 2]\r\n    std::cout \u003c\u003c \"diff: \" \u003c\u003c grad_data \u003c\u003c std::endl; //[-1210, -12100]\r\n\r\n    return 0;\r\n}\r\n```\r\n## Quick Reference\r\n\r\n### Forward \r\n\r\n__ForwardVar\u003cT\u003e__:\r\n- class representing a variable for forward AD\r\n- `set_value(T x)`: sets value to x\r\n- `get_value()`: gets underlying value\r\n- `set_adjoint(T x)`: sets adjoint to x\r\n- `get_adjoint()`: gets underlying adjoint\r\n\r\n__Unary Functions__:\r\n- unary minus: `operator-`\r\n- trig functions: `sin, cos, tan, asin, acos, atan`\r\n- others: `exp, log, sqrt`\r\n\r\n__Operators__:\r\n- binary: `+,-,*,/`\r\n- increment: `+=`\r\n\r\n### Reverse \r\n\r\n__Shape Types__:\r\n- `ad::scl, ad::vec, ad::mat`\r\n\r\n__VarView\u003cT, ShapeType=scl\u003e__:\r\n- This is only useful for users who really want to optimize for performance\r\n- `ShapeType` must be one of the types listed above\r\n- `T` is the underlying value type\r\n- `VarView(T* v, T* a, rows=1, cols=1)`:\r\n    - constructs to view values starting from v,\r\n      adjoints starting from a, and has the shape of rows x cols.\r\n    - vector shapes must pass rows\r\n    - matrix shapes must pass both rows and cols\r\n- `VarView()`\r\n    - constructs with nullptrs\r\n- `.bind(T* begin)`: views values starting from begin\r\n- `.bind_adj(T* begin)`: views adjoints starting from begin\r\n\r\n__Var\u003cT, ShapeType=scl\u003e__:\r\n- A `Var` is a `VarView` (views itself)\r\n- Main difference with `VarView` is that it owns the values and adjoints\r\n- Users will primarily use this class to represent AD variables.\r\n- API is same as `VarView`\r\n\r\n__Unary Functions (vectorized if multi-dimensional)__:\r\n- unary minus: `operator-`\r\n- trig functions: `sin, cos, tan, asin, acos, atan`\r\n- Hyperbolic: `sinh, cosh, tanh`\r\n- Neural network activations: `sigmoid`\r\n- others: `exp, log, sqrt, erf`\r\n\r\n__Operators__:\r\n- binary: `+,-,*,/`\r\n- modification: `+=`, `-=`, `*=`, `/=`\r\n- comparison: `\u003c,\u003c=,\u003e,\u003e=,==,!=,\u0026\u0026,||`\r\n    - Note: `\u0026\u0026` and `||` are undefined behavior for \r\n      multi-dimensional non-boolean expressions\r\n- placeholder: `operator=`\r\n    - only overloaded for `VarView` expressions\r\n- glue: `operator,`\r\n    - any expressions can be \"glued\" using this operator\r\n- unary minus: `operator-`\r\n- trig functions: `sin, cos, tan, asin, acos, atan`\r\n- others: `exp, log, sqrt`\r\n\r\n__Special Expressions__:\r\n- `ad::constant(T)`:\r\n- `ad::constant(const Eigen::Vector\u003cT, Eigen::Dynamic, 1\u003e\u0026)`:\r\n- `ad::constant(const Eigen::Matrix\u003cT, Eigen::Dynamic, Eigen::Dynamic\u003e\u0026)`:\r\n- `ad::constant_view(T*)`:\r\n- `ad::constant_view(T*, rows)`:\r\n- `ad::constant_view(T*, rows, cols)`:\r\n- `ad::det\u003cpolicy\u003e(m)`:\r\n    - determinant of matrix `m`\r\n    - `policy` must be one of: `DetFullPivLU`, `DetLDLT`, `DetLLT`\r\n        - see `Eigen` documentation for each of these (without the prefix `Det`) for \r\n          when they apply.\r\n- `ad::dot(m, v)`:\r\n    - represents matrix product with a matrix and a (column) vector\r\n- `ad::for_each(begin, end, f)`:\r\n    - generalization of operator,\r\n    - represents evaluating expressions generated by `f` when fed with elements\r\n      from `begin` to `end`.\r\n- `ad::if_else(cond, if, else)`:\r\n    - represents an if-else statement\r\n    - `cond` MUST be a scalar expression\r\n    - `if` and `else` must have the exact same shape\r\n- `ad::log_det\u003cpolicy\u003e(m)`\r\n    - same as `det\u003cpolicy\u003e(m)` but computes log-abs-determinant\r\n    - `policy` must be one of: `LogDetFullPivLU`, `LogDetLDLT`, `LogDetLLT`\r\n- `ad::norm(v)`:\r\n    - represents the squared norm of a vector or Frobenius norm for matrix\r\n- `ad::pow\u003cn\u003e(e)`:\r\n    - compile-time known, integer-powered expression\r\n- `ad::prod(begin, end, f)`:\r\n    - represents the product of expressions generated by `f`\r\n      when fed with elements from `begin` to `end`.\r\n- `ad::prod(e)`:\r\n    - represents the product of all _elements_ of the expression `e`\r\n    - e.g. if `e` is a vector expression, it represents the product of all its elements.\r\n- `ad::sum(begin, end, f)`:\r\n- `ad::sum(e)`:\r\n    - same as prod but represents summation\r\n- `ad::transpose(e)`:\r\n\t- matrix or vector transpose.\r\n\r\n__Stats Expressions__:\r\nAll log-pdfs are adjusted to omit constants.\r\nParameters can have various combinations of shapes and follow the usual vectorized notion.\r\n- `ad::bernoulli(x, p)`\r\n- `ad::cauchy_adj_log_pdf(x, loc, scale)`\r\n- `ad::normal_adj_log_pdf(x, mu, s)`\r\n- `ad::uniform_adj_log_pdf(x, min, max)`\r\n- `ad::wishart_adj_log_pdf(X, V, n)`\r\n\r\n## Contact\r\n\r\nIf you have any questions about FastAD, please [open an issue](https://github.com/JamesYang007/FastAD/issues/new).\r\nWhen opening an issue, please describe in the fullest detail with a minimal example to recreate the problem.\r\n\r\nFor other general questions that cannot be resolved through opening issues,\r\nfeel free to [send me an email](mailto:jamesyang916@gmail.com).\r\n\r\n## Contributors\r\n\r\n| **James Yang** | **Kent Hall** | **Jean-Christophe Ruel** | **ZhouYao** |\r\n| :---: | :---: | :---: | :---: |\r\n| [![JamesYang007](https://avatars3.githubusercontent.com/u/5008832?s=100\u0026v=4)](https://github.com/JamesYang007) | [![Kent](https://avatars3.githubusercontent.com/u/4146614?s=100\u0026v=4)](https://github.com/kentjhall) | [![Jean-Christophe Ruel](https://avatars3.githubusercontent.com/u/16375770?s=100\u0026v=4)](https://github.com/jeanchristopheruel) | [](https://github.com/kilasuelika) |\r\n| \u003ca href=\"http://github.com/JamesYang007\" target=\"_blank\"\u003e`github.com/JamesYang007`\u003c/a\u003e | \u003ca href=\"http://github.com/kentjhall\" target=\"_blank\"\u003e`github.com/kentjhall`\u003c/a\u003e | \u003ca href=\"http://github.com/jeanchristopheruel\" target=\"_blank\"\u003e`github.com/jeanchristopheruel`\u003c/a\u003e | \u003ca href=\"http://github.com/kilasuelika\" target=\"_blank\"\u003e`github.com/kilasuelika`\u003c/a\u003e |\r\n\r\n## Third Party Tools\r\n\r\nMany third party tools were used for this project.\r\n\r\n- [Clang](https://clang.llvm.org/): main compiler used for development.\r\n- [CMake](https://cmake.org/): build automation.\r\n- [Codacy](https://app.codacy.com/welcome/organizations): rigorous code analysis.\r\n- [Coveralls](https://coveralls.io/): for measuring and uploading [code coverage](https://coveralls.io/github/JamesYang007/FastAD).\r\n- [Cpp Coveralls](https://github.com/eddyxu/cpp-coveralls): for measuring code coverage in Coveralls.\r\n- [Eigen](http://eigen.tuxfamily.org/index.php?title=Main_Page): matrix library that we wrapped to create AD expressions.\r\n- [GCC](https://gcc.gnu.org/): compiler used to develop in linux environment.\r\n- [Github Changelog Generator](https://github.com/github-changelog-generator/github-changelog-generator): generate [CHANGELOG](https://github.com/JamesYang007/FastAD/blob/master/CHANGELOG.md).\r\n- [Google Benchmark](https://github.com/google/benchmark): benchmark against various methods.\r\n- [GoogleTest](https://github.com/google/googletest): unit-test and integration-test.\r\n- [Travis](https://travis-ci.org/): continuous integration for Linux and MacOS. See [.travis.yml](https://github.com/JamesYang007/FastAD/blob/master/.travis.yml) for more details.\r\n- [Valgrind](http://valgrind.org/): check memory leak/error.\r\n\r\n## License\r\n\r\n- **[MIT license](http://opensource.org/licenses/mit-license.php)**\r\n- Copyright 2019 ©JamesYang007.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesyang007%2Ffastad","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjamesyang007%2Ffastad","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesyang007%2Ffastad/lists"}