https://github.com/MikeInnes/diff-zoo

Differentiation for Hackers
https://github.com/MikeInnes/diff-zoo

Last synced: about 2 months ago
JSON representation

Differentiation for Hackers

• Host: GitHub
• URL: https://github.com/MikeInnes/diff-zoo
• Owner: MikeInnes
• License: mit
• Created: 2018-11-01T15:04:59.000Z (almost 6 years ago)
• Default Branch: master
• Last Pushed: 2021-07-09T11:52:53.000Z (about 3 years ago)
• Last Synced: 2024-06-19T07:42:12.622Z (3 months ago)
• Language: Julia
• Homepage:
• Size: 87.9 KB
• Stars: 771
• Watchers: 19
• Forks: 63
• Open Issues: 11
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Differentiation for Hackers

[![Build Status](https://travis-ci.org/MikeInnes/diff-zoo.svg?branch=master)](https://travis-ci.org/MikeInnes/diff-zoo)

The goal of this handbook is to demystify *algorithmic differentiation*, the

tool that underlies modern machine learning. It begins with a calculus-101 style

understanding and gradually extends this to build toy implementations of systems

similar to PyTorch and TensorFlow. I have tried to clarify the relationships

between every kind of differentiation I can think of – including forward and

reverse, symbolic, numeric, tracing and source transformation. Where typical real-word ADs are mired in implementation details, these implementations are designed to be coherent enough that the real, fundamental differences – of which there are surprisingly few – become obvious.

The intro notebook is recommended to start with, but otherwise notebooks do not have a fixed order.

* [Intro](https://github.com/MikeInnes/diff-zoo/blob/notebooks/intro.ipynb) – explains the basics, beginning with a simple symbolic differentiation routine.

* [Back & Forth](https://github.com/MikeInnes/diff-zoo/blob/notebooks/backandforth.ipynb) – discusses the difference between forward and reverse mode AD.

* [Forward](https://github.com/MikeInnes/diff-zoo/blob/notebooks/forward.ipynb) – discusses forward-mode AD and its relationship to symbolic and numerical differentiation.

* [Tracing](https://github.com/MikeInnes/diff-zoo/blob/notebooks/tracing.ipynb) – discusses tracing-based implementations of reverse mode, as used by TensorFlow and PyTorch.

* [Reverse](https://github.com/MikeInnes/diff-zoo/blob/notebooks/reverse.ipynb) – discusses a more powerful reverse mode based on source transformation (not complete).

If you want to run the notebooks locally, they can be built by running the

`src/notebooks.jl` script using Julia. They should appear inside a `/notebooks`

folder. Alternatively, you can run through the scripts in Juno.