{"id":19297846,"url":"https://github.com/analogdevicesinc/ai8x-training","last_synced_at":"2025-04-04T10:07:54.265Z","repository":{"id":37818658,"uuid":"265377155","full_name":"analogdevicesinc/ai8x-training","owner":"analogdevicesinc","description":"Model Training for ADI's MAX78000 and MAX78002 Edge AI Devices","archived":false,"fork":false,"pushed_at":"2025-01-14T00:04:03.000Z","size":231469,"stargazers_count":98,"open_issues_count":1,"forks_count":93,"subscribers_count":18,"default_branch":"develop","last_synced_at":"2025-03-28T09:07:21.470Z","etag":null,"topics":["ai","analog-devices","artificial-intelligence","deep-learning","machine-learning","max78000","max78002","maxim","maxim-integrated"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/analogdevicesinc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-19T21:58:05.000Z","updated_at":"2025-03-26T14:04:50.000Z","dependencies_parsed_at":"2024-04-05T14:47:21.380Z","dependency_job_id":"7ee3f271-404e-4953-88fd-a9a7042c33a8","html_url":"https://github.com/analogdevicesinc/ai8x-training","commit_stats":null,"previous_names":["analogdevicesinc/ai8x-training","maximintegratedai/ai8x-training"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/analogdevicesinc%2Fai8x-training","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/analogdevicesinc%2Fai8x-training/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/analogdevicesinc%2Fai8x-training/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/analogdevicesinc%2Fai8x-training/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/analogdevicesinc","download_url":"https://codeload.github.com/analogdevicesinc/ai8x-training/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247157281,"owners_count":20893220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","analog-devices","artificial-intelligence","deep-learning","machine-learning","max78000","max78002","maxim","maxim-integrated"],"created_at":"2024-11-09T23:06:08.941Z","updated_at":"2025-04-04T10:07:54.233Z","avatar_url":"https://github.com/analogdevicesinc.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ADI MAX78000/MAX78002 Model Training and Synthesis\n\nNovember 7, 2024\n\n**Note: This branch requires PyTorch 2. Please see the archive-1.8 branch for PyTorch 1.8 support. [KNOWN_ISSUES](KNOWN_ISSUES.txt) contains a list of known issues.**\n\nADI’s MAX78000/MAX78002 project is comprised of five repositories:\n\n1. **Start here**:\n **[Top Level Documentation](https://github.com/analogdevicesinc/MaximAI_Documentation)**\n2. The software development kit (MSDK), which contains drivers and example programs ready to run on the evaluation kits (EVkit and Feather):\n [Analog Devices MSDK](https://github.com/analogdevicesinc/msdk)\n3. The training repository, which is used for deep learning *model development and training*:\n [ai8x-training](https://github.com/analogdevicesinc/ai8x-training/tree/develop) **(described in this document)**\n4. The synthesis repository, which is used to *convert a trained model into C code* using the “izer” tool:\n [ai8x-synthesis](https://github.com/analogdevicesinc/ai8x-synthesis/tree/develop) **(described in this document)**\n5. The reference design repository, which contains host applications and sample applications for reference designs such as [MAXREFDES178 (Cube Camera)](https://www.analog.com/en/design-center/reference-designs/maxrefdes178.html):\n [refdes](https://github.com/analogdevicesinc/MAX78xxx-RefDes)\n *Note: Examples for EVkits and Feather boards are part of the MSDK*\n\n_Open the `.md` version of this file in a markdown enabled viewer, for example Typora (\u003chttp://typora.io\u003e).\nSee \u003chttps://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet\u003e for a description of Markdown. A [PDF copy of this file](README.pdf) is available in this repository. The GitHub rendering of this document does not show the mathematical formulas. Use the ≡ button to access the table of contents on GitHub._\n\n---\n\n[TOC]\n\n## Part Numbers\n\nThis document covers several of ADI’s ultra-low power machine learning accelerator systems. They are sometimes referred to by their die types. The following shows the die types and their corresponding part numbers:\n\n| Die Type | Part Number(s) |\n| -------- | ------------------------------ |\n| *AI84* | *Unreleased test chip* |\n| AI85 | **MAX78000** (full production) |\n| AI87 | **MAX78002** (full production) |\n\n## Overview\n\nThe following graphic shows an overview of the development flow:\n\n![Development Flow](docs/DevelopmentFlow.png)\n\n## Installation\n\n### File System Layout\n\nIncluding the MSDK, the expected/resulting file system layout will be:\n\n ..../ai8x-training/\n ..../ai8x-synthesis/\n ..../ai8x-synthesis/sdk/ [or a different path selected by the user]\n\nwhere “....” is the project root, for example `~/Documents/Source/AI`.\n\n### Prerequisites\n\nThis software requires PyTorch. *TensorFlow / Keras support is deprecated.*\n\nPyTorch operating system and hardware support are constantly evolving. This document does not cover all possible combinations of operating system and hardware. Instead, this document describes how to install PyTorch on one officially supported platform.\n\n#### Platform Recommendation and Full Support\n\nFull support and documentation are provided for the following platform:\n\n* CPU: 64-bit amd64/x86_64 “PC” with [Ubuntu Linux 20.04 LTS or 22.04 LTS](https://ubuntu.com/download/server)\n* GPU for hardware acceleration (optional but highly recommended): Nvidia with [CUDA 12.1](https://developer.nvidia.com/cuda-toolkit-archive) or later\n* [PyTorch 2.3](https://pytorch.org/get-started/locally/) on Python 3.11.x\n\nLimited support and advice for using other hardware and software combinations is available as follows.\n\n#### Operating System Support\n\n##### Linux\n\n**The only officially supported platforms for model training** are Ubuntu Linux 20.04 LTS and 22.04 LTS on amd64/x86_64, either the desktop or the [server version](https://ubuntu.com/download/server).\n\n*Note that hardware acceleration using CUDA is \u003cu\u003enot available\u003c/u\u003e in PyTorch for Raspberry Pi 4 and other \u003cu\u003eaarch64/arm64\u003c/u\u003e devices, even those running Ubuntu Linux 20.04/22.04. See also [Development on Raspberry Pi 4 and 400](https://github.com/analogdevicesinc/ai8x-synthesis/blob/develop/docs/RaspberryPi.md) (unsupported).*\n\nThis document also provides instructions for installing on RedHat Enterprise Linux / CentOS 8 with limited support.\n\n##### Windows\n\nOn Windows 10 version 21H2 or newer, and Windows 11, after installing the Windows Subsystem for Linux (WSL2), Ubuntu Linux 20.04 or 22.04 can be used inside Windows with full CUDA acceleration, please see *[Windows Subsystem for Linux](https://github.com/analogdevicesinc/ai8x-synthesis/blob/develop/docs/WSL2.md).* For the remainder of this document, follow the steps for Ubuntu Linux.\n\nIf WSL2 is not available, it is also possible (but not recommended due to inherent compatibility issues and slightly degraded performance) to run this software natively on Windows. Please see *[Native Windows Installation](https://github.com/analogdevicesinc/ai8x-synthesis/blob/develop/docs/Windows.md)*.\n\n##### macOS\n\nThe software works on macOS and uses MPS acceleration on Apple Silicon. On Intel CPUs, model training suffers from the lack of hardware acceleration.\n\n##### Virtual Machines (Unsupported)\n\nThis software works inside a virtual machine running Ubuntu Linux 20.04 or 22.04. However, GPU passthrough is potentially difficult to set up and \u003cu\u003enot always available\u003c/u\u003e for Linux VMs, so there may be no CUDA hardware acceleration. Certain Nvidia cards support [vGPU software](https://www.nvidia.com/en-us/data-center/graphics-cards-for-virtualization/); see also [vGPUs and CUDA](https://docs.nvidia.com/cuda/vGPU/), but vGPU features may come at substantial additional cost and vGPU software is not covered by this document.\n\n##### Docker Containers (Unsupported)\n\nThis software also works inside Docker containers. However, CUDA support inside containers requires Nvidia Docker ([see blog entry](https://developer.nvidia.com/blog/nvidia-docker-gpu-server-application-deployment-made-easy/)) and is not covered by this document.\n\n#### PyTorch and Python\n\nThe officially supported version of [PyTorch is 2.3](https://pytorch.org/get-started/locally/) running on Python 3.11.x. Newer versions will typically work, but are not covered by support, documentation, and installation scripts.\n\n#### Hardware Acceleration\n\nWhen going beyond simple models, model training does not work well without hardware acceleration – Nvidia CUDA, AMD ROCm, or Apple Silicon MPS. The network loader (“izer”) does \u003cu\u003enot\u003c/u\u003e require hardware acceleration, and very simple models can also be trained on systems without hardware acceleration.\n\n* CUDA requires modern Nvidia GPUs. This is the most compatible, and best supported hardware accelerator.\n* ROCm requires certain AMD GPUs, see [blog entry](https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-available-as-python-package/).\n* MPS requires Apple Silicon (M1 or newer) and macOS 12.3 or newer.\n* PyTorch does not include CUDA support for aarch64/arm64 systems. *Rebuilding PyTorch from source is not covered by this document.*\n\n\n##### Using Multiple GPUs\n\nWhen using multiple GPUs (graphics cards), the software will automatically use all available GPUs and distribute the workload. To prevent this (for example, when the GPUs are not balanced), set the `CUDA_VISIBLE_DEVICES` environment variable. Use the `--gpus` command line argument to set the default GPU.\n\n#### Shared (Multi-User) and Remote Systems\n\nOn a shared (multi-user) system that has previously been set up, only local installation is needed. CUDA and any `apt-get` or `brew` tasks are not necessary, with the exception of the CUDA [Environment Setup](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#environment-setup).\n\nThe `screen` command (or alternatively, the more powerful `tmux`) can be used inside a remote terminal to disconnect a session from the controlling terminal, so that a long running training session doesn’t abort due to network issues, or local power saving. In addition, screen can log all console output to a text file.\n\nExample:\n\n```shell\n$ ssh targethost\ntargethost$ screen -L # or screen -r to resume, screen -list to list\ntargethost$\nCtrl+A,D to disconnect\n```\n\n`man screen` and `man tmux` describe the software in more detail.\n\n#### Additional Software\n\nThe following software is optional, and can be replaced with other similar software of the user’s choosing.\n\n1. Code Editor\n Visual Studio Code, \u003chttps://code.visualstudio.com\u003e or the VSCodium version, \u003chttps://vscodium.com\u003e, with the “Remote - SSH” plugin; *to use Visual Studio Code on Windows as a full development environment (including debug), see \u003chttps://github.com/analogdevicesinc/VSCode-Maxim\u003e*\n Sublime Text, \u003chttps://www.sublimetext.com\u003e\n2. Markdown Editor\n Typora, \u003chttp://typora.io\u003e\n3. Serial Terminal\n CoolTerm, \u003chttp://freeware.the-meiers.org\u003e\n Serial, \u003chttps://apps.apple.com/us/app/serial/id877615577?mt=12\u003e\n Putty, \u003chttps://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html\u003e\n Tera Term, \u003chttps://osdn.net/projects/ttssh2/releases/\u003e\n4. Graphical Git Client\n GitHub Desktop, \u003chttps://desktop.github.com\u003e\n Git Fork, \u003chttps://git-fork.com\u003e\n5. Diff and Merge Tool\n Beyond Compare, \u003chttps://scootersoftware.com\u003e\n\n### Project Installation\n\n#### Free Disk Space\n\nA minimum of 64 GB of free disk space is recommended, and datasets can be many times this size and should be stored separately. Check the available space on the target file system before continuing using\n\n```shell\n$ df -kh\nFilesystem Size Used Avail Use% Mounted on\n...\n/dev/sda2 457G 176G 259G 41% /\n```\n\n#### System Packages\n\nSome additional system packages are required, and installation of these additional packages requires administrator privileges. Note that this is the only time administrator privileges are required.\n\n##### macOS\n\nOn macOS use:\n\n```shell\n$ brew install libomp libsndfile tcl-tk sox\n```\n\n##### Linux (Ubuntu), including WSL2)\n\n```shell\n$ sudo apt-get install -y make build-essential libssl-dev zlib1g-dev \\\n libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \\\n libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev \\\n libsndfile-dev portaudio19-dev libsox-dev\n```\n\n###### RedHat Enterprise Linux / CentOS 8\n\nWhile Ubuntu 20.04 LTS and 22.04 LTS are the supported distributions, the MAX78000/MAX78002 software packages run fine on all modern Linux distributions that also support CUDA. The *apt-get install* commands above must be replaced with distribution specific commands and package names. Unfortunately, there is no obvious 1:1 mapping between package names from one distribution to the next. The following example shows the commands needed for RHEL/CentOS 8.\n\n*Two of the required packages are not in the base repositories. Enable the EPEL and PowerTools repositories:*\n\n```shell\n$ sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm\n$ sudo dnf config-manager --set-enabled powertools\n```\n\n*Proceed to install the required packages:*\n\n```shell\n$ sudo dnf group install \"Development Tools\"\n$ sudo dnf install openssl-devel zlib-devel \\\n bzip2-devel readline-devel sqlite-devel wget llvm \\\n xz-devel tk tk-devel libffi-devel \\\n libsndfile libsndfile-devel portaudio-devel\n```\n\n#### Python 3.11\n\n*The software in this project uses Python 3.11.8 or a later 3.11.x version.*\n\nFirst, check whether there is a default Python interpreter and whether it is version 3.11.x:\n\n```shell\n$ python --version\nCommand 'python' not found, did you mean:\n command 'python3' from deb python3\n command 'python' from deb python-is-python3\n# no default python, install pyenv\n\n$ python --version\nPython 2.7.18\n# wrong version, pyenv required\n```\n\nPython 2 **will not function correctly** with the MAX78000/MAX78002 tools. If the result is Python **3.11**.x, skip ahead to [git Environment](#git-environment). For *any* other version (for example, 2.7, 3.7, 3.8, 3.9, 3.10), or no version, continue here.\n\n*Note: For the purposes of the MAX78000/MAX78002 tools, “python3” is not a substitute for “python”. Please install pyenv when `python --version` does not return version 3.11.x, \u003cu\u003eeven if\u003c/u\u003e “python3” is available.*\n\n*Note for advanced users: `sudo apt-get install python-is-python3` on Ubuntu 20.04 will install Python 3 as the default Python version; however, it may not be version 3.11.x.*\n\n##### pyenv\n\nIt is not necessary to install Python 3.11 system-wide, or to rely on the system-provided Python. To manage Python versions, instead use `pyenv` (\u003chttps://github.com/pyenv/pyenv\u003e). This allows multiple Python versions to co-exist on the same system without interfering with the system or with one another.\n\nOn macOS:\n\n```shell\n$ brew install pyenv pyenv-virtualenv\n```\n\nOn Linux:\n\n```shell\n$ curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash # NOTE: Verify contents of the script before running it!!\n```\n\nThen, follow the terminal output of the pyenv-installer and add pyenv to your shell by modifying one or more of `~/.bash_profile`, `~/.bashrc`, `~/.zshrc`, `~/.profile`, or `~/.zprofile`. The instructions differ depending on the shell (bash or zsh).\n\nFor example, on *Ubuntu 20.04 inside WSL2* add the following to `~/.bashrc`:\n\n```shell\n# WSL2\nexport PYENV_ROOT=\"$HOME/.pyenv\"\nexport PATH=\"$PYENV_ROOT/bin:$PATH\"\neval \"$(pyenv init --path)\"\neval \"$(pyenv virtualenv-init -)\"\n```\n\nTo display the instructions again at any later time:\n\n```shell\n$ ~/.pyenv/bin/pyenv init\n\n# (The below instructions are intended for common\n# shell setups. See the README for more guidance\n# if they don't apply and/or don't work for you.)\n\n# Add pyenv executable to PATH and\n# enable shims by adding the following\n# to ~/.profile and ~/.zprofile:\n...\n...\n```\n\n*Note: Installing both conda and pyenv in parallel may cause issues. Ensure that the pyenv initialization tasks are executed \u003cu\u003ebefore\u003c/u\u003e any conda related tasks.*\n\nNext, close the Terminal, open a new Terminal and install Python 3.11.8.\n\nOn macOS:\n\n```shell\n$ env \\\n PATH=\"$(brew --prefix tcl-tk)/bin:$PATH\" \\\n LDFLAGS=\"-L$(brew --prefix tcl-tk)/lib\" \\\n CPPFLAGS=\"-I$(brew --prefix tcl-tk)/include\" \\\n PKG_CONFIG_PATH=\"$(brew --prefix tcl-tk)/lib/pkgconfig\" \\\n CFLAGS=\"-I$(brew --prefix tcl-tk)/include\" \\\n PYTHON_CONFIGURE_OPTS=\"--with-tcltk-includes='-I$(brew --prefix tcl-tk)/include' --with-tcltk-libs='-L$(brew --prefix tcl-tk)/lib -ltcl8.6 -ltk8.6'\" \\\n pyenv install 3.11.8\n```\n\nOn Linux, including WSL2:\n\n```shell\n$ pyenv install 3.11.8\n```\n\n#### git Environment\n\nIf the local git environment has not been previously configured, add the following commands to configure e-mail and name. The e-mail must match GitHub (including upper/lower case):\n\n```shell\n$ git config --global user.email \"first.last@example.com\"\n$ git config --global user.name \"First Last\"\n```\n\n#### Nervana Distiller\n\n[Nervana Distiller](https://github.com/analogdevicesinc/distiller) is automatically installed as a git sub-module with the other packages. Distiller is used for its scheduling and model export functionality.\n\n### Upstream Code\n\nChange to the project root and run the following commands. Use your GitHub credentials if prompted.\n\n```shell\n$ cd \u003cyour/project\u003e\n$ git clone --recursive https://github.com/analogdevicesinc/ai8x-training.git\n$ git clone --recursive https://github.com/analogdevicesinc/ai8x-synthesis.git\n```\n\n#### Creating the Virtual Environment\n\nTo create the virtual environment and install basic wheels:\n\n```shell\n$ cd ai8x-training\n```\n\nUsing the instructions above checks out the `develop` branch which supports PyTorch 2.3. The `main` branch is updated less frequently, but possibly more stable. To change branches, use the command `git checkout`, for example `git checkout main`.\nFor PyTorch 1.8 support, use the archive.\n\nIf using pyenv, set the local directory to use Python 3.11.8.\n\n```shell\n$ pyenv local 3.11.8\n```\n\nIn all cases, verify that a 3.11.x version of Python is used:\n\n```shell\n$ python --version\nPython 3.11.8\n```\n\nIf this does \u003cu\u003e*not*\u003c/u\u003e return version 3.11.x, please install and initialize [pyenv](#python-311).\n\nThen continue with the following:\n\n```shell\n$ python -m venv .venv --prompt ai8x-training\n$ echo \"*\" \u003e .venv/.gitignore\n```\n\nIf this command returns an error message similar to *“The virtual environment was not created successfully because ensurepip is not available,”* please install and initialize [pyenv](#python-311).\n\nOn macOS and Linux, including WSL2, activate the environment using\n\n```shell\n$ source .venv/bin/activate\n```\n\nOn native Windows, instead use:\n\n```shell\n$ source .venv/Scripts/activate\n```\n\nThen continue with\n\n```shell\n(ai8x-training) $ pip3 install -U pip wheel setuptools\n```\n\nThe next step differs depending on whether the system uses CUDA 12.1+, or not.\n\nFor CUDA 12:\n\n```shell\n(ai8x-training) $ pip3 install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu121\n```\n\nFor ROCm 5.7:\n\n```shell\n(ai8x-training) $ pip3 install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/rocm5.7\n```\n\nFor all other systems, including macOS:\n\n```shell\n(ai8x-training) $ pip3 install -r requirements.txt\n```\n\n##### Repository Branches\n\nUsing the instructions above checks out the `develop` branch which supports PyTorch 2.3. The `main` branch is updated less frequently, but possibly more stable. To change branches, use the command `git checkout`, for example `git checkout main`.\nFor PyTorch 1.8 support, use the archive.\n\n###### TensorFlow / Keras\n\nSupport for TensorFlow / Keras is deprecated.\n\n#### Updating to the Latest Version\n\nAfter additional testing, `develop` is merged into the main branch at regular intervals.\n\nAfter a small delay of typically a day, a “Release” tag is created on GitHub for all non-trivial merges into the main branch. GitHub offers email alerts for all activity in a project, or for new releases only. Subscribing to releases only substantially reduces email traffic.\n\n*Note: Each “Release” automatically creates a code archive. It is recommended to use a git client to access (pull from) the main branch of the repository using a git client instead of downloading the archives.*\n\nIn addition to code updated in the repository itself, **submodules and Python libraries may have been updated as well**.\n\nMajor upgrades (such as updating from PyTorch 1.8 to PyTorch 2.3) are best done by removing all installed wheels. This can be achieved most easily by creating a new folder and starting from scratch at [Upstream Code](#upstream-code). Starting from scratch is also recommended when upgrading the Python version.\n\nFor minor updates, pull the latest code and install the updated wheels:\n\n```shell\n(ai8x-training) $ git pull\n(ai8x-training) $ git submodule update --init\n(ai8x-training) $ pip3 install -U pip setuptools\n(ai8x-training) $ pip3 install -U -r requirements.txt # add --extra-index-url if needed, as shown above\n```\n\n##### MSDK Updates\n\nPlease *also* update the MSDK or use the Maintenance Tool as documented in the [Analog Devices MSDK documentation](https://github.com/analogdevicesinc/msdk). The Maintenance Tool automatically updates the MSDK.\n\n##### Python Version Updates\n\nUpdating Python may require updating `pyenv` first. Should `pyenv install 3.11.8` fail,\n\n```shell\n$ pyenv install 3.11.8\npython-build: definition not found: 3.11.8\n```\n\nthen `pyenv` must be updated. On macOS, use:\n\n```shell\n$ brew update \u0026\u0026 brew upgrade pyenv\n...\n$\n```\n\nOn Linux (including WSL2), use:\n\n```shell\n$ cd $(pyenv root) \u0026\u0026 git pull \u0026\u0026 cd -\nremote: Enumerating objects: 19021, done.\n...\n$\n```\n\nThe update should now succeed:\n\n```shell\n$ pyenv install 3.11.8\nDownloading Python-3.11.8.tar.xz...\n-\u003e https://www.python.org/ftp/python/3.11.8/Python-3.11.8.tar.xz\nInstalling Python-3.11.8...\n...\n$ pyenv local 3.11.8\n```\n\n\n\n#### Synthesis Project\n\nThe `ai8x-synthesis` project does not require hardware acceleration.\n\nStart by deactivating the `ai8x-training` environment if it is active.\n\n```shell\n(ai8x-training) $ deactivate\n```\n\nThen, create a second virtual environment:\n\n```shell\n$ cd \u003cyour/project\u003e\n$ cd ai8x-synthesis\n```\n\nIf you want to use the main branch, switch to “main” using the optional command `git checkout main`.\n\nIf using pyenv, run:\n\n```shell\n$ pyenv local 3.11.8\n```\n\nIn all cases, make sure Python 3.11.x is the active version:\n\n```shell\n$ python --version\nPython 3.11.8\n```\n\nIf this does \u003cu\u003e*not*\u003c/u\u003e return version 3.11.x, please install and initialize [pyenv](#python-311).\n\nThen continue:\n\n```shell\n$ python -m venv .venv --prompt ai8x-synthesis\n$ echo \"*\" \u003e .venv/.gitignore\n```\n\nActivate the virtual environment. On macOS and Linux (including WSL2), use\n\n```shell\n$ source .venv/bin/activate\n```\n\nOn native Windows, instead use\n\n```shell\n$ source .venv/Scripts/activate\n```\n\nFor all systems, continue with:\n\n```shell\n(ai8x-synthesis) $ pip3 install -U pip setuptools\n(ai8x-synthesis) $ pip3 install -r requirements.txt\n```\n\n##### Repository Branches and Updates\n\nBranches and updates for `ai8x-synthesis` are handled similarly to the [`ai8x-training`](#repository-branches) project.\n\n#### Installation is now Complete\n\nWith the installation of Training and Synthesis projects completed it is important to remember to activate the proper Python virtual environment when switching between projects. If scripts begin failing in a previously working environment, the cause might be that the incorrect virtual environment is active or that no virtual environment has been activated.\n\n\n\n### Embedded Software Development Kit (MSDK)\n\nThe Software Development Kit (MSDK) for MAX78000 and MAX78002 is used to compile, flash, and debug the output of the *ai8x-synthesis* (“izer”) tool. It also enables general software development for the microcontroller cores of the MAX78000 and MAX78002. It consists of the following components:\n\n* Peripheral Drivers\n* Board Support Packages (BSPs)\n* Libraries\n* Examples\n* Toolchain\n * Arm GCC\n * RISC-V GCC\n * Make\n * OpenOCD\n\nThere are two ways to install the MSDK.\n\n#### Method 1: MSDK Installer\n\nAn automatic installer is available for the MSDK. Instructions for downloading, installing, and getting started with the MSDK’s supported development environments are found in the [**MSDK User Guide**](https://analogdevicesinc.github.io/msdk/USERGUIDE/).\n\nAfter installation and setup, continue with the [Final Check](#final-check).\n\n#### Method 2: Manual Installation\n\nThe MSDK is also available as a [git repository](https://github.com/analogdevicesinc/msdk), which can be used to obtain the latest development resources. The repository contains all of the MSDK’s components _except_ the Arm GCC, RISC-V GCC, and Make. These can be downloaded and installed manually.\n\n1. Clone the MSDK repository (recommendation: change to the *ai8x-synthesis* folder first):\n\n ```shell\n $ git clone https://github.com/analogdevicesinc/msdk.git sdk\n ```\n\n2. Download and install the Arm Embedded GNU Toolchain from [https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads](https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads).\n\n * Recommended version: 12.3.Rel1 *(newer versions may or may not work correctly)*\n * Recommended installation location: `/usr/local/arm-gnu-toolchain-12.3.rel1/`\n\n3. Download and install the RISC-V Embedded GNU Toolchain from [https://github.com/xpack-dev-tools/riscv-none-elf-gcc-xpack/releases](https://github.com/xpack-dev-tools/riscv-none-elf-gcc-xpack/releases/)\n\n * Recommended version: 12.3.0-2 *(newer versions may or may not work correctly)*\n * Recommended installation location: `/usr/local/xpack-riscv-none-elf-gcc-12.3.0-2/`\n\n4. Install GNU Make\n\n * (Linux/macOS) “make” is available on most systems by default. If not, it can be installed via the system package manager.\n\n * (Windows) Install [MSYS2](https://www.msys2.org/) first, then install “make” using the MSYS2 package manager:\n\n ```shell\n $ pacman -S --needed base filesystem msys2-runtime make\n ```\n\n5. Install packages for OpenOCD. OpenOCD binaries are available in the “openocd” sub-folder of the ai8x-synthesis repository. However, some additional dependencies are required on most systems. See [openocd/README.md](https://github.com/analogdevicesinc/ai8x-synthesis/blob/develop/openocd/README.md) for a list of packages to install, then return here to continue.\n\n6. Add the location of the toolchain binaries to the system path.\n\n On Linux and macOS, copy the following contents into `~/.profile`...\n On macOS, _also_ copy the following contents into `~/.zprofile`...\n ...adjusting for the actual `PATH` to the compilers and the system’s architecture (`TARGET_ARCH`):\n\n ```shell\n # Arm GCC\n ARMGCC_DIR=/usr/local/gcc-arm-none-eabi-10.3-2021.10 # Change me!\n echo $PATH | grep -q -s \"$ARMGCC_DIR/bin\"\n if [ $? -eq 1 ] ; then\n PATH=$PATH:\"$ARMGCC_DIR/bin\"\n export PATH\n export ARMGCC_DIR\n fi\n \n # RISC-V GCC\n RISCVGCC_DIR=/usr/local/xpack-riscv-none-elf-gcc-12.3.0-2 # Change me!\n echo $PATH | grep -q -s \"$RISCVGCC_DIR/bin\"\n if [ $? -eq 1 ] ; then\n     PATH=$PATH:\"$RISCVGCC_DIR/bin\"\n     export PATH\n     export RISCVGCC_DIR\n fi\n \n # OpenOCD\n OPENOCD_DIR=$HOME/Documents/Source/ai8x-synthesis/openocd/bin/TARGET_ARCH # Change me!\n echo $PATH | grep -q -s \"$OPENOCD_DIR\"\n if [ $? -eq 1 ] ; then\n     PATH=$PATH:$OPENOCD_DIR\n     export PATH\n     export OPENOCD_DIR\n fi\n ```\n\n On Windows, add the toolchain paths to the system `PATH` variable (search for “edit the system environment variables” in the Windows search bar).\n\nOnce the tools above have been installed, continue to the [Final Check](#final-check) step below.\n\n#### Final Check\n\nAfter a successful manual or MSDK installation, the following commands will run from on the terminal and display their version numbers:\n\n* `arm-none-eabi-gcc -v`\n* `arm-none-eabi-gdb -v`\n* `make -v`\n* `openocd -v`\n\n`gen-demos-max78000.sh` and `gen-demos-max78002.sh` will create code that is compatible with the MSDK and copy it into the MSDK’s Example directories.\n\n---\n\n\n\n## MAX78000 and MAX78002 Hardware and Resources\n\nMAX78000/MAX78002 are embedded accelerators. Unlike GPUs, MAX78000/MAX78002 do not have gigabytes of memory, and cannot support arbitrary data (image) sizes.\n\n### Overview\n\nA typical CNN operation consists of pooling followed by a convolution. While these are traditionally expressed as separate layers, pooling can be done “in-flight” on MAX78000/MAX78002 for greater efficiency.\n\nTo minimize data movement, the accelerator is optimized for convolutions with in-flight pooling on a sequence of layers. MAX78000 and MAX78002 also support in-flight element-wise operations, pass-through layers and 1D convolutions (without element-wise operations):\n\n![CNNInFlight](docs/CNNInFlight.png)\n\nThe MAX78000/MAX78002 accelerators contain 64 parallel processors. There are four quadrants that contain 16 processors each.\n\nEach processor includes a pooling unit and a convolutional engine with dedicated weight memory:\n\n![Overview](docs/Overview.png)\n\nData is read from [data memory](#data-memory) associated with the processor, and written out to any data memory located within the accelerator. To run a deep convolutional neural network, multiple layers are chained together, where each layer’s operation is individually configurable. The output data from one layer is used as the input data for the next layer, for up to 32 layers (where *in-flight* pooling and *in-flight* element-wise operations do not count as layers).\n\nThe following picture shows an example view of a 2D convolution with pooling:\n![Example](docs/CNNOverview.png)\n\n### Data, Weights, and Processors\n\nData memory, weight memory, and processors are interdependent.\n\nIn the MAX78000/MAX78002 accelerator, processors are organized as follows:\n\n* Each processor is connected to its own dedicated weight memory instance.\n* A group of four processors shares one data memory instance.\n* A quadrant of sixteen processors shares certain common controls and can be operated tethered to another quadrant, or independently/separately.\n\nAny given processor can:\n\n* Read from its dedicated weight memory,\n* Read from the data memory instance it shares with three other processors, and\n* As part of output processing, write to *any* data memory instance.\n\n#### Weight Memory\n\n*Note: Depending on context, weights may also be referred to as “kernels” or “masks”. Additionally, weights are also part of a network’s “parameters”.*\n\nFor each of the four 16-processor quadrants, weight memory and processors can be visualized as follows. Assuming one input channel processed by processor 0, and 8 output channels, the 8 shaded kernels will be used:\n\n![Weight Memory Map](docs/KernelMemory.png)\n\n*Note: Weights that are not 3×3×8 (= 72-bits) per kernel are packed to save space. For example, when using 1×1 8-bit kernels, 9 kernels will be packed into a single 72-bit memory word.*\n\n#### Data Memory\n\nData memory in MAX78000/MAX78002 is needed for:\n\n* Input data (unless [FIFOs](#fifos) are used),\n* All layer’s activation data, and\n* Inference outputs.\n\nData memory connections can be visualized as follows:\n\n\u003cimg src=\"docs/DataMemory.png\" alt=\"Data Memory Map\" style=\"zoom: 67%;\" /\u003e\n\nAll input data must be located in the data memory instance the processor can access. Conversely, output data can be written to any data memory instance inside the accelerator (but not to general purpose SRAM on the Arm microcontroller bus).\n\nThe data memory instances inside the accelerator are single-port memories. This means that only one access operation can happen per clock cycle. When using the HWC data format (see [Channel Data Formats](#channel-data-formats)), this means that each of the four processors sharing the data memory instance will receive one byte of data per clock cycle (since each 32-bit data word consists of four packed channels).\n\n##### Multi-Pass\n\nWhen data has more channels than active processors, “multi-pass” is used. Each processor works on more than one channel, using multiple sequential passes, and each data memory holds more than four channels.\n\nAs data is read using multiple passes, and all available processor work in parallel, the first pass reads channels 0 through 63, the second pass reads channels 64 through 127, etc., assuming 64 processors are active.\n\nFor example, if 192-channel data is read using 64 active processors, Data Memory 0 stores three 32-bit words: channels 0, 1, 2, 3 in the first word, 64, 65, 66, 67 in the second word, and 128, 129, 130, 131 in the third word. Data Memory 1 stores channels 4, 5, 6, 7 in the first word, 68, 69, 70, 71 in the second word, and 132, 133, 134, 135 in the third word, and so on. The first processor processes channel 0 in the first pass, channel 64 in the second pass, and channel 128 in the third pass.\n\n*Note: Multi-pass also works with channel counts that are not a multiple of 64, and can be used with fewer than 64 active processors.*\n\n*Note: For all multi-pass cases, the processor count per pass is rounded up to the next multiple of 4.*\n\n### Streaming Mode\n\nThe machine also implements a streaming mode. Streaming allows input data dimensions that exceed the available per-channel data memory in the accelerator. *Note: Depending on the model and application, [Data Folding](#data-folding) may have performance benefits over Streaming Mode.*\n\nThe following illustration shows the basic principle: In order to produce the first output pixel of the second layer, not all data needs to be present at the input. In the example, a 5×5 input needs to be available.\n\n\u003cimg src=\"docs/Streaming.png\" alt=\"Illustration of Streaming Mode\"/\u003e\n\nIn the accelerator implementation, data is shifted into the Tornado memory in a sequential fashion, so prior rows will be available as well. In order to produce the _blue_ output pixel, input data up to the blue input pixel must be available.\n\n![Streaming-Rows](docs/Streaming-Rows.png)\n\nWhen the _yellow_ output pixel is produced, the first (_black_) pixel of the input data is no longer needed and its data can be discarded:\n\n![Streaming-NextPixel](docs/Streaming-NextPixel.png)\n\nThe number of discarded pixels is network specific and dependent on pooling strides and the types of convolution. In general, streaming mode is only useful for networks where the output data dimensions decrease from layer to layer (for example, by using a pooling stride).\n\n*Note: Streaming mode requires the use of [FIFOs](#fifos).*\n\nFor concrete examples on how to implement streaming mode with a camera, please see the [Camera Streaming Guide](https://github.com/analogdevicesinc/MaximAI_Documentation/blob/main/Guides/Camera_Streaming_Guide.md).\n\n#### FIFOs\n\nSince the data memory instances are single-port memories, software would have to temporarily disable the accelerator in order to feed it new data. Using FIFOs, software can input available data while the accelerator is running. The accelerator will autonomously fetch data from the FIFOs when needed, and stall (pause) when no enough data is available.\n\nThe MAX78000/MAX78002 accelerator has two types of FIFO:\n\n##### Standard FIFOs\n\nThere are four dedicated FIFOs connected to processors 0-3, 16-19, 32-35, and 48-51, supporting up to 16 input channels (in HWC format) or four channels (CHW format). These FIFOs work when used from the Arm Cortex-M4 core and from the RISC-V core.\n\nThe standard FIFOs are selected using the `--fifo` argument for `ai8xize.py`.\n\n##### Fast FIFO\n\nThe fast FIFO is only available from the RISC-V core, and runs synchronously with the RISC-V for increased throughput. It supports up to four input channels (HWC format) or a single channel (CHW format). The fast FIFO is connected to processors 0, 1, 2, 3 or 0, 16, 32, 48.\n\nThe fast FIFO is selected using the `--fast-fifo` argument for `ai8xize.py`.\n\n*The code generator inserts FIFO-full checks for either type of FIFO. When the data source rate is equal to or slower than the network speed, these checks are not needed. Use `--no-fifo-wait` to suppress them. The checks are necessary when the data source can deliver faster than the network can process the data.*\n\n### Number Format\n\nAll weights, bias values and data are stored and computed in Q7 format (signed two’s complement 8-bit integers, [–128...+127]). See \u003chttps://en.wikipedia.org/wiki/Q_%28number_format%29\u003e.\n\nThe 8-bit value $w$ is defined as:\n\n$$ w = (-a_7 2^7+a_6 2^6+a_5 2^5+a_4 2^4+a_3 2^3+a_2 2^2+a_1 2^1+a_0)/128 $$\n\n\u003cimg src=\"docs/Numbers.png\" alt=\"76543210\" style=\"zoom:67%;\" /\u003e\n\nExamples:\n| Binary | Value |\n|:---------:|-------------:|\n| 0000 0000 | 0 |\n| 0000 0001 | 1/128 |\n| 0000 0010 | 2/128 |\n| 0111 1110 | 126/128 |\n| 0111 1111 | 127/128 |\n| 1000 0000 | −128/128 (–1)|\n| 1000 0001 | −127/128 |\n| 1000 0010 | −126/128 |\n| 1111 1110 | −2/128 |\n| 1111 1111 | −1/128 |\n\nOn MAX78000/MAX78002, _weights_ can be 1, 2, 4, or 8 bits wide (configurable per layer using the `quantization` key). Bias values are always 8 bits wide. Data is 8 bits wide, *except for the last layer that can optionally output 32 bits of unclipped data in Q17.14 format when not using activation.*\n\n| weight bits | min | max |\n| :---------------------------: | ---: | ---: |\n| 8 | –128 | +127 |\n| 4 | –8 | 7 |\n| 2 | –2 | 1 |\n| 1 | –1 | 0 |\n| *MAX78002 only* 1 | –1 | +1 |\n\nNote that for –1/0 1-bit weights (and, to a lesser degree, –1/+1 1-bit weights and 2-bit weights) require the use of bias to produce useful results. Without bias, all sums of products of activated data from a prior layer would be negative, and activation of that data would always be zero.\n\nIn other cases, using bias in convolutional layers does not improve inference performance. In particular, [Quantization](#quantization)-Aware Training (QAT) optimizes the weight distribution, possibly deteriorating the distribution of the bias values.\n\n#### Rounding\n\nMAX78000/MAX78002 rounding (for the CNN sum of products) uses “round half towards positive infinity”, i.e. $y=⌊0.5+x⌋$. This rounding method is not the default method in either Excel or Python/NumPy. The rounding method can be achieved in NumPy using `y = np.floor(0.5 + x)` and in Excel as `=FLOOR.PRECISE(0.5 + X)`.\n\nBy way of example:\n\n| Input | Rounded |\n|:-------------------------|:-------:|\n| +3.5 | +4 |\n| +3.25, +3.0, +2.75, +2.5 | +3 |\n| +2.25, +2.0, +1.75, +1.5 | +2 |\n| +1.25, +1.0, +0.75, +0.5 | +1 |\n| +0.25, 0, –0.25, –0.5 | 0 |\n| –0.75, –1.0, –1.25, –1.5 | –1 |\n| –1.75, –2.0, –2.25, –2.5 | –2 |\n| –2.75, –3.0, –3.25, –3.5 | –3 |\n\n#### Addition\n\nAddition works similarly to regular two’s-complement arithmetic.\n\nExample:\n$$ w_0 = 1/64 → 00000010 $$\n$$ w_1 = 1/2 → 01000000 $$\n$$ w_0 + w_1 = 33/64 → 01000010 $$\n\n#### Saturation and Clipping\n\nValues smaller than $–128⁄128$ are saturated to $–128⁄128$ (1000 0000). Values larger than $+127⁄128$ are saturated to $+127⁄128$ (0111 1111).\n\nThe MAX78000/MAX78002 CNN sum of products uses full resolution for both products and sums, so the saturation happens only at the very end of the computation.\n\nExample 1:\n\n$$ w_0 = 127/128 → 01111111 $$\n$$ w_1 = 127/128 → 01111111 $$\n$$ w_0 + w_1 = 254/128 → saturate → 01111111 (= 127/128) $$\n\nExample 2:\n\n$$ w_0 = -128/128 → 10000000 $$\n$$ w_1 = -128/128 → 10000000 $$\n$$ w_0 + w_1 = -256/128 → saturate → 10000000 (= -128/128) $$\n\n#### Multiplication\n\nSince operand values are implicitly divided by 128, the product of two values has to be shifted in order to maintain magnitude when using a standard multiplier (e.g., 8×8):\n\n$$ w_0 * w_1 = \\frac{w'_0}{128} * \\frac{w'_1}{128} = \\frac{w'_0 * w'_1}{128} ≫ 7 $$\n\nIn software,\n\n* Determine the sign bit: $s = sign(w_0) * sign(w_1)$\n* Convert operands to absolute values: $w'_0 = abs(w_0); w'_1 = abs(w_1)$\n* Multiply using standard multiplier: $w'_0 * w'_1 = w''_0/128 * w''_1/128; r' = w''_0 * w''_1$\n* Shift: $r'' = r' ≫ 7$\n* Round up/down depending on $r'[6]$\n* Apply sign: $r = s * r''$\n\nExample 1:\n\n$$ w_0 = 1/64 → 00000010 $$\n$$ w_1 = 1/2 → 01000000 $$\n$$ w_0 * w_1 = 1/128 → shift, truncate → 00000001 (= 1/128) $$\n\nA “standard” two’s-complement multiplication would return 00000000 10000000. The MAX78000/MAX78002 data format discards the rightmost bits.\n\nExample 2:\n\n$$ w_0 = 1/64 → 00000010 $$\n$$ w_1 = 1/4 → 00100000 $$\n$$ w_0 * w_1 = 1/256 → shift, truncate → 00000000 (= 0) $$\n\n“Standard” two’s-complement multiplication would return 00000000 01000000, the MAX78000/MAX78002 result is truncated to 0 after the shift operation.\n\n#### Sign Bit\n\nOperations preserve the sign bit.\n\nExample 1:\n\n$$ w_0 = -1/64 → 11111110 $$\n$$ w_1 = 1/4 → 00100000 $$\n$$ w_0 * w_1 = -1/256 → shift, truncate → 00000000 (= 0) $$\n\n* Determine the sign bit: $s = sign(-1/64) * sign(1/4) = -1 * 1 = -1$\n* Convert operands to absolute values: $w'_0 = abs(-1/64); w'_1 = abs(1/4)$\n* Multiply using standard multiplier: $r' = 1/64 ≪ 7 * 1/4 ≪ 7 = 2 * 32 = 64$\n* Shift: $r'' = r' ≫ 7 = 64 ≫ 7 = 0$\n* Apply sign: $r = s * r'' = -1 * 0 = 0$\n\nExample 2:\n\n$$ w_0 = -1/64 → 11111110 $$\n$$ w_1 = 1/2 → 01000000 $$\n$$ w_0 * w_1 = -1/128 → shift, truncate → 11111111 (= -1/128) $$\n\n* Determine the sign bit: $s = sign(-1/64) * sign(1/2) = -1 * 1 = -1$\n* Convert operands to absolute values: $w'_0 = abs(-1/64); w'_1 = abs(1/2)$\n* Multiply using standard multiplier: $r' = 1/64 ≪ 7 * 1/2 ≪ 7 = 2 * 64 = 128$\n* Shift: $r'' = r' ≫ 7 = 128 ≫ 7 = 1$\n* Apply sign: $r = s * r'' = -1 * 1 ≫ 7 = -1/128$\n\nExample 3:\n\n$$ w_0 = 127/128 → 01111111 $$\n$$ w_1 = 1/128 → 00000001 $$\n$$ w_0 * w_1 = 128/128 → saturation → 01111111 (= 127/128) $$\n\n### Channel Data Formats\n\n#### HWC (Height-Width-Channels)\n\nAll internal data are stored in HWC format, four channels per 32-bit word. Assuming 3-color (or 3-channel) input, one byte of the 32-bit word will be unused. The highest frequency in this data format is the channel, so the channels are interleaved.\n\nExample:\n\n![0BGR 0BGR 0 BGR 0BGR...](docs/HWC.png)\n\n#### CHW (Channels-Height-Width)\n\nThe input layer (and *only* the input layer) can alternatively also use the CHW format (a sequence of channels). The highest frequency in this data format is the width W or X-axis, and the lowest frequency is the channel C. Assuming an RGB input, all red pixels are followed by all green pixels, followed by all blue pixels.\n\nExample:\n\n![RRRRRR...GGGGGG...BBBBBB...](docs/CHW.png)\n\n\n\n#### Considerations for Choosing an Input Format\n\nThe accelerator supports both HWC and CHW input formats to avoid unnecessary data manipulation. Choose the format that results in the least amount of data movement for a given input.\n\nInternal layers and the output layer always use the HWC format.\n\nIn general, HWC is faster since each memory read can deliver data to up to four processors in parallel. On the other hand, four processors must share one data memory instance, which reduces the maximum allowable dimensions of the input layer.\n\n#### CHW Input Data Format and Consequences for Weight Memory Layout\n\nWhen using the CHW data format, only one of the four processors sharing the data memory instance can be used. The next channel needs to use a processor connected to a different data memory instance, so that the machine can deliver one byte per clock cycle to each enabled processor.\n\nBecause each processor has its own dedicated weight memory, this will introduce “gaps” in the weight memory map, as shown in the following illustration:\n\n![Kernel Memory Gaps](docs/KernelMemoryGaps.png)\n\n\n### Active Processors and Layers\n\nFor each layer, a set of active processors must be specified. The number of input channels for the layer must be equal to, or be a multiple of, the active processors, and the input data for that layer must be located in data memory instances accessible to the selected processors.\n\nIt is possible to specify a relative offset into the data memory instance that applies to all processors.\n_Example:_ Assuming HWC data format, specifying the offset as 16,384 bytes (or 0x4000) will cause processors 0-3 to read their input from the second half of data memory 0, processors 4-7 will read from the second half of data memory instance 1, etc.\n\nFor most simple networks with limited data sizes, it is easiest to ping-pong between the first and second halves of the data memories – specify the data offset as 0 for the first layer, 0x4000 for the second layer, 0 for the third layer, etc. This strategy avoids overlapping inputs and outputs when a given processor is used in two consecutive layers.\n\nEven though it is supported by the accelerator, the Network Generator will not be able to check for inadvertent overwriting of unprocessed input data by newly generated output data when overlapping data or streaming data. Use the `--overlap-data` command line switch to disable these checks, and to allow overlapped data.\n\n### Layers and Weight Memory\n\nFor each layer, the weight memory start column is automatically configured by the Network Loader. The start column must be a multiple of 4, and the value applies to all processors.\n\nThe following example shows the weight memory layout for two layers. The first layer (L0) has 8 inputs and 10 outputs, and the second layer (L1) has 10 inputs and 2 outputs.\n\n![Layers and Weight Memory](docs/KernelMemoryLayers.png)\n\n#### Bias Memories\n\nBias values are stored in separate bias memories. There are four bias memory instances available, and a layer can access any bias memory instance where at least one processor is enabled. By default, bias memories are automatically allocated using a modified Fit-First Descending (FFD) algorithm. Before considering the required resource sizes in descending order, and placing values in the bias memory with the most available resources, the algorithm places those bias values that require a single specified bias memory. The bias memory allocation can optionally be controlled using the [`bias_group`](#bias_group-optional) configuration option.\n\n### Weight Storage Example\n\nThe file `ai84net.xlsx` contains an example for a single-channel CHW input using the `AI84Net5` network (this example also supports up to four channels in HWC).\n\n*Note*: As described above, multiple CHW channels must be loaded into separate memory instances. When using a large number of channels, this can cause “holes” in the processor map, which in turn can cause subsequent layers’ kernels to require padding.\n\nThe Network Loader prints a kernel map that shows the kernel arrangement based on the provided network description. It will also flag cases where kernel or bias memories are exceeded.\n\n### Example: `Conv2d`\n\nThe following picture shows an example of a `Conv2d` with 1×1 kernels, five input channels, two output channels, and a data size of 2×2. The inputs are shown on the left, and the outputs on the right, and the kernels are shown lined up with the associated inputs — the number of kernel rows matches the number of input channels, and the number of kernel columns matches the number of output channels. The lower half of the picture shows how the data is arranged in memory when HWC data is used for both input and output.\n\n![Conv2Dk1x1](docs/Conv2Dk1x1.png)\n\n### Activation Functions\n\nMAX78000/MAX78002 hardware provides several activation functions.\n\n#### None\n\nThere is always an implicit non-linearity when outputting 8-bit data since outputs are [clamped](#saturation-and-clipping) to $[–128, +127]$ (or $[–128/128, +127/128]$ during training). Due to the clamping, “no activation” behaves similar to PyTorch’s `nn.Hardtanh(min_value=-128[/128], max_value=127[/128])`.\n\n\u003cimg src=\"docs/noactivation.png\" alt=\"no activation\" style=\"zoom:33%;\" /\u003e\n\n#### ReLU\n\nAll output values are [clipped (saturated)](#saturation-and-clipping) to $[0, +127]$. Because of this, `ReLU` behaves more similar to PyTorch’s `nn.Hardtanh(min_value=0, max_value=127[/128])` than to PyTorch’s `nn.ReLU()`.\n\n\u003cimg src=\"docs/relu.png\" alt=\"relu\" style=\"zoom:33%;\" /\u003e\n\n#### Abs\n\n`Abs` returns the absolute value for all inputs, and then [clamps](#saturation-and-clipping) the outputs to $[0, +127]$, similar to PyTorch `abs()` followed by `nn.Hardtanh(min_value=0, max_value=127[/128])`.\n\n\u003cimg src=\"docs/abs.png\" alt=\"abs\" style=\"zoom:33%;\" /\u003e\n\n### Limitations of MAX78000 Networks\n\nThe MAX78000 hardware does not support arbitrary network parameters. Specifically,\n\n* `Conv2d`:\n \n * Kernel sizes must be 1×1 or 3×3.\n *Note: Stacked 3×3 kernels can achieve the effect of larger kernels. For example, two consecutive layers with 3×3 kernels have the same receptive field as a 5×5 kernel. To achieve the same activation as a 5×5 kernel, additional layers are necessary.*\n *Note: 2×2 kernels can be emulated by setting one row and one column of 3×3 kernels to zero.*\n * Padding can be 0, 1, or 2. Padding always uses zeros.\n * Stride is fixed to [1, 1].\n * Dilation is fixed to 1.\n * Groups must be 1.\n \n* `Conv1d`:\n \n * Kernel lengths must be 1 through 9.\n * Padding can be 0, 1, or 2.\n * Stride is fixed to 1.\n * Dilation can be 1 to 1023 for kernel lengths 1, 2, or 3 and is fixed to 1 for kernels with length greater than 3.\n \n* `ConvTranspose2d`:\n\n * Kernel sizes must be 3×3.\n * Padding can be 0, 1, or 2.\n * Stride is fixed to [2, 2]. Output padding is fixed to 1.\n\n* A programmable layer-specific shift operator is available at the output of a convolution, see [`output_shift` (Optional)](#output_shift-optional).\n\n* The supported [activation functions](#activation-functions) are `ReLU` and `Abs`, and a limited subset of `Linear`. *Note that due to [clipping](#saturation-and-clipping), non-linearities are introduced even when not explicitly specifying an activation function.*\n\n* Pooling:\n * Both max pooling and average pooling are available, with or without convolution.\n \n * Pooling does not support padding.\n \n * Pooling more than 64 channels requires the use of a “fused” convolution in the same layer, unless the pooled dimensions are 1×1.\n \n * Pooling strides can be 1 through 16. For 2D pooling, the stride is the same for both dimensions.\n \n * For 2D pooling, supported pooling kernel sizes are 1×1 through 16×16, including non-square kernels. 1D pooling supports kernel sizes from 1 through 16. *Note: Pooling kernel size values do not have to be the same as the pooling stride.*\n \n * Dilation must be 1.\n \n * Average pooling is implemented both using `floor()`and using rounding (half towards positive infinity). Use the `--avg-pool-rounding` switch to turn on rounding in the training software and the Network Generator.\n \n Example:\n \n * _floor:_ Since there is a quantization step at the output of the average pooling, a 2×2 `AvgPool2d` of `[[0, 0], [0, 3]]` will return $\\lfloor \\frac{3}{4} \\rfloor = 0$.\n * _rounding:_ 2×2 `AvgPool2d` of `[[0, 0], [0, 3]]` will return $\\lfloor \\frac{3}{4} \\rceil = 1$.\n \n* The number of input channels must not exceed 1024 per layer.\n\n* The number of output channels must not exceed 1024 per layer.\n\n * Bias is supported for up to 512 output channels per layer.\n\n* The number of layers must not exceed 32 (where pooling and element-wise operations do not add to the count when preceding a convolution).\n\n* The maximum dimension (number of rows or columns) for input or output data is 1023.\n \n* Streaming mode:\n \n * When using data greater than 8192 pixels per channel (approximately 90×90 when width = height) in HWC mode, or 32,768 pixels per channel (181×181 when width = height) in CHW mode, and [Data Folding](#data-folding) techniques are not used, then `streaming` mode must be used.\n * When using `streaming` mode, the product of any layer’s input width, input height, and input channels divided by 64 rounded up must not exceed 2^21: $width * height * ⌈\\frac{channels}{64}⌉ \u003c 2^{21}$; _width_ and _height_ must not exceed 1023.\n * Streaming is limited to 8 consecutive layers or fewer, and is limited to four FIFOs (up to 4 input channels in CHW and up to 16 channels in HWC format), see [FIFOs](#fifos).\n * For streaming layers, bias values may not be added correctly in all cases.\n * The *final* streaming layer must use padding.\n * Layers that use 1×1 kernels without padding are automatically replaced with equivalent layers that use 3×3 kernels with padding.\n \n* The weight memory supports up to 768 * 64 3×3 Q7 kernels (see [Number Format](#number-format)), for a total of [432 KiB of kernel memory](https://github.com/analogdevicesinc/ai8x-synthesis/blob/develop/docs/AHBAddresses.md).\n When using 1-, 2- or 4-bit weights, the capacity increases accordingly.\n When using more than 64 input or output channels, weight memory is shared, and effective capacity decreases proportionally (for example, 128 input channels require twice as much space as 64 input channels, and a layer with \u003cu\u003eboth\u003c/u\u003e 128 input and 128 output channels requires \u003cu\u003efour\u003c/u\u003e times as much space as a layer with only 64 input channels and 64 output channels).\n Weights must be arranged according to specific rules detailed in [Layers and Weight Memory](#layers-and-weight-memory).\n\n* There are 16 instances of 32 KiB data memory ([for a total of 512 KiB](https://github.com/analogdevicesinc/ai8x-synthesis/blob/develop/docs/AHBAddresses.md)). When not using streaming mode, any data channel (input, intermediate, or output) must completely fit into one memory instance. This limits the first-layer input to 32,768 pixels per channel in the CHW format (181×181 when width = height). However, when using more than one input channel, the HWC format may be preferred, and all layer outputs are in HWC format as well. In those cases, it is required that four channels fit into a single memory instance — or 8192 pixels per channel (approximately 90×90 when width = height).\n Note that the first layer commonly creates a wide expansion (i.e., a large number of output channels) that needs to fit into data memory, so the input size limit is mostly theoretical. In many cases, [Data Folding](#data-folding) (distributing the input data across multiple channels) can effectively increase both the input dimensions as well as improve model performance.\n\n* The hardware supports 1D and 2D convolution layers, 2D transposed convolution layers (upsampling), element-wise addition, subtraction, binary OR, binary XOR as well as fully connected layers (`Linear`), which are implemented using 1×1 convolutions on 1×1 data:\n * The maximum number of input neurons is 1024, and the maximum number of output neurons is 1024 (16 each per processor used).\n \n * `Flatten` functionality is available to convert 2D input data for use by fully connected layers, see [Fully Connected Layers](#fully-connected-linear-layers).\n \n * When “flattening” two-dimensional data, the input dimensions (C×H×W) must satisfy C×H×W ≤ 16,384, and H×W ≤ 256. Pooling cannot be used at the same time as flattening.\n \n * Element-wise operators support from 2 up to 16 inputs.\n \n * Element-wise operators can be chained in-flight with pooling and 2D convolution (where the order of pooling and element-wise operations can be swapped).\n \n * For convenience, a `Softmax` operator is supported in software.\n \n* Since the internal network format is HWC in groups of four channels, output concatenation only works properly when all components of the concatenation other than the last have multiples of four channels.\n\n* Supported element-wise operations are `add`, `sub`, `bitwise xor`, and `bitwise or`. Element-wise operations can happen “in-flight” in the same layer as a convolution.\n\n* Groups, and depthwise separable convolutions are not supported. *Note: Batch normalization should be folded into the weights, see [Batch Normalization](#batch-normalization).*\n\n\n\n### Limitations of MAX78002 Networks\n\nThe MAX78002 hardware does not support arbitrary network parameters. Specifically,\n\n* `Conv2d`:\n\n * Kernel sizes must be 1×1 or 3×3.\n *Note: Stacked 3×3 kernels can achieve the effect of larger kernels. For example, two consecutive layers with 3×3 kernels have the same receptive field as a 5×5 kernel. To achieve the same activation as a 5×5 kernel, additional layers are necessary.*\n *Note: 2×2 kernels can be emulated by setting one row and one column of 3×3 kernels to zero.*\n * Padding can be 0, 1, or 2. Padding always uses zeros.\n * Stride is fixed to [1, 1].\n * Dilation can be 1 to 16.\n * Groups can be 1, or the same as the number of input and output channels (depthwise separable convolution).\n\n* `Conv1d`:\n\n * Kernel lengths must be 1 through 9.\n * Padding can be 0, 1, or 2, unless there are more than 64 input channels, when padding must be 0.\n * Stride is fixed to 1.\n * Dilation can be 1 to 2047 for kernel lengths 1, 2, or 3 and is fixed to 1 for kernels with length greater than 3.\n * Groups can be 1, or the same as the number of input and output channels (depthwise separable convolution).\n\n* `ConvTranspose2d`:\n\n * Kernel sizes must be 3×3.\n * Padding can be 0, 1, or 2.\n * Stride is fixed to [2, 2]. Output padding is fixed to 1.\n\n* A programmable layer-specific shift operator is available at the output of a convolution, see [`output_shift` (Optional)](#output_shift-optional).\n\n* The supported [activation functions](#activation-functions) are `ReLU` and `Abs`, and a limited subset of `Linear`. *Note that due to [clipping](#saturation-and-clipping), non-linearities are introduced even when not explicitly specifying an activation function.*\n\n* Pooling:\n\n * Both max pooling and average pooling are available, with or without convolution.\n\n * Pooling does not support padding.\n\n * Pooling strides can be 1 through 16. For 2D pooling, the stride is the same for both dimensions.\n\n * For 2D pooling, supported pooling kernel sizes are 1×1 through 16×16, including non-square kernels. 1D pooling supports kernel sizes from 1 through 16. *Note: Pooling kernel size values do not have to be the same as the pooling stride.*\n\n * Dilation is supported from 1 to 16, independently for both dimensions.\n\n * Average pooling is implemented both using `floor()`and using rounding (half towards positive infinity). Use the `--avg-pool-rounding` switch to turn on rounding in the training software and the Network Generator.\n\n Example:\n\n * _floor:_ Since there is a quantization step at the output of the average pooling, a 2×2 `AvgPool2d` of `[[0, 0], [0, 3]]` will return $\\lfloor \\frac{3}{4} \\rfloor = 0$.\n * _rounding:_ 2×2 `AvgPool2d` of `[[0, 0], [0, 3]]` will return $\\lfloor \\frac{3}{4} \\rceil = 1$.\n\n* The number of input channels must not exceed 2048 per layer.\n\n* The number of output channels must not exceed 2048 per layer.\n\n* The number of layers must not exceed 128 (where pooling and element-wise operations do not add to the count when preceding a convolution).\n\n* The maximum dimension (number of rows or columns) for input or output data is 2047.\n\n* Streaming mode:\n\n * When using data greater than 20,480 pixels per channel in HWC mode (143×143 when height = width), or 81,920 pixels in CHW mode (286×286 when height = width), and [Data Folding](#data-folding) techniques are not used, then `streaming` mode must be used.\n * When using `streaming` mode, the product of any layer’s input width, input height, and input channels divided by 64 rounded up must not exceed 2^21: $width * height * ⌈\\frac{channels}{64}⌉ \u003c 2^{21}$; _width_ and _height_ must not exceed 2047.\n * Streaming is limited to 8 consecutive layers or fewer, and is limited to four FIFOs (up to 4 input channels in CHW and up to 16 channels in HWC format), see [FIFOs](#fifos).\n * Layers that use 1×1 kernels without padding are automatically replaced with equivalent layers that use 3×3 kernels with padding.\n * Streaming layers must use convolution (i.e., the `Conv1d`, `Conv2d`, or `ConvTranspose2d` [operators](#operation)).\n\n* The weight memory of processors 0, 16, 32, and 48 supports up to 5,120 3×3 Q7 kernels (see [Number Format](#number-format)), all other processors support up to 4,096 3×3 Q7 kernels, for a total of [2,340 KiB of kernel memory](https://github.com/analogdevicesinc/ai8x-synthesis/blob/develop/docs/AHBAddresses.md).\n When using 1-, 2- or 4-bit weights, the capacity increases accordingly. The hardware supports two different flavors of 1-bit weights, either 0/–1 or +1/–1.\n When using more than 64 input or output channels, weight memory is shared, and effective capacity decreases.\n Weights must be arranged according to specific rules detailed in [Layers and Weight Memory](#layers-and-weight-memory).\n\n* The total of [1,280 KiB of data memory](https://github.com/analogdevicesinc/ai8x-synthesis/blob/develop/docs/AHBAddresses.md) is split into 16 sections of 80 KiB each. When not using streaming mode, any data channel (input, intermediate, or output) must completely fit into one memory instance. This limits the first-layer input to 81,920 pixels per channel in CHW format (286×286 when height = width). However, when using more than one input channel, the HWC format may be preferred, and all layer outputs are in HWC format as well. In those cases, it is required that four channels fit into a single memory section — or 20,480 pixels per channel (143×143 when height = width).\n Note that the first layer commonly creates a wide expansion (i.e., a large number of output channels) that needs to fit into data memory, so the input size limit is mostly theoretical. In many cases, [Data Folding](#data-folding) (distributing the input data across multiple channels) can effectively increase both the input dimensions as well as improve model performance.\n\n* The hardware supports 1D and 2D convolution layers, 2D transposed convolution layers (upsampling), element-wise addition, subtraction, binary OR, binary XOR as well as fully connected layers (`Linear`), which are implemented using 1×1 convolutions on 1×1 data:\n\n * The maximum number of input neurons is 1024, and the maximum number of output neurons is 1024 (16 each per processor used).\n * `Flatten` functionality is available to convert 2D input data for use by fully connected layers, see [Fully Connected Layers](#fully-connected-linear-layers).\n * When “flattening” two-dimensional data, the input dimensions (C×H×W) must satisfy C×H×W ≤ 16,384, and H×W ≤ 256. Pooling cannot be used at the same time as flattening.\n * Element-wise operators support from 2 up to 16 inputs.\n * Element-wise operators can be chained in-flight with pooling and 2D convolution (where the order of pooling and element-wise operations can be swapped).\n * For convenience, a `Softmax` operator is supported in software.\n\n* The MAX78002 hardware supports executing layers sequentially or in programmed order, and it supports conditional branching based on data and address values and ranges and match counts.\n\n* The MAX78002 hardware supports starting a network at any pre-programmed layer *(streaming is only supported in the first 8 layers)*. This can be used to run more than one network, and transitioning from one network to another.\n\n* Since the internal network format is HWC in groups of four channels, output concatenation only works properly when all components of the concatenation other than the last have multiples of four channels.\n\n* The MAX78002 hardware supports several processing speedups that accesses memory instances in parallel. The tools are capable of generating code that supports these speedups.\n\n* Supported element-wise operations are `add`, `sub`, `xor`, and `or`. Element-wise operations can happen “in-flight” in the same layer as a convolution, *except* when the input is multi-pass (i.e., more than 64 channels), *and* a bias addition is also requested.\n\n* *Note: Batch normalization should be folded into the weights, see [Batch Normalization](#batch-normalization).*\n\n\n\n### Fully Connected (Linear) Layers\n\nm×n fully connected layers can be realized in hardware by “flattening” 2D input data of dimensions C×H×W into m=C×H×W channels of 1×1 input data. The hardware will produce n channels of 1×1 output data. When chaining multiple fully connected layers, the flattening step is omitted. The following picture shows 2D data, the equivalent flattened 1D data, and the output.\n\nFor MAX78000/MAX78002, the product C×H×W must not exceed 16,384.\n\n![MLP](docs/MLP.png)\n\n### Upsampling (Fractionally-Strided 2D Convolutions)\n\nThe hardware supports 2D upsampling (“fractionally-strided convolutions,” sometimes called “deconvolution” even though this is not strictly mathematically correct). The PyTorch equivalent is `ConvTranspose2d` with a stride of 2.\n\nThe example shows a fractionally-strided convolution with a stride of 2, a pad of 1, and a 3×3 kernel. This “upsamples” the input dimensions from 3×3 to output dimensions of 6×6.\n\n![fractionallystrided](docs/fractionallystrided.png)\n\n---\n\n## Model Training and Quantization\n\n### Hardware Acceleration\n\nIf hardware acceleration is not available, skip the following two steps and continue with [Training Script](#training-script).\n\nBefore the first training session, check that hardware acceleration is available and recognized by PyTorch:\n\n ```shell\n (ai8x-training) $ python check_cuda.py\n System: linux\n Python version: 3.11.8 (main, Mar 4 2024, 15:29:36) [GCC 11.4.0]\n PyTorch version: 2.3.1+cu121\n CUDA/ROCm acceleration: available in PyTorch\n MPS acceleration: NOT available in PyTorch\n ```\n\nCUDA can be diagnosed using `nvidia-smi -q`:\n\n```shell\n(ai8x-training) $ nvidia-smi -q\n...\nDriver Version : 545.23.06\nCUDA Version : 12.3\n\nAttached GPUs : 2\nGPU 00000000:01:00.0\n Product Name : NVIDIA TITAN RTX\n Product Brand : Titan\n...\n```\n\n### Training Script\n\nThe main training software is `train.py`. It drives the training aspects, including model creation, checkpointing, model save, and status display (see `--help` for the many supported options, and the `scripts/train_*.sh` scripts for example usage).\n\nThe `models/` folder contains models that fit into the MAX78000 or MAX78002’s weight memory. These models rely on the MAX78000/MAX78002 hardware operators that are defined in `ai8x.py`.\n\nTo train the FP32 model for MNIST on MAX78000 or MAX78002, run `scripts/train_mnist.sh` from the `ai8x-training` project. This script will place checkpoint files into the log directory. Training makes use of the Distiller framework, but the `train.py` software has been modified slightly to improve it and add some MAX78000/MAX78002 specifics.\n\n#### Distributed Training\n\nOn systems with multiple GPUs, the training script supports `DistributedDataParallel`. To use distributed training, prefix the training script with `scripts/distributed.sh`. For example, run `scripts/distributed.sh scripts/train_mnist.sh`. Note that (at this time) distributed training is only supported locally.\n\nSince training can take a significant amount of time, the training script does not overwrite any weights previously produced. Results are placed in sub-directories under `logs/` named with the date and time when training began. The latest results are always soft-linked to by `latest-log_dir` and `latest_log_file`.\n\n#### Troubleshooting\n\n1. If the training script returns `ModuleNotFoundError: No module named 'numpy'`, please activate the virtual environment using `source .venv/bin/activate`, or on native Windows without WSL2, `source .venv/scripts/activate`.\n\n2. If the training script crashes, or if it returns an internal error (such as `CUDNN_STATUS_INTERNAL_ERROR`), it may be necessary to limit the number of PyTorch workers to 1 (this has been observed running on native Windows). Add `--workers=1` when running any training script, for example;\n\n ```shell\n $ scripts/train_mnist.sh --workers=1\n ```\n\n3. On resource constrained systems, training may abort with an error message such as `RuntimeError: unable to open shared memory object \u003c/torch_..._...\u003e in read-write mode`. Add `--workers=0` when running the training script.\n\n4. By default, many systems limit the number of open file descriptors. `train.py` checks this limit and prints `WARNING: The open file limit is 2048. Please raise the limit (see documentation)` when the limit is low. When the limit is too low, certain actions might abort:\n\n ```shell\n (ai8x-training) $ scripts/evaluate_facedet_tinierssd.sh \n WARNING: The open file limit is 1024. Please raise the limit (see documentation).\n ...\n --- test ---------------------\n 165656 samples (256 per mini-batch)\n {'multi_box_loss': {'alpha': 2, 'neg_pos_ratio': 3}, 'nms': {'min_score': 0.75, 'max_overlap': 0.3, 'top_k': 20}}\n Traceback (most recent call last):\n ...\n RuntimeError: unable to open shared memory object \u003c/torch_202118_3977843486\u003e in read-write mode\n OSError: [Errno 24] Too many open files\n ...\n ```\n\n To fix this issue, check `ulimit -n` (the soft limit) as well as `ulimit -n -H` (the hard limit) and raise the file descriptor limit using `ulimit -n NUMBER` where NUMBER cannot exceed the hard limit. Note that on many Linux systems, the defaults can be configured in `/etc/security/limits.conf`.\n\n5. Datasets with larger-dimension images may require substantial amounts of system RAM. For example, `scripts/train_kinetics.sh` is configured for systems with 64 GB of RAM. When the system runs out of memory, training is abruptly killed and the error is logged to system journal. The following examples are from a system with 48 GB of RAM:\n\n ```shell\n ...\n Epoch: [13][ 142/ 142] Overall Loss 1.078153 Objective Loss 1.078153 Top1 64.062500 LR 0.000500 Time 1.247024 \n --- validate (epoch=13)-----------\n 1422 samples (32 per mini-batch)\n Epoch: [13][ 10/ 45] Loss 1.082790 Top1 60.937500 \n Epoch: [13][ 20/ 45] Loss 1.099474 Top1 60.312500 \n Epoch: [13][ 30/ 45] Loss 1.113100 Top1 59.791667 \n Killed\n ```\n\n and from the system journal:\n\n ```shell\n kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-11289.scope,task=python,pid=226828,uid=1000\n kernel: Out of memory: Killed process 226828 (python) total-vm:81269752kB, anon-rss:5711328kB, file-rss:146056kB, shmem-rss:648024kB, UID:1000 pgtables:97700kB oom_score_adj:0\n kernel: oom_reaper: reaped process 226828 (python), now anon-rss:0kB, file-rss:145268kB, shmem-rss:648024kB\n ```\n\n Training might succeed after reducing the batch size, reducing image dimensions, or pruning the dataset. Unfortunately, the only real fix for this issue is more system RAM. In the example, `kinetics_get_datasets()` from `datasets/kinetics.py` states “The current implementation of using 2000 training and 150 test examples per class at 240×240 resolution and 5 frames per second requires around 50 GB of RAM.”\n\n6. On CUDA-capable machines, the training script by default uses PyTorch 2’s [`torch.compile()` feature](https://pytorch.org/docs/stable/generated/torch.compile.html) which improves execution speed. However, some models may not support this feature. It can be disabled using the command line option\n `--compiler-mode none`\n Disabling `torch.compile()` may also be necessary when using AMD ROCm acceleration.\n\n\n### Example Training Session\n\nUsing the MNIST dataset and a simple model as an example, run `scripts/train_mnist.sh`. The following is the shortened output of an MNIST training session:\n\n```shell\n(ai8x-training) $ scripts/train_mnist.sh \nConfiguring device: MAX78000, simulate=False.\nLog file for this run: logs/2021.07.13-111453/2021.07.13-111453.log\n{'start_epoch': 10, 'weight_bits': 8}\nOptimizer Type: \u003cclass 'torch.optim.sgd.SGD'\u003e\nOptimizer Args: {'lr': 0.1, 'momentum': 0.9, 'dampening': 0, 'weight_decay': 0.0001, 'nesterov': False}\nDownloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz\nDownloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz to data/MNIST/raw/train-images-idx3-ubyte.gz\n9913344it [00:01, 5712259.71it/s] \nExtracting data/MNIST/raw/train-images-idx3-ubyte.gz to data/MNIST/raw\n\n...\n\nDataset sizes:\n training=54000\n validation=6000\n test=10000\nReading compression schedule from: policies/schedule.yaml\n\n\nTraining epoch: 54000 samples (256 per mini-batch)\nNamed tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.)\n\nEpoch: [0][ 10/ 211] Overall Loss 2.298435 Objective Loss 2.298435 Top1 13.710937 Top5 52.070313 LR 0.100000 Time 0.054167 \nEpoch: [0][ 20/ 211] Overall Loss 2.267082 Objective Loss 2.267082 Top1 16.464844 Top5 58.535156 LR 0.100000 Time 0.039278 \n...\nEpoch: [0][ 211/ 211] Overall Loss 0.867936 Objective Loss 0.867936 Top1 71.101852 Top5 92.837037 LR 0.100000 Time 0.025054 \n\n--- validate (epoch=0)-----------\n6000 samples (256 per mini-batch)\nEpoch: [0][ 10/ 24] Loss 0.295286 Top1 91.367188 Top5 99.492188 \nEpoch: [0][ 20/ 24] Loss 0.293729 Top1 91.054688 Top5 99.550781 \nEpoch: [0][ 24/ 24] Loss 0.296180 Top1 91.000000 Top5 99.550000 \n==\u003e Top1: 91.000 Top5: 99.550 Loss: 0.296\n\n==\u003e Confusion:\n[[581 2 3 1 2 3 4 3 2 4]\n [ 0 675 4 1 3 0 1 4 0 0]\n [ 5 6 501 21 11 2 4 25 7 4]\n [ 1 4 7 549 3 5 0 11 2 1]\n [ 2 6 7 0 525 1 3 9 0 12]\n [ 0 8 2 10 5 464 3 8 6 12]\n [ 13 18 1 0 10 8 574 0 6 1]\n [ 1 11 8 7 3 4 0 588 0 3]\n [ 26 4 7 5 9 9 16 5 482 21]\n [ 4 9 5 7 36 8 0 19 6 521]]\n\n==\u003e Best [Top1: 91.000 Top5: 99.550 Sparsity:0.00 Params: 71148 on epoch: 0]\nSaving checkpoint to: logs/2021.07.13-111453/checkpoint.pth.tar\n\n...\n\nTraining epoch: 54000 samples (256 per mini-batch)\nEpoch: [199][ 10/ 211] Overall Loss 0.033614 Objective Loss 0.033614 Top1 98.984375 Top5 100.000000 LR 0.000100 Time 0.052778 \n...\nEpoch: [199][ 211/ 211] Overall Loss 0.027310 Objective Loss 0.027310 Top1 99.181481 Top5 99.992593 LR 0.000100 Time 0.024874 \n\n--- validate (epoch=199)-----------\n6000 samples (256 per mini-batch)\nEpoch: [199][ 10/ 24] Loss 0.027533 Top1 98.984375 Top5 100.000000 \nEpoch: [199][ 20/ 24] Loss 0.028965 Top1 98.984375 Top5 100.000000 \nEpoch: [199][ 24/ 24] Loss 0.028365 Top1 98.983333 Top5 100.000000 \n==\u003e Top1: 98.983 Top5: 100.000 Loss: 0.028\n\n==\u003e Confusion:\n[[599 0 1 1 0 0 3 0 0 1]\n [ 0 685 0 1 0 0 0 2 0 0]\n [ 0 1 581 0 0 0 0 2 2 0]\n [ 0 0 1 578 0 2 0 1 1 0]\n [ 0 1 1 0 558 0 0 0 1 4]\n [ 1 0 0 2 0 513 1 0 1 0]\n [ 2 1 0 0 1 0 625 0 2 0]\n [ 0 1 3 1 0 0 0 619 0 1]\n [ 1 0 1 1 1 1 2 0 577 0]\n [ 0 0 0 0 2 1 0 6 2 604]]\n\n==\u003e Best [Top1: 99.283 Top5: 100.000 Sparsity:0.00 Params: 71148 on epoch: 180]\nSaving checkpoint to: logs/2021.07.13-111453/qat_checkpoint.pth.tar\n--- test ---------------------\n10000 samples (256 per mini-batch)\nTest: [ 10/ 40] Loss 0.017528 Top1 99.453125 Top5 100.000000 \nTest: [ 20/ 40] Loss 0.015671 Top1 99.492188 Top5 100.000000 \nTest: [ 30/ 40] Loss 0.013522 Top1 99.583333 Top5 100.000000 \nTest: [ 40/ 40] Loss 0.013415 Top1 99.590000 Top5 100.000000 \n==\u003e Top1: 99.590 Top5: 100.000 Loss: 0.013\n\n==\u003e Confusion:\n[[ 980 0 0 0 0 0 0 0 0 0]\n [ 0 1133 1 0 0 0 0 1 0 0]\n [ 1 0 1025 1 0 0 0 5 0 0]\n [ 0 0 0 1010 0 0 0 0 0 0]\n [ 0 0 0 0 978 0 2 0 0 2]\n [ 0 0 0 3 0 888 1 0 0 0]\n [ 0 1 0 0 1 2 953 0 1 0]\n [ 0 1 0 0 0 0 0 1026 0 1]\n [ 0 0 2 1 1 1 0 1 967 1]\n [ 0 0 0 0 5 2 0 3 0 999]]\n\n\nLog file for this run: logs/2021.07.13-111453/2021.07.13-111453.log\n\n```\n\nFor classification, the “Top-1” score refers to the percentage of samples that returned the correct class (the correct target label), while “Top-5” is the percentage of samples the correct answer was one of the five highest ranked predictions. The “Loss” shows the output of the loss function that the training session aims to minimize (the “loss” numbers may be larger than 1, depending on the dataset and model). “LR” is the learning rate, and depending on the learning rate schedule used, LR may decrease as training progresses.\n\nThe “Confusion Matrix” shows both the target (expected) label on the vertical (Y) axis, as well as the highest ranked prediction on the horizontal (X) axis. If the network returns 100% expected labels, then only the diagonal (top left to bottom right) will contain values greater than 0.\n\nWhen enabling TensorBoard (see [TensorBoard](#tensorboard)), these and other statistics are also available in graphical form:\n\n![confusionmatrix](docs/confusionmatrix.png)\n\n### Command Line Arguments\n\nThe following table describes the most important command line arguments for `train.py`. Use `--help` for a complete list.\n\n| Argument | Description | Example |\n| -------------------------- | ------------------------------------------------------------ | ------------------------------- |\n| `--help` | Complete list of arguments | |\n| *Device selection* | | |\n| `--device` | Set device (default: AI84) | `--device MAX78000` |\n| *Model and dataset* | | |\n| `-a`, `--arch`, `--model` | Set model (collected from models folder) | `--model ai85net5` |\n| `-f`, `--out-fold-ratio` | Fold ratio for the model output (default: 1). Fold ratio 1 means no folding. | `--out-fold-ratio 4` |\n| `--dataset` | Set dataset (collected from datasets folder) | `--dataset MNIST` |\n| `--data` | Path to dataset (default: data) | `--data /data/ml` |\n| *Training* | | |\n| `--epochs` | Number of epochs to train (default: 90) | `--epochs 100` |\n| `-b`, `--batch-size` | Mini-batch size (default: 256) | `--batch-size 512` |\n| `--compress` | Set compression and learning rate schedule | `--compress schedule.yaml` |\n| `--lr`, `--learning-rate` | Set initial learning rate | `--lr 0.001` |\n| `--deterministic` | Seed random number generators with fixed values | |\n| `--resume-from` | Resume from previous checkpoint | `--resume-from chk.pth.tar` |\n| `--qat-policy` | Define QAT policy in YAML file (default: policies/qat_policy.yaml). Use “None” to disable QAT. | `--qat-policy qat_policy.yaml` |\n| `--nas` | Enable network architecture search | |\n| `--nas-policy` | Define NAS policy in YAML file | `--nas-policy nas/nas_policy.yaml` |\n| `--regression` | Select regression instead of classification (changes Loss function, and log output) | |\n| `--compiler-mode` | Select [TorchDynamo optimization mode](https://pytorch.org/docs/stable/generated/torch.compile.html) (default: enabled on CUDA capable machines) | `--compiler-mode none` |\n| `--dr` | Set target embedding dimensionality for dimensionality reduction |`--dr 64` |\n| `--scaf-lr` | Initial learning rate for sub-center ArcFace loss optimizer | |\n| `--scaf-scale` |Scale hyperparameter for sub-center ArcFace loss | |\n| `--scaf-margin` |Margin hyperparameter for sub-center ArcFace loss | |\n| `--backbone-checkpoint` |Path to checkpoint from which to load backbone weights | |\n| *Display and statistics* | | |\n| `--enable-tensorboard` | Enable logging to TensorBoard (default: disabled) | |\n| `--confusion` | Display the confusion matrix | |\n| `--param-hist` | Collect parameter statistics | |\n| `--pr-curves` | Generate precision-recall curves | |\n| `--embedding` | Display embedding (using projector) | |\n| *Hardware* | | |\n| `--use-bias` | The `bias=True` parameter is passed to the model. The effect of this parameter is model-dependent (the parameter does nothing, affects some operations, or all operations). | |\n| `--avg-pool-rounding` | Use rounding for AvgPool | |\n| *Evaluation* | | |\n| `-e`, `--evaluate` | Evaluate previously trained model | |\n| `--8-bit-mode`, `-8` | Simulate quantized operation for hardware device (8-bit data). Used for evaluation only. | |\n| `--exp-load-weights-from` | Load weights from file | |\n| *Export* | | |\n| `--summary onnx` | Export trained model to ONNX (default name: to model.onnx) — *see description below* | |\n| `--summary onnx_simplified` | Export trained model to simplified [ONNX](https://onnx.ai/) file (default name: model.onnx) | |\n| `--summary-filename` | Change the file name for the exported model | `--summary-filename mnist.onnx` |\n| `--save-sample` | Save data[index] from the test set to a NumPy pickle for use as sample data | `--save-sample 10` |\n| `--slice-sample` | For models that require RGB input, when the sample from the dataset has additional channels, slice the sample into 3 channels | |\n\n#### ONNX Model Export\n\nThe ONNX model export (via `--summary onnx` or `--summary onnx_simplified`) is primarily intended for visualization of the model. ONNX does not support all of the operators that `ai8x.py` uses, and these operators are therefore removed from the export (see function `onnx_export_prep()` in `ai8x.py`). The ONNX file does contain the trained weights and *may* therefore be usable for inference under certain circumstances. However, it is important to note that the ONNX file **will not** be usable for training (for example, the ONNX `floor` operator has a gradient of zero, which is incompatible with quantization-aware training as implemented in `ai8x.py`).\n\n### Observing GPU Resources\n\n`nvidia-smi` can be used in a different terminal during training to examine the GPU resource usage of the training process. In the following example, the GPU is using 100% of its compute capabilities, but not all of the available memory. In this particular case, the batch size could be increased to use more memory.\n\n```shell\n$ nvidia-smi\n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 470.42.01 Driver Version: 470.42.01 CUDA Version: 11.4 |\n|-------------------------------+----------------------+----------------------+\n| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n|===============================+======================+======================|\n| 0 GeForce RTX 208... Off | 00000000:01:00.0 On | N/A |\n| 39% 65C P2 152W / 250W | 3555MiB / 11016MiB | 100% Default |\n+-------------------------------+----------------------+----------------------+\n...\n```\n\n### Custom nn.Modules\n\nThe `ai8x.py` file contains customized PyTorch classes (subclasses of `torch.nn.Module`). Any model that is designed to run on MAX78000/MAX78002 should use these classes. There are three main changes over the default classes in `torch.nn.Module`:\n\n1. Additional “Fused” operators that model in-flight pooling and activation.\n2. Rounding, clipping and activation that matches the hardware.\n3. Support for quantized operation (when using the `-8` command line argument).\n\n#### set_device()\n\n`ai8x.py` defines the `set_device()` function which configures the training system:\n\n```python\ndef set_device(\n device,\n simulate,\n round_avg,\n verbose=True,\n):\n```\n\nwhere *device* is `85` (the MAX78000 device code) or `87` (the MAX78002 device code), *simulate* is `True` when clipping and rounding are set to simulate hardware behavior, and *round_avg* picks one of the two hardware rounding modes for AvgPool.\n\n#### update_model()\n\n`ai8x.py` defines `update_model()`. This function is called after loading a checkpoint file, and recursively applies output shift, weight scaling, and quantization clamping to the model.\n\n\n\n#### List of Predefined Modules\n\nThe following modules are predefined:\n\n| Name | Description / PyTorch equivalent |\n| ---------------------- | --------------------------------------- |\n| Conv2d | Conv2d |\n| FusedConv2dReLU | Conv2d, followed by ReLU |\n| FusedConv2dAbs | Conv2d, followed by Abs |\n| MaxPool2d | MaxPool2d |\n| FusedMaxPoolConv2d | MaxPool2d, followed by Conv2d |\n| FusedMaxPoolConv2dReLU | MaxPool2d, followed by Conv2d, and ReLU |\n| FusedMaxPoolConv2dAbs | MaxPool2d, followed by Conv2d, and Abs |\n| AvgPool2d | AvgPool2d |\n| FusedAvgPoolConv2d | AvgPool2d, followed by Conv2d |\n| FusedAvgPoolConv2dReLU | AvgPool2d, followed by Conv2d, and ReLU |\n| FusedAvgPoolConv2dAbs | AvgPool2d, followed by Conv2d, and Abs |\n| ConvTranspose2d | ConvTranspose2d |\n| FusedConvTranspose2dReLU | ConvTranspose2d, followed by ReLU |\n| FusedConvTranspose2dAbs | ConvTranspose2d, followed by Abs |\n| FusedMaxPoolConvTranspose2d | MaxPool2d, followed by ConvTranspose2d |\n| FusedMaxPoolConvTranspose2dReLU | MaxPool2d, followed by ConvTranspose2d, and ReLU |\n| FusedMaxPoolConvTranspose2dAbs | MaxPool2d, followed by ConvTranspose2d, and Abs |\n| FusedAvgPoolConvTranspose2d | AvgPool2d, followed by ConvTranspose2d |\n| FusedAvgPoolConvTranspose2dReLU | AvgPool2d, followed by ConvTranspose2d, and ReLU |\n| FusedAvgPoolConvTranspose2dAbs | AvgPool2d, followed by ConvTranspose2d, and Abs |\n| Linear | Linear |\n| FusedLinearReLU | Linear, followed by ReLU |\n| FusedLinearAbs | Linear, followed by Abs |\n| Conv1d | Conv1d |\n| FusedConv1dReLU | Conv1d, followed by ReLU |\n| FusedConv1dAbs | Conv1d, followed by Abs |\n| MaxPool1d | MaxPool1d |\n| FusedMaxPoolConv1d | MaxPool1d, followed by Conv1d |\n| FusedMaxPoolConv1dReLU | MaxPool1d, followed by Conv1d, and ReLU |\n| FusedMaxPoolConv1dAbs | MaxPool1d, followed by Conv1d, and Abs |\n| AvgPool1d | AvgPool1d |\n| FusedAvgPoolConv1d | AvgPool1d, followed by Conv1d |\n| FusedAvgPoolConv1dReLU | AvgPool1d, followed by Conv1d, and ReLU |\n| FusedAvgPoolConv1dAbs | AvgPool1d, followed by Conv1d, and Abs |\n| Add | Element-wise Add |\n| Sub | Element-wise Sub |\n| BitwiseOr | Element-wise bitwise Or |\n| BitwiseXor | Element-wise bitwise Xor |\n\n#### Dropout\n\nDropout modules such as `torch.nn.Dropout()` and `torch.nn.Dropout2d()` are automatically disabled during inference, and can therefore be used for training without affecting inference. [Dropout](https://en.wikipedia.org/wiki/Dilution_(neural_networks)) can improve generalization by reducing overfitting, but should not be used for “analytical” functions.\n\n*Note: Using [batch normalization](#batch-normalization) in conjunction with dropout can sometimes degrade training results.*\n\n#### view(), reshape() and Flatten\n\nThere are two supported cases for `view()` or `reshape()`.\n\n1. Conversion between 1D data and 2D data: Both the batch dimension (first dimension) and the channel dimension (second dimension) must stay the same. The height/width of the 2D data must match the length of the 1D data (i.e., H×W = L).\n Examples:\n `x = x.view(x.size(0), x.size(1), -1) # 2D to 1D`\n `x = x.view(x.shape[0], x.shape[1], 16, -1) # 1D to 2D`\n *Note: `x.size()` and `x.shape[]` are equivalent.*\n When reshaping data, `in_dim:` must be specified in the model description file.\n2. Conversion from 1D and 2D to Fully Connected (“flattening”): The batch dimension (first dimension) must stay the same, and the other dimensions are combined (i.e., M = C×H×W or M = C×L).\n Example:\n `x = x.view(x.size(0), -1) # Flatten`\n An alternate way to express the flatten operation is `torch.nn.Flatten()`.\n\n#### Support for Quantization\n\nThe hardware always uses signed integers for data and weights. While data is always 8-bit, weights can be configured on a per-layer basis. However, training makes use of floating point values for both data and weights, while also clipping (clamping) values.\n\n##### Data\n\nWhen using the `-8` command line switch, all module outputs are quantized to 8-bit in the range [-128...+127] to simulate hardware behavior. The `-8` command line switch is designed for *evaluating quantized weights* against a test set, in order to understand the impact of quantization. *Note that model training always uses floating point values, and therefore `-8` is not compatible with training.*\n\nThe last layer can optionally use 32-bit output for increased precision. This is simulated by adding the parameter `wide=True` to the module function call.\n\n##### Weights and Activations: Quantization-Aware Training (QAT)\n\nQuantization-aware training (QAT) is enabled by default. QAT is controlled by a policy file, specified by `--qat-policy`.\n\n* After `start_epoch` epochs, an intermediate epoch with no backpropagation will be realized to collect activation statistics. Each layer's activation ranges will be determined based on the range \u0026 resolution trade-off from the collected activations. Then, QAT will start and an additional parameter (`output_shift`) will be learned to shift activations for compensating weights \u0026 biases scaling down.\n* `weight_bits` describes the number of bits available for weights.\n* `overrides` allows specifying the `weight_bits` on a per-layer basis.\n* `outlier_removal_z_score` defines the z-score threshold for outlier removal during activation range calculation. (default: 8.0)\n* `shift_quantile` defines the quantile of the parameters distribution to be used for the `output_shift` parameter. (default: 1.0)\n\nBy default, weights are quantized to 8-bits after 30 epochs as specified in `policies/qat_policy.yaml`. A more refined example that specifies weight sizes for individual layers can be seen in `policies/qat_policy_cifar100.yaml`.\n\nQuantization-aware training can be \u003cu\u003edisabled\u003c/u\u003e by specifying `--qat-policy None`.\n\nThe proper choice of `start_epoch` is important for achieving good results, and the default policy’s `start_epoch` may be much too small. As a rule of thumb, set `start_epoch` to a very high value (e.g., 1000) to begin, and then observe where in the training process the model stops learning. This epoch can be used as `start_epoch`, and the final network metrics (after an additional number of epochs) should be close to the non-QAT metrics. *Additionally, ensure that the learning rate after the `start_epoch` epoch is relatively small.*\n\nFor more information, please also see [Quantization](#quantization) and [QATv2](https://github.com/analogdevicesinc/ai8x-training/blob/develop/docs/QATv2.md).\n\n#### Batch Normalization\n\nBatch normalization after `Conv1d` and `Conv2d` layers is supported using “fusing.” The fusing operation merges the effect of batch normalization layers into the parameters of the preceding convolutional layer, by modifying weights and bias values of that preceding layer. For detailed information about batch normalization fusing/fusion/folding, see Section 3.2 of the following paper: \u003chttps://arxiv.org/pdf/1712.05877.pdf\u003e.\n\nAfter fusing/folding, the network will no longer contain any batchnorm layers. The effects of batch normalization will instead be expressed by modified weights and biases of the preceding convolutional layer.\n\n* When using [Quantization-Aware Training (QAT)](#quantization-aware-training-qat), batchnorm layers \u003cu\u003eare automatically folded\u003c/u\u003e during training and no further action is needed.\n* When using [Post-Training Quantization](#post-training-quantization), the `batchnormfuser.py` script (see [BatchNorm Fusing](#batchnorm-fusing)) must be called before `quantize.py` to explicitly fuse the batchnorm layers.\n\n*Note: Using batch normalization in conjunction with [dropout](#dropout) can sometimes degrade training results.*\n\n### Adapting Pre-existing Models\n\nIn some cases, it may be possible to use generic models that were designed for non-MAX78000/MAX78002 platforms. To adapt pre-existing models to MAX78000/MAX78002, several steps are needed:\n\n1. Check that all operators are supported in hardware (see [List of Predefined Modules](#list-of-predefined-modules), [Dropout](#dropout), and [Batch Normalization](#batch-normalization)).\n2. Check that the model size, parameter count, and parameters to the operators are supported (see [Limitations of MAX78000 Networks](#limitations-of-max78000-networks) and [Limitations of MAX78002 Networks](#limitations-of-max78002-networks)). For example, padding must always be zero-padding, and `Conv2d()` supports 1×1 and 3×3 kernels.\n3. Change from PyTorch *nn.modules* to the *ai8x* versions of the modules. For example, `nn.Conv2d(…)` ⟶ `ai8x.Conv2d(…)`.\n4. Merge modules where possible (for example, `MaxPool2d()` + `Conv2d()` + `ReLU()` = `FusedMaxPoolConv2dReLU()`).\n5. [Re-train](#model-training-and-quantization) the model. *This is necessary to correctly model clipping and quantization effects of the hardware.*\n\n### Model Comparison and Feature Attribution\n\nTensorBoard can be used for model comparison and feature attribution.\n\n#### TensorBoard\n\n[TensorBoard](https://www.tensorflow.org/tensorboard/) support is built into `train.py`. When enabled using `--enable-tensorboard`, it provides a local web server that can be started before, during, or after training, and it picks up all data that is written to the `logs/` directory.\n\nFor classification models, TensorBoard supports the optional `--param-hist` and `--embedding` command line arguments. `--embedding` randomly selects up to 100 data points from the last batch of each verification epoch. These can be viewed in the “projector” tab in TensorBoard.\n\n`--pr-curves` adds support for displaying precision-recall curves.\n\nTo start the TensorBoard server, use a second terminal window:\n\n```shell\n(ai8x-training) $ tensorboard --logdir='./logs'\nTensorBoard 2.4.1 at http://127.0.0.1:6006/ (Press CTRL+C to quit)\n```\n\nOn a shared system, add the `--port 0` command line option.\n\nThe training progress can be observed by starting TensorBoard and pointing a web browser to the port indicated.\n\n##### Examples\n\nTensorBoard produces graphs and displays metrics that may help optimize the training process, and can compare the performance of multiple training sessions and their settings. Additionally, TensorBoard can show a graphical representation of the model and its parameters, and help discover labeling errors. For more information, please see the [TensorBoard web site](https://www.tensorflow.org/tensorboard/).\n\n\u003cimg src=\"docs/lr.png\" alt=\"learning rate\" style=\"zoom: 50%;\" /\u003e\u003cimg src=\"docs/top1.png\" alt=\"top-1\" style=\"zoom:50%;\" /\u003e\u003cimg src=\"docs/objectiveloss.png\" alt=\"objective loss\" style=\"zoom:42%;\" /\u003e\u003cimg src=\"docs/histogram.png\" alt=\"histogram\" style=\"zoom:50%;\" /\u003e\u003cimg src=\"docs/model.png\" alt=\"model\" style=\"zoom:50%;\" /\u003e\u003cimg src=\"docs/projector.png\" alt=\"projector\" style=\"zoom:50%;\" /\u003e\n\n##### Remote Access to TensorBoard\n\nWhen using a remote system, use `ssh` in another terminal window to forward the remote port to the local machine:\n\n```shell\n$ ssh -L 6006:127.0.0.1:6006 targethost\n```\n\nWhen using PuTTY, port forwarding is achieved as follows:\n\n![putty-forward](docs/putty-forward.jpg)\n\n\n\n### BatchNorm Fusing\n\nBatchnorm fusing (see [Batch Normalization](#batch-normalization)) is needed as a separate step \u003cu\u003eonly when both\u003c/u\u003e the following are true:\n\n1. Batch Normalization is used in the network and\n2. [Quantization-Aware Training (QAT)](#quantization-aware-training-qat) is \u003cu\u003enot\u003c/u\u003e used (i.e., when [post-training quantization](#post-training-quantization) is active).\n\nIn order to perform batchnorm fusing, the `batchnormfuser.py` tool must be run *before* the `quantize.py` script.\n\n*Note: Most of the examples either don’t use batchnorm, so no fusing is needed, or they use QAT, so batchnorm fusing happens automatically.*\n\n#### Command Line Arguments\n\nThe following table describes the command line arguments for `batchnormfuser.py`:\n\n| Argument | Description | Example |\n| ------------------- | ------------------------------------------------------------ | ---------------------------------------- |\n| `-i`, `--inp_path` | Set input checkpoint path | `-i logs/2020.06.05-235316/best.pth.tar` |\n| `-o`, `--out_path` | Set output checkpoint path for saving fused model | `-o best_without_bn.pth.tar` |\n| `-oa`, `--out_arch` | Set output architecture name (architecture without batchnorm layers) | `-oa ai85simplenet` |\n\n\n\n### Data Folding\n\n*Data Folding* is data reshaping operation. When followed by a Conv2d operation, it is equivalent to a convolution operation on the original image with a larger kernel and a larger stride.\n\nOn MAX78000 and MAX78002, data folding is beneficial because it increases available resolution and reduces latency. A typical 3-channel RGB image uses only three processors in the first layer which increases latency, and restricts the image dimensions to what can be fit into the data memories associated with three processors.\n\nBy creating many low resolution sub-images and concatenating them through the channel dimension, up to 64 processors and their associated data memories can be used. This results in a higher maximum effective resolution, and increased throughput in the first layer.\n\nFor certain models (see `models/ai85net-unet.py` in the training repository) this also improves model performance, due to the increase in effective kernel size and stride.\n\nNote that data folding must be applied during model training. During inference, there is no additional overhead; the input data is simply loaded to different processors/memory addresses.\n\n\n\n### Quantization\n\nThere are two main approaches to quantization — quantization-aw","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanalogdevicesinc%2Fai8x-training","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanalogdevicesinc%2Fai8x-training","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanalogdevicesinc%2Fai8x-training/lists"}