{"id":19277557,"url":"https://github.com/enzymead/enzyme-sc22","last_synced_at":"2025-04-21T23:32:25.776Z","repository":{"id":38817296,"uuid":"470251802","full_name":"EnzymeAD/enzyme-sc22","owner":"EnzymeAD","description":null,"archived":false,"fork":false,"pushed_at":"2022-09-13T18:18:15.000Z","size":701,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-01T17:09:39.316Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EnzymeAD.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-15T16:51:32.000Z","updated_at":"2024-06-13T08:01:02.000Z","dependencies_parsed_at":"2023-01-17T16:15:29.013Z","dependency_job_id":null,"html_url":"https://github.com/EnzymeAD/enzyme-sc22","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EnzymeAD%2Fenzyme-sc22","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EnzymeAD%2Fenzyme-sc22/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EnzymeAD%2Fenzyme-sc22/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EnzymeAD%2Fenzyme-sc22/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EnzymeAD","download_url":"https://codeload.github.com/EnzymeAD/enzyme-sc22/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250150952,"owners_count":21383260,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T21:06:14.702Z","updated_at":"2025-04-21T23:32:25.466Z","avatar_url":"https://github.com/EnzymeAD.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Benchmark Repository for \"Scalable Automatic Differentiation of Multiple Parallel Paradigms through Compiler Augmentation\", to appear in SC'22\n\nWe have introduced a composable and generic LLVM-based mechanism to\ndifferentiate a variety of parallel programming models. To illustrate\nthe composability of Enzyme’s differentiation of parallel frameworks, we\napply it to several distinct parallel variations of LULESH, and\nminiBUDE. We evaluate LULESH variations that use MPI, OpenMP, hybrid\nMPI+OpenMP, MPI.jl, and the RAJA portable parallel programming\nframework, written in C++ and Julia. To further compare our performance\nagainst current literature, we are comparing the automatic\ndifferentiation performance to the CoDiPack-differentiated LULESH. Our\nevaluation on miniBUDE was designed to validate our automatic\ndifferentiation performance claims on LULESH on a second, distinct\napplication, as well as testing Enzyme’s ability to automatically\ndifferentiate Julia’s shared-memory parallelism. We evaluate an OpenMP\nversion in C++, and a Julia-version utilizing tasks. To differentiate\nboth two Julia codes, we extend Enzyme.jl, Enzyme’s Julia bindings.\nBecause Enzyme is a tool that takes arbitrary existing code as LLVM IR\nand computes the derivative (and gradient) of that function, LLVM is a\nprerequisite for Enzyme. The particular LLVM built here enables turning\non/off OpenMP optimizations for an ablation analysis.\n\nEvaluation of Enzyme upon these benchmarks allowed the paper to validate\nour automatic differentiation performance claims of both efficiency and\nscalability. As such, the original (referred to as the primal) and\ndifferentiated versions of these benchmarks were evaluated on varying\nthread counts and ranks, as well\n\n## 1. Machine\n\nExperiments for the paper were run on an AWS c6i.metal instance with hyper-threading\nand Turbo Boost disabled, running Ubuntu 20.04 running on a dual-socket\nIntel Xeon Platinum 8375C CPU at 2.9 GHz with 32 cores each and 256 GB\nRAM.\n\n## 2. Obtaining the code\n\nAll the codes and benchmarks are available on Github in this repository.\nWe first obtain the code:\n```console\ncd $HOME\ngit clone --recursive https://github.com/EnzymeAD/enzyme-sc22\ncd enzyme-sc22\n```\n\nThis repository contains submodules for the benchmarks and codes listed\nbelow.\n\n\u003cdiv id=\"tbl:code_details\"\u003e\n\n| Code        | Link                                               | Hash      |\n|:------------|:---------------------------------------------------|:----------|\n| BUDE        | [github.com/wsmoses/Enzyme-BUDE](github.com/wsmoses/Enzyme-BUDE)     | `28b6d6e` |\n| LULESH-CoDi | [github.com/wsmoses/CODI-LULESH](github.com/wsmoses/CODI-LULESH)      | `566b2ef` |\n| LULESH-CPP  | [github.com/wsmoses/Enzyme-MPI](github.com/wsmoses/Enzyme-MPI)       | `47e0a3e` |\n| LULESH-RAJA | [github.com/wsmoses/LULESH-MPI-RAJA](github.com/wsmoses/LULESH-MPI-RAJA)  | `45146d3` |\n| LULESH.jl   | [github.com/JuliaLabs/LULESH.jl](github.com/JuliaLabs/LULESH.jl)      | `2338418` |\n| Enzyme      | [github.com/EnzymeAD/Enzyme](github.com/EnzymeAD/Enzyme)          | `5c89a86` |\n| LLVM        | [github.com/jdoerfert/llvm-project](github.com/jdoerfert/llvm-project)   | `354c7f8` |\n\n\u003c/div\u003e\n\n\u003cspan id=\"tbl:code_details\" label=\"tbl:code_details\"\u003e\u003c/span\u003e\n\nTo evaluate the artifact, we offer several options.\n\n### 2-A Download CI Artifacts\nYou can download the build artifacts from this repository's CI.\nEvery push to this repository will automatically build, test, and\nupload all the benchmarks (see\n\u003chttps://github.com/EnzymeAD/enzyme-sc22/tree/main/.github/workflows\u003e\nfor the precise build commands). One can download the benchmarks\nbuilt by CI by selecting the “Actions” tab, selecting the latest\nbuild of the corresponding benchmark, and clicking the binary below\nthe “Artifacts” header. One may then skip the rest of\nthe section that involves downloading or building the experiments.\nNote that the binary is built on Ubuntu X86 and one will need a\ncompatible system (e.g. not ARM, not macOS, to run the prebuilt\nbinary from CI).\n\n### 2-B Docker\nYou may use a pre-built docker image (`wsmoses/enzymesc22`).\nThe docker image can then be invoked with\n    the following command:\n```console\nsudo docker run --privileged -it wsmoses/enzymesc22:latest /bin/bash\n```\nWe begin by installing OpenMPI and Julia and build LLVM and Enzyme.\n\n###  2-C Build From Source\nThis procedure for building the compilers and tests from source is outlined below.\n\n#### MPI\n\nOur tests with MPI require OpenMPI which can be obtained in Ubuntu using\nthe following command.\n```console\nsudo apt-get install -y autoconf cmake gcc g++ gfortran ninja-build libopenmpi-dev numactl\n```\n#### Julia\n\nThe Lulesh.jl and miniBUDE.jl tests were run using Julia 1.7. Julia at\nthis version must be found in your path before being able to run the\nJulia tests. To obtain a working Julia installation see\n\u003chttps://julialang.org/downloads/\u003e and follow the provided installation\ninstructions. You can add julia executable to the PATH variable using:\n```console\nexport PATH=/home/ubuntu/julia-1.7.3/bin/:$PATH\n```\n#### llvm-project\n\nWe first need to build the LLVM compiler toolchain before we can\nsubsequently link the compiler plugin of Enzyme against our built LLVM\nversion. For our compiler toolchain we used a fork of LLVM 15 (main)\nwhich enables OpenMPOpt to be completely disabled. To install LLVM,\nplease follow the following steps:\n```console\ncd $HOME/enzyme-sc22/llvm-project\nmkdir build \u0026\u0026 cd build\ncmake ../llvm -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS=\"llvm;clang;openmp\" -DLLVM_TARGETS_TO_BUILD=X86\nninja\n# This may take a while\n# clang is now be available in llvm-project/build/bin/clang\n```\n#### Enzyme\n\nWe now must build Enzyme based off of our chosen LLVM version.\n```console\ncd $HOME/enzyme-sc22/Enzyme/enzyme\nmkdir build\ncd build\ncmake .. -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_DIR=../../../llvm-project/build\nninja\n# ClangEnzyme-15.so will now be available in Enzyme/enzyme/build/Enzyme/\n```\nTo aid in the building of the benchmarks, it is necessary to setup two\nenvironment variables.\n```console\nexport ENZYME_PATH=$HOME/enzyme-sc22/Enzyme/enzyme/build/Enzyme/ClangEnzyme-15.so\nexport CLANG_PATH=$HOME/enzyme-sc22/llvm-project/build/bin\n```\n\n#### LULESH-CPP\n\nThe following commands can be used to build all the executables for LULESH C++ tests.\n```console\ncd $HOME/enzyme-sc22/LULESH-CPP\nmake -j\n# Binaries available in enzyme-sc22/LULESH-CPP\n# ser-single-forward.exe\n# ser-single-gradient.exe\n# omp-single-forward.exe\n# omp-single-gradient.exe\n# ompM-single-forward.exe\n# ompM-single-gradient.exe\n# ompOpt-single-forward.exe\n# ompOpt-single-gradient.exe\n```\n\n#### LULESH-RAJA\n\nThe following commands can be used to build all the executables for LULESH RAJA tests.\n```console\ncd $HOME/enzyme-sc22/LULESH-RAJA\nmkdir build\ncd build\ncmake .. -G Ninja -DENABLE_OPENMP=ON -DLLVM_BUILD=$CLANG_PATH/.. -DENZYME=$ENZYME_PATH -DMPI_INCLUDE=/usr/lib/x86_64-linux-gnu/openmpi/include\nninja\n# Binaries available in LULESH-RAJA/build/bin\n# lulesh-v2.0-RAJA-seq.exe\n# lulesh-v2.0-RAJA-seq-grad.exe\n# lulesh-v2.0-RAJA-omp.exe\n# lulesh-v2.0-RAJA-omp-gradient.exe\n# lulesh-v2.0-RAJA-ompOpt.exe\n# lulesh-v2.0-RAJA-ompOpt-gradient.exe\n# lulesh-v2.0-RAJA-seq-mpi.exe\n# lulesh-v2.0-RAJA-seq-mpi-grad.exe\n```\n\n#### LULESH.jl\n\nYou may then need to explicitly run various setup routines within\nJulia’s package manager. To fix the Julia setup for the test, perform\nthe following to enter an interactive shell.\n```console\ncd $HOME/enzyme-sc22/LULESH.jl\njulia --project -e \"import Pkg; Pkg.instantiate()\"\njulia --project\n    julia\u003e import MPI\n    julia\u003e MPI.install_mpiexecjl(;destdir=\".\",force=true)\n# The `mpiexecjl` executable should now\n# exist in $HOME/enzyme-sc22/LULESH.jl\n```\n\n#### LULESH-CoDiPack\n\nThe following commands can be used to build all the CoDiPack versions of LULESH.\n```console\ncd $HOME/enzyme-sc22/CODI-LULESH/lulesh-forward\nmake\n# Binaries available in CODI-LULESH/lulesh-forward/\n# lulesh2.0\ncd $HOME/enzyme-sc22/CODI-LULESH/lulesh-gradient\nmake\n# Binaries available in CODI-LULESH/lulesh-gradient/\n# lulesh2.0\n```\n\n#### BUDE\n\nThe following commands can be used to build all the executables.\n```console\ncd $HOME/enzyme-sc22/BUDE/openmp\nmake -j\n# Binaries available in enzyme-sc22/BUDE/openmp\n# ./ser-single-forward.exe\n# ./ser-single-gradient.exe\n# ./omp-single-forward.exe\n# ./omp-single-gradient.exe\n# ./ompOpt-single-forward.exe\n# ./ompOpt-single-gradient.exe\n```\n\n#### miniBUDE.jl\n\nThe following commands can be used to build all the executables.\n```console\ncd $HOME/enzyme-sc22/BUDE/miniBUDE.jl/\njulia --project=Threaded -e \"import Pkg; Pkg.instantiate()\"\n#No executables are created\n```\n\n## 3. Evaluation of Benchmarks\n\n### 3-A Disabling/Enabling Hyperthreading\n\nTo obtain reproducible results that are not subject to oddities\nresulting from thread mapping, we recommend the disabling of\nhyperthreading, if appropriate for the particular test case being run.\nWe have provided two scripts that can be easily edited for this purpose.\nNote that these scripts assume the use of the same dual-socket 32-core\nper CPU machine and can be modified to disable the appropriate cores for\na different machine. They can be run as follows:\n```console\ncd $HOME/enzyme-sc22\n./disable.sh\n```\nor\n```console\ncd $HOME/enzyme-sc22\n./enable.sh\n```\n\n### 3-B Executing Benchmarks\nOnce the preliminary setup is complete, we can now enter one of the test\ndirectories, build, and run the corresponding benchmark.\n\nWe have created Python3 scripts for running all the executables and performing scaling analyses. The python scripts run the executables at different MPI rank and OpenMP thread counts, corresponding to the experiments we performed in the paper. The raw timing numbers for the graphs presented in the paper are thus reproduced by running the experiments using the provided scripts on the test machine. If running on a machine of a different size, these scripts can be edited to use the available number of cores on your machine.\n\nAfter executing a benchmark, the raw data output from evaluating the benchmarks is contained in .txt files labeled with the parameters (rank/thread count/problem size/etc) of the individual test, like below:\n```\n$ cat omp-mpi-forward_1_2_100_48.txt \nRunning problem size 48^3 per domain until completion\nNum processors: 1\nNum threads: 2\nTotal number of elements: 110592 \n\nTo run other sizes, use -s \u003cinteger\u003e.\nTo run a fixed number of iterations, use -i \u003cinteger\u003e.\nTo run a more or less balanced region set, use -b \u003cinteger\u003e.\nTo change the relative costs of regions, use -c \u003cinteger\u003e.\nTo print out progress, use -p\nTo write an output file for VisIt, use -v\nSee help (-h) for more options\n\nRun completed:\n   Problem size        =  48\n   MPI tasks           =  1\n   Iteration count     =  100\n   Final Origin Energy =  5.417664e+06\n   Testing Plane 0 of Energy Array on rank 0:\n        MaxAbsDiff   = 2.328306e-10\n        TotalAbsDiff = 1.139172e-09\n        MaxRelDiff   =         -nan\n\nElapsed time         =          4 (s)\nGrind time (us/z/c)  = 0.36487352 (per dom)  ( 4.0352092 overall)\nFOM                  =  2740.6758 (z/s)\n```\n\nThe use of a helper Python3 script `res.py` will create a `results.txt` file which will summarize all of the corresponding tests in that directory, with first the runtime for the original code, followed by the derivative code.\n```\n$ python3 res.py \n$ cat results.txt \n1,2,48,       4\n8,2,48,      14\n27,2,48,      43\n1,2,48,      37\n8,2,48,      80\n27,2,48, 2.2e+02\n```\n\n#### LULESH-CPP\nTo run the evaluation:\n```console\ncd $HOME/enzyme-sc22/LULESH-CPP/bench/\ncd omp-mpi\n./script.py\n./res.py\n# output of benchmark times in results.txt\ncd ../omp-single\n./script.py\n./res.py\n# output of benchmark times in results.txt\ncd ../ompOpt-single\n./script.py\n./res.py\n# output of benchmark times in results.txt\ncd ../ser-mpi-strong-scaling\n./script.py\n./res.py\n# output of benchmark times in results.txt\ncd ../ser-mpi-weak-scaling\n./script.py\n./res.py\n# output of benchmark times in results.txt\n```\n#### LULESH-RAJA\n\nTo run the evaluation:\n```console\ncd $HOME/enzyme-sc22/LULESH-RAJA/bench\ncd omp-mpi\n./script.py\n./res.py\n# output of benchmark times in results.txt\ncd ../omp-single\n./script.py\n./res.py\n# output of benchmark times in results.txt\ncd ../ompOpt-single\n./script.py\n./res.py\n# output of benchmark times in results.txt\ncd ../ser-mpi-strong-scaling\n./script.py\n./res.py\n# output of benchmark times in results.txt\ncd ../ser-mpi-weak-scaling\n./script.py\n./res.py\n# output of benchmark times in results.txt\n```\n\n#### LULESH.jl\n\nTo run the evaluation:\n```console\ncd $HOME/enzyme-sc22/LULESH.jl/bench/\ncd ser-mpi-strong-scaling\n./script.py\n./res.py\n# output of benchmark times in results.txt\ncd ../ser-mpi-weak-scaling\n./script.py\n./res.py\n# output of benchmark times in results.txt\n```\n\n#### LULESH-CoDiPack\n\nTo run the evaluation:\n```console\ncd $HOME/enzyme-sc22/CODI-LULESH/bench/\ncd ser-mpi-strong-scaling\n./script.py\n./res.py\n# output of benchmark times in results.txt\ncd ../ser-mpi-weak-scaling\n./script.py\n./res.py\n```\n\n#### BUDE\nWe have created a Python3 script for running all the executables and\nperforming scaling analysis.\n```console\ncd $HOME/enzyme-sc22/BUDE/openmp/bench\ncd omp-single\n./script.py\n./res.py\n# output of benchmark times in results.txt\ncd ../ompOpt-single\n./script.py\n./res.py\n# output of benchmark times in results.txt\n```\n\n#### miniBUDE.jl\n\nWe have created a Python3 script for running all the executables and\nperforming scaling analysis.\n```console\ncd $HOME/enzyme-sc22/BUDE/miniBUDE.jl/bench/\ncd thread-strong-scaling\n./script.py\n./res.py\n# output of benchmark times in results.txt\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fenzymead%2Fenzyme-sc22","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fenzymead%2Fenzyme-sc22","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fenzymead%2Fenzyme-sc22/lists"}