{"id":28503180,"url":"https://github.com/turinglang/adtests","last_synced_at":"2026-01-30T11:24:57.837Z","repository":{"id":263190339,"uuid":"889580529","full_name":"TuringLang/ADTests","owner":"TuringLang","description":"AD x Turing.jl status","archived":false,"fork":false,"pushed_at":"2025-06-29T12:58:46.000Z","size":640,"stargazers_count":0,"open_issues_count":11,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-29T13:26:32.179Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://turinglang.org/ADTests/","language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TuringLang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-11-16T17:33:02.000Z","updated_at":"2025-06-19T22:09:42.000Z","dependencies_parsed_at":"2025-04-10T00:32:00.504Z","dependency_job_id":"882d1413-9d96-4b22-a7cd-b81ffde7d3f4","html_url":"https://github.com/TuringLang/ADTests","commit_stats":null,"previous_names":["penelopeysm/modeltests.jl","turinglang/adtests"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TuringLang/ADTests","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TuringLang%2FADTests","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TuringLang%2FADTests/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TuringLang%2FADTests/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TuringLang%2FADTests/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TuringLang","download_url":"https://codeload.github.com/TuringLang/ADTests/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TuringLang%2FADTests/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263741509,"owners_count":23504249,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-08T17:06:10.507Z","updated_at":"2026-01-30T11:24:57.832Z","avatar_url":"https://github.com/TuringLang.png","language":"Julia","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ADTests\n\nThis repository is used to tabulate the current performance of various automatic differentiation (AD) backends with Turing.jl models.\n\n## Let me see the results!\n\nhttps://turinglang.org/ADTests\n\n## I want to add more AD backends!\n\nYou can modify the list of AD types in `main.jl`.\n\n## I want to add more models!\n\nYou can modify the list of models by:\n\n1. **Adding a new call to `@include_model {category_heading} {model_name}` in `main.jl`.**\n   Both `category_heading` and `model_name` should be strings.\n\n   `category_heading` is used to determine which table the model appears under on the website.\n   This should be self-explanatory if you look at the [current website](https://turinglang.org/ADTests).\n   \n2. **Adding a new file, `models/{model_name}.jl`.**\n   \n   The basic structure of this file should look like this, where `model_name` is replaced accordingly:\n\n   ```julia\n   #=\n   (1) You can add any explanatory comments here if necessary\n   =#\n\n   # (2) Imports if necessary\n   using MyOtherPackage: some_function\n\n   # (3) Define data if necessary\n   data = ...\n\n   # (4) Define the model\n   @model function model_name(data, ...)\n       # Define your model here\n       ...\n   end\n\n   # (5) Instantiate the model\n   model = model_name(data, ...)\n   ```\n\n   **(1) Description**\n\n   Ideally, `model_name` would be self-explanatory, i.e. it would serve to illustrate exactly one feature and the name would indicate this.\n   However, if necessary, you can add further explanatory comments inside the model definition file.\n\n   **(2) Dependencies**\n\n   Inside this file, you do not need to call `using Turing` or any of the AD backends.\n   (This also means you do not need to import anything that Turing re-exports, such as distributions.)\n\n   However, you will have to make sure to import any other packages that your model requires.\n   (If this package is not already present in the project environment, you will also have to add it to `Project.toml`.)\n\n   **(3) Data definition**\n\n   Each file in `models/` is evaluated within its own module, so you can declare data variables, etc. without worrying about name clashes.\n\n   **(4) Model definition**\n\n   Models can be defined as usual with `@model function model_name(...)`.\n\n   **(5) Model instantiation**\n\n   The last line in the file should be the creation of the Turing model object using `model = model_name(...)`.\n   (It is mandatory for the model object to be called `model`.)\n\n\u003e [!IMPORTANT]  \n\u003e Note that for CI to run properly, `model_name` **must** be consistent between the following:\n\u003e \n\u003e - The name of the model itself i.e. `@model function model_name(...)`\n\u003e - The filename i.e. `models/model_name.jl`\n\u003e - The name of the model in `main.jl` i.e. `@include_model \"Category Heading\" \"model_name\"`\n\n(This setup does admittedly feel a bit fragile.\nUnfortunately I could not find a simpler way to get all the components (Julia, Python, web app) to work together in an automated fashion.\nHopefully it is a small price to pay for the ability to just add a new model and have it be automatically included on the website.)\n\n## I want to edit the website!\n\nThe website is a small Svelte app in the `web` directory.\n\nTo build this website locally, you will need to:\n\n1. Install `pnpm` if you don't already have it.\n   (`npm` is fine too, just replace `pnpm` with `npm run` in the commands below)\n\n2. Download the JSON files from the `gh-pages` branch, and place them in the `web/src/data` directory.\n   Currently, there are three JSON files: `adtests.json`, `manifest.json`, and `model_definitions.json`.\n   These represent the latest results from running the AD tests on CI.\n\n3. `cd web`\n\n4. `pnpm install`\n\n5. `pnpm dev`\n\n6. Open `http://localhost:5173` in your browser.\n\n## I want to see the HTML generated by a PR!\n\nThe latest workflow run across all PRs will be published to https://turinglang.org/ADTests/pr.\n\nThis is a bit messy, but works for now on the assumption that there aren't many PRs being worked on simultaneously.\n\n## What's going on?\n\nThe workflow is the most complicated part of this repository.\nThis section attempts to explain it from the 'bottom up'; if you prefer a 'top down' approach start by looking at the GitHub Action workflow, `.github/workflows/test.yml`.\n\nUnder the hood, the main thing that actually runs the AD tests / benchmarks is `main.jl`.\nYou can run `julia --project=. main.jl` and it will print some usage information.\nHowever, it is the Python script `ad.py` that controls how this Julia script is called.\n\nFundamentally, the idea is that we want to loop over every combination of model and adtype and test it.\nHowever, because GitHub Actions limits jobs to 6 hours, it is impractical to run every combination in the same job.\nWhat we do is to run one job per model.\nThis is accomplished by first storing the names of the models and adtypes in the `$GITHUB_OUTPUT` variable using `python ad.py setup`, which can then be read by the next job.\n\nThe next job is a CI matrix split by model name; each of the sub-jobs invokes `python ad.py run --model {model_name}`, which loops over each adtype and calls the Julia script with the model name and adtype as arguments.\nThe purpose of having this Python -\u003e Julia setup (as opposed to doing the looping inside Julia itself) is to guard against the Julia process crashing, which can happen sporadically with Enzyme.\nIf the Julia process successfully finishes, it will print the result which is picked up by the Python script; if it crashes, we just record the result as 'error'.\n\nFinally, the results are collated and sent to the final job in the workflow, which is `python ad.py html`.\nThis bit of the Python script is responsible for generating the three JSON files which the web app uses.\n(Fun fact: collating these results is also somewhat involved because we can't just write to `$GITHUB_OUTPUT`; it turns out that [output from different jobs in a matrix will override each other](https://github.com/orgs/community/discussions/26639), so the output can't share the same key, and there's [no way to dynamically specify the output key](https://github.com/actions/runner/pull/2477).\nThankfully, [there is an existing action](https://github.com/beacon-biosignals/matrix-output) which is designed to get around this problem by uploading artefacts instead of using `$GITHUB_OUTPUT`.)\n\nOverall, what this means is that the entire table can be generated in around 10 minutes (longer if you need to install + precompile dependencies, but on GitHub Actions dependencies will for the most part have been cached).\n\n## Can I run this locally?\n\nIf you just want to run one specific combination of model and adtype, you can run `julia --project=. main.jl --run \u003cmodel\u003e \u003cadtype\u003e`.\n\nIf you want to run one model with all adtypes, you can run `uv run ad.py run --model \u003cmodel_name\u003e`.\n\nYou probably don't want to run all models with all adtypes, as that takes a really long time.\n\nIf you just want to build the website using results that were obtained from CI, that's described in the section above.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fturinglang%2Fadtests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fturinglang%2Fadtests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fturinglang%2Fadtests/lists"}