{"id":20670493,"url":"https://github.com/nyu-systems/gauntlet","last_synced_at":"2026-03-07T09:32:30.558Z","repository":{"id":37895804,"uuid":"187662797","full_name":"nyu-systems/gauntlet","owner":"nyu-systems","description":"Finding bugs in P4 compilers using translation validation.","archived":false,"fork":false,"pushed_at":"2025-10-05T17:46:26.000Z","size":24794,"stargazers_count":37,"open_issues_count":12,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-10-05T19:30:36.257Z","etag":null,"topics":["p4","p4c","translation-validation","z3"],"latest_commit_sha":null,"homepage":"","language":"P4","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nyu-systems.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-05-20T15:01:58.000Z","updated_at":"2025-10-05T17:46:30.000Z","dependencies_parsed_at":"2024-06-29T18:54:48.583Z","dependency_job_id":"6024d9a2-7589-4d6b-83bc-3858d97387a0","html_url":"https://github.com/nyu-systems/gauntlet","commit_stats":null,"previous_names":["nyu-systems/gauntlet"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/nyu-systems/gauntlet","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyu-systems%2Fgauntlet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyu-systems%2Fgauntlet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyu-systems%2Fgauntlet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyu-systems%2Fgauntlet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nyu-systems","download_url":"https://codeload.github.com/nyu-systems/gauntlet/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyu-systems%2Fgauntlet/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30210842,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T09:02:10.694Z","status":"ssl_error","status_checked_at":"2026-03-07T09:02:08.429Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["p4","p4c","translation-validation","z3"],"created_at":"2024-11-16T20:21:03.846Z","updated_at":"2026-03-07T09:32:30.519Z","avatar_url":"https://github.com/nyu-systems.png","language":"P4","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n[![Build Status](https://github.com/nyu-systems/gauntlet/actions/workflows/pytest.yaml/badge.svg)](https://github.com/nyu-systems/gauntlet/actions/workflows/pytest.yaml)\n\n# The Gauntlet Tool Suite\n\n- [Introduction](#introduction)\n- [Requirements](#requirements)\n  * [Frameworks for Model-Based Testing](#frameworks-for-model-based-testing)\n- [Instructions](#instructions)\n  * [Generating a Random Program](#generating-a-random-program)\n  * [Retrieving Gauntlet Semantics for a P4 Program](#retrieving-gauntlet-semantics-for-a-p4-program)\n  * [Validating a P4C Program](#validating-a-p4c-program)\n  * [Model-Based Testing](#model-based-testing)\n  * [Fuzz-Testing at Scale](#fuzz-testing-at-scale)\n- [Fuzz-Testing Support Matrix](#fuzz-testing-support-matrix)\n- [Bugs Found in P4 Compilers](#bugs-found-in-p4-compilers)\n- [Citing This Project](#citing-this-project)\n\n## Introduction\n**DISCLAIMER**: This project has switched to a C++-based interpreter, which is not as feature-complete. For example, parser loops and the core extern functions are not implemented yet. The parser semantics are also not well tested. If you are interested in the original and comprehensive Python-based interpreter, please check out the [old](https://github.com/p4gauntlet/gauntlet/tree/gauntlet_old) branch.\n\nGauntlet is a set of tools designed to find bugs in programmable data-plane compilers. More precisely, Gauntlet targets the\n[P4 language](https://p4.org/) ecosystem and  the P4-16 reference compiler ([p4c](https://github.com/p4lang/p4c/)).\n\nThe goal is to ensure that a P4 compiler correctly translates a given input P4 program to its target-specific binary. The compiler must not crash and preserve the semantics of the program as originally written. The suite has three major components:\n\n1. **Bludgeon**, a fuzz tester that generates random P4 programs using libraries repurposed from `p4c`.\n\n2.  **Translation Validation**, which analyzes the intermediate representation of a program after each compiler pass and identifies potential discrepancies. We support translation validation for the open-source p4c compiler front- and mid-end libraries.\n\n3. **Model-based Testing**, which infers input and and corresponding output for a particular P4 program and generates end-to-end test packets. We have currently implemented model-based testing for the [bmv2 simple-switch](https://github.com/p4lang/behavioral-model) and the Tofino hardware switch.\n\nFor more details and a broad overview of the concepts in Gauntlet, refer to our [OSDI paper](https://www.usenix.org/conference/osdi20/presentation/ruffy).\n\n##  Requirements\nThis repository run best with a recent version of Ubuntu (22.04). The minimum required Python version is 3.6 ([f-strings](https://www.python.org/dev/peps/pep-0498/)).\n\nAll tools require `p4c` to be installed. The fuzz tester and P4-to-Z3 converter are also p4c extensions which need to be copied or symlinked into the `extensions` folder of the compiler. The `do_install.sh` contains detailed command instructions. Most dependencies can be installed by running `./do_install.sh` in the source folder (**Careful**, the installation assumes root privileges and installs several large packages).\n\nTo check whether everything has been installed correctly you can run `python3 -m pytest test.py -vrf`. This will take about 30 minutes.\n\n###  Frameworks for Model-based Testing\nModel-based testing requires a full test harness. Gauntlet currently supports the [bmv2 simple-switch](https://github.com/p4lang/behavioral-model) and the Tofino packet test framework. The behavioral model can be installed running the installation script with the option `./do_install.sh INSTALL_BMV2=ON`.\n\nThe Tofino test framework requires access to the SDK and a manual setup. Gauntlet's scripts assume that the folder is installed under `tofino/bf_src`. We typically run the installation script as `./tofino/bf_src/p4studio_build/p4studio_build.py --use-profile p416_examples_profile`.\n\n\n## Instructions\n### Generating a Random Program\nAfter successful installation, you can generate a random P4 program via the `modules/p4c/build/p4bludgeon out.p4 --arch top`  command. To generate Tofino code, the flag needs to be set to  `modules/p4c/build/p4bludgeon --output out.p4 --arch tna`.\nA typical crash checking workflow might be:\n\n    modules/p4c/build/p4bludgeon --output out.p4 --arch top \u0026\u0026 modules/p4c/build/p4test out.p4\n\n### Retrieving Gauntlet Semantics for a P4 Program\nFor debugging purposes, you can run\n\n    bin/get_p4_semantics out.p4\n\nto retrieve the semantic representation of a particular P4 program. This will print the Z3 formula of each pipe in the package. These semantics can be used for equality comparison or test-case inference.\n\n### Validating a P4C Program\nTo validate that a program is compiled correctly by `p4c`, you can run\n\n     modules/p4c/build/p4bludgeon --output out.p4 --arch top \u0026\u0026 bin/validate_p4_translation out.p4\n`bin/validate_p4_translation` checks if a sequence of P4 programs are all equivalent to each other using the `bin/check_prog_equality` program as a sub routine. This sequence is produced by running p4c on an input P4 program. When p4c is run on an input P4 program, it produces a sequence of P4 programs, where each P4 program corresponds to the version of the input P4 program after a p4c optimization pass. This allows us to validate whether compilation/translation is working correctly and to pinpoint the faulty optimization pass if it isn't\nworking correctly.\n\n### Model-Based Testing [DEPRECATED]\n\nModel-based testing requires the behavioral model or the Tofino compiler to be installed. The correct binaries and include files need to be instrumented in the `src/generate_p4_test.py` file. An example command is\n\n     modules/p4c/build/p4bludgeon --output out.p4 --arch v1model \u0026\u0026 bin/generate_test_case -i out.p4 -r\nThis sequence of commands will first generate a random program, infer expected input and output values, convert them to a test file (in this case, they are stf files) and finally run a full test. If the observed output differs from the expected output, the tool will throw  an error. The `-r` flag denotes randomization of the input, it is optional.\nTo run model-based testing for the Tofino back end, `sudo` will have to be used.\n\n     modules/p4c/build/p4bludgeon --output out.p4 --arch tna \u0026\u0026 sudo -E bin/generate_test_case -i out.p4 -r --arch tna\n\n### Fuzz-Testing at Scale\nWe also include facilities to fuzz test the compilers at scale.\n\n    bin/test_random_progs -i 1000\n To generate and compile a thousand programs using P4C's `p4test`.\n\n    sudo -E bin/test_random_progs -i 1000 --arch tna\n\n To generate and compile a thousand programs using the Tofino compiler.\n\n     bin/test_random_progs -i 1000 -v\n\n To compile and validate a thousand programs using P4C's `p4test`.\n\n     bin/test_random_progs -i 1000 --arch v1model -b\n\n To generate and fuzz test a thousand programs on the simple switch.\n\n     sudo -E bin/test_random_progs -i 1000 --arch tna -b\n\n To generate and fuzz test a thousand programs on the Tofino compiler.\n\n## Fuzz-Testing Support Matrix\n\n| Architecture | Compiler | Bludgeon Support | Validation Testing | Model-based Testing |\n| ------------- | ------------- | ------------- | ------------- | ------------- |\n| psa | `p4c-bm2-psa` | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |\n| tna | `p4c-bf` | :heavy_check_mark: | :x: | :heavy_check_mark: |\n| top | `p4test` | :heavy_check_mark: | :heavy_check_mark: | :x: |\n| v1model | `p4c-bm2-ss` | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |\n\n## Bugs Found in P4 Compilers\n\nWe also track the bugs we have found. A detailed breakdown can be found in the [bugs](bugs) folder.\n\n## Citing This Project\n\nTo cite our work please refer to our paper:\n\n```tex\n@inproceedings {gauntlet,\n  title = {Gauntlet: Finding Bugs in Compilers for Programmable Packet Processing},\n  booktitle = {14th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 20)},\n  year = {2020},\n  url = {https://www.usenix.org/conference/osdi20/presentation/ruffy},\n  publisher = {{USENIX} Association},\n  month = nov,\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnyu-systems%2Fgauntlet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnyu-systems%2Fgauntlet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnyu-systems%2Fgauntlet/lists"}