{"id":19055932,"url":"https://github.com/schemathesis/web-api-fuzzing-project","last_synced_at":"2025-07-06T14:06:50.880Z","repository":{"id":37100321,"uuid":"354606912","full_name":"schemathesis/web-api-fuzzing-project","owner":"schemathesis","description":null,"archived":false,"fork":false,"pushed_at":"2023-07-01T13:29:25.000Z","size":59813,"stargazers_count":13,"open_issues_count":10,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-24T04:32:48.423Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/schemathesis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-04-04T17:35:40.000Z","updated_at":"2025-02-15T09:34:07.000Z","dependencies_parsed_at":"2025-04-13T12:26:56.696Z","dependency_job_id":null,"html_url":"https://github.com/schemathesis/web-api-fuzzing-project","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/schemathesis/web-api-fuzzing-project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schemathesis%2Fweb-api-fuzzing-project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schemathesis%2Fweb-api-fuzzing-project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schemathesis%2Fweb-api-fuzzing-project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schemathesis%2Fweb-api-fuzzing-project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/schemathesis","download_url":"https://codeload.github.com/schemathesis/web-api-fuzzing-project/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schemathesis%2Fweb-api-fuzzing-project/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263914453,"owners_count":23529078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T23:47:29.291Z","updated_at":"2025-07-06T14:06:50.863Z","avatar_url":"https://github.com/schemathesis.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web API Fuzzing Project (WAFP)\n\nThe WAFP project is a test suite for evaluating various characteristics of Web API fuzzers.\nWAFP is fully runnable as a CLI tool that spins up fuzzing targets \u0026 runs fuzzers against them.\n\n## Citation\n\nIf you use WAFP in research, please cite our paper\n[*Deriving Semantics-Aware Fuzzers from Web API Schemas*](https://arxiv.org/abs/2112.10328)\nby Zac Hatfield-Dodds (@Zac-HD) and Dmitry Dygalo (@Stranger6667) - we built it to\nevaluate Schemathesis, it's designed to be extensible.  Our goal was to make future\nstudies as easy -- and easy to compare -- as possible.\n\nUse it as-is, or extend it and contribute new tools, targets, or integrations\nback to our repo so that others can benefit from your hard work!\n\nIf you just want to grab results, see Zenodo: [unprocessed data (23 GB)](https://zenodo.org/record/5339649)\nand [processed data (263 MB)](https://zenodo.org/record/5392010).\n\n## Installation\n\nWAFP is built around Docker and is tested against the `20.10.0` version. Check [the official Docker docs](https://docs.docker.com/get-docker/) for installation guide.\nOther dependencies are managed via `poetry` (check out the [installation guide](https://github.com/sdispater/poetry#installation)):\n\n```\npoetry install\n```\n\nIt also automatically installs WAFP CLI to the current environment that is available via the `wafp` entry point.\n\n## Getting started\n\nTo run a fuzzer against a target, you need to pass their names in CLI:\n\n```\nwafp schemathesis:Default jupyter_server:Default --output-dir=./artifacts\n```\n\nThe command above will run the `Default` variant of Schemathesis against the Jupyter Server target and will store\nall available artifacts in the `./artifacts` directory.\n\nAlternatively you can run it via ``poetry``:\n\n```\npoetry run wafp schemathesis:Default jupyter_server --output-dir=./artifacts\n```\n\nIf you want to run the whole suite, use the `run.py` script:\n\n```\npython run.py --output-dir=./artifacts --iterations=30\n```\n\nIt will run all the defined combinations for 30 times and store the artifacts in the `./artifacts` directory.\nThe combinations are defined in the `COMBINATIONS` variable in the `run.py` file. It excludes combinations that are known\nto not work for some reason (usually due to fuzzer failures).\n\n## Fuzzing targets\n\nEvery fuzzing target is a web application that runs via `docker-compose`. WAFP provides an API on top of\n`docker-compose` that allows fuzzers to work with targets in a unified fashion.\n\nA target is represented as a directory with at least two components:\n\n- `__init__.py` file. Contains target's Python API \u0026 metadata;\n- `docker-compose.yml` file. Docker-compose project for the target.\n\nBut generally, there could be any dependencies needed to build a `docker-compose` project.\nAll available targets are located in `src/wafp/targets/catalog`.\n\nYou can run targets with the following command (replace `\u003ctarget-name\u003e` with any target name from the catalog):\n\n```\npython -m wafp.targets \u003ctarget-name\u003e --output-dir=./artifacts\n```\n\n### Target structure\n\nPython API for a target consists of one or more classes inherited from `wafp.targets.BaseTarget`. Each class requires\nimplementing at least four methods:\n\n- `get_base_url`. Service base URL. All URLs used in API calls will extend this value;\n- `get_schema_location`. URL or a filesystem path to the API schema;\n- `is_ready`. Detects whether the target is ready for fuzzing. It is called on each stdout line emitted by the `docker-compose` stack;\n- `get_metadata`. Describes the programming language, API schema type, and other meta information.\n\nTargets are parametrized with TCP ports, and by default, they start working on a random port that is available via the `port` attribute.\n\nHere is an example of the `httpbin` target:\n\n```python\nfrom wafp.targets import BaseTarget, Metadata\n\n\nclass Default(BaseTarget):\n    def get_base_url(self) -\u003e str:\n        # A common case that has no additional path\n        return f\"http://0.0.0.0:{self.port}/\"\n\n    def get_schema_location(self) -\u003e str:\n        return f\"http://0.0.0.0:{self.port}/spec.json\"\n\n    def is_ready(self, line: bytes) -\u003e bool:\n        return b\"Listening at: \" in line\n\n    def get_metadata(self) -\u003e Metadata:\n        return Metadata.flasgger(\n            flask_version=\"1.0.2\",\n            flasgger_version=\"0.9.0\",\n            openapi_version=\"2.0\",\n            validation_from_schema=False,\n        )\n```\n\nDocker-compose:\n\n```\nversion: '3'\nservices:\n  web:\n    build:\n      context: https://github.com/postmanlabs/httpbin.git#f8ec666b4d1b654e4ff6aedd356f510dcac09f83\n    init: true\n    environment:\n      - PORT=3000\n    ports:\n      - '${PORT-3000}:80'\n```\n\nCompose files should support the `PORT` environment variable and provide a proper port mapping.\n\nRunning the target from the example above:\n\n```python\ntarget = Default()\ntarget.start()\n# ... Run fuzzing ...\ntarget.stop()\ntarget.cleanup()\n```\n\nSome targets may require additional actions to be prepared for fuzzing, for example, creating a user and getting credentials.\nYou can extract headers from `docker-compose` output via the `get_headers` method:\n\n```python\nimport re\n...\n\nclass Default(BaseTarget):\n    ...\n    def get_headers(self, line: bytes) -\u003e Dict[str, str]:\n        match = re.search(b\"token=(.+)\", line)\n        if match is None:\n            return {}\n        token = match.groups()[0]\n        return {\"Authorization\": f\"token {token.decode()}\"}\n```\n\nCredentials can be obtained in the `after_start` hook. At this moment, the target is ready to accept network requests:\n\n```python\nimport requests\n...\n\nclass Default(BaseTarget):\n    ...\n    def after_start(self, stdout: bytes, headers: Dict[str, str]) -\u003e None:\n        base_url = self.get_base_url()\n        # Authorize \u0026 get the token\n        response = requests.post(\n            f\"{base_url}/authorizations/token\",\n            json={\"username\": \"root\", \"password\": \"test\"}\n        )\n        token = response.json()[\"token\"]\n        headers[\"Authorization\"] = f\"token {token}\"\n```\n\n### Sentry integration\n\nSome targets provide Sentry integration, and it is possible to collect all errors reported during a fuzzing run.\nTo enable the integration, you need to pass the `sentry_dsn` argument during the target initialization or provide the `--sentry-dsn` CLI option.\nTo collect errors from the used Sentry instance you need to provide more info:\n\n```python\n# Target initialization\ntarget = target.Default(\n    sentry_dsn=\"https://c4715cd284cf4f509c32e49f27643f30@sentry.company.com/42\"\n)\n# Load all artifacts including errors reported to Sentry\nartifacts = target.collect_artifacts(\n    # Your Sentry instance base URL\n    sentry_url=\"https://sentry.company.com\",\n    # Sentry access token\n    sentry_token=\"7a7d025aafe34326b789356b62d2b6dc01af594c33ca48a3a0f76421a137ef9a\",\n    # The slug of the organization the target project belongs to\n    sentry_organization=\"my_org\",\n    # The slug of the project\n    sentry_project=\"target\",\n)\n```\n\nThe `artifacts` variable will contain container logs and Sentry events as Python dictionaries wrapped into the `Artifact` class.\n\nWAFP uses the `GET /api/0/projects/{organization_slug}/{project_slug}/events/` endpoint to retrieve events data.\nSee more info in Sentry documentation - https://docs.sentry.io/api/events/list-a-projects-events/\n\nIf you'd like to use the `run.py` file to run all combinations, you'll need to add `sentry_dsn` keys to the desired combinations in the `COMBINATIONS` variable in the `run.py` file.\n\nAs Sentry does not process events immediately, you'll need to download them separately, when the processing is done in your Sentry instance.\n\nTo load the events you need the latest stable Rust version (see the [rustup](https://rustup.rs/) docs for the installation instructions) and run the following command in the `sentry_events` directory:\n\n```\ncargo run --release \u003cpath-to-artifacts\u003e --token \u003cyour Sentry API token\u003e --url \u003cyour Sentry instance URL\u003e\n```\n\nIt will load all the events relevant to the artifacts and store them in the same artifacts directory. Note, it might take a while to download all the events.\n\n## Fuzzers\n\nAPI fuzzers are also run via `docker-compose` and are available via a similar interface:\n\n```\npython -m wafp.fuzzers schemathesis:Default \\\n  --schema=\u003cSchema file or URL\u003e \\\n  --base-url=\u003cService base URL\u003e \\\n  --output-dir=./artifacts\n```\n\nEach fuzzer can be represented as one or more variants - you can have different running modes as different variants.\nFor example, there are four different variants for Schemathesis:\n\n- `schemathesis:Default` - checks only for 5xx HTTP response codes\n- `schemathesis:AllChecks` - runs all available checks\n- `schemathesis:StatefulOld` - additionally execute stateful tests via Schemathesis's deprecated approach\n- `schemathesis:StatefulNew` - utilizes the state-machine-based stateful testing\n\nFuzzers' names are derived from Python packages they are in - you can find them in the `./src/wafp/fuzzers/catalog` directory.\n\n## Artifacts processing\n\nTo process the artifacts you need the latest stable Rust version (see the [rustup](https://rustup.rs/) docs for the installation instruction).\n\nRun the following command in the `postprocessing` directory:\n\n```\ncargo run --release \u003cpath-to-artifacts\u003e \u003coutput-directory\u003e\n```\n\nThe output directory will have the same top-level structure as the input one. Sub-directories named by the following pattern - `\u003cfuzzer\u003e-\u003ctarget\u003e-\u003citeration-number\u003e`. Then, each directory may have the following files:\n\n- `metadata.json`. Metadata about a test run - tested fuzzer name, run duration, etc\n- `fuzzer.json` - Structured fuzzer output\n- `deduplicated_cases.json` - Deduplicated reported failures, when fuzzers provide it\n- `sentry.json` - Cleaned Sentry events for this run\n- `target.json` - Parsed stdout for Gitlab \u0026 Disease.sh targets that are tested without Sentry integration\n\n## Related projects\n\n- [HypoFuzz](https://hypofuzz.com/). Putting smart fuzzing into the world's best testing workflow for Python. HypoFuzz runs your property-based test suite, using cutting-edge fuzzing techniques and coverage instrumentation to find even the rarest inputs which trigger an error.\n- [Schemathesis.io](https://schemathesis.io/). A modern API testing tool that allows you to find bugs faster without leaving your browser. Schemathesis.io be available soon!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschemathesis%2Fweb-api-fuzzing-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fschemathesis%2Fweb-api-fuzzing-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschemathesis%2Fweb-api-fuzzing-project/lists"}