{"id":34497351,"url":"https://github.com/coreruleset/go-ftw","last_synced_at":"2026-01-14T22:18:01.517Z","repository":{"id":39619304,"uuid":"325886645","full_name":"coreruleset/go-ftw","owner":"coreruleset","description":"Web Application Firewall Testing Framework - Go version","archived":false,"fork":false,"pushed_at":"2026-01-12T21:11:27.000Z","size":1377,"stargazers_count":161,"open_issues_count":25,"forks_count":36,"subscribers_count":11,"default_branch":"main","last_synced_at":"2026-01-12T23:02:58.317Z","etag":null,"topics":["testing","testing-tools","waf"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coreruleset.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-12-31T23:09:12.000Z","updated_at":"2026-01-08T02:57:14.000Z","dependencies_parsed_at":"2025-12-29T03:03:49.627Z","dependency_job_id":null,"html_url":"https://github.com/coreruleset/go-ftw","commit_stats":null,"previous_names":["fzipi/go-ftw"],"tags_count":41,"template":false,"template_full_name":null,"purl":"pkg:github/coreruleset/go-ftw","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coreruleset%2Fgo-ftw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coreruleset%2Fgo-ftw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coreruleset%2Fgo-ftw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coreruleset%2Fgo-ftw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coreruleset","download_url":"https://codeload.github.com/coreruleset/go-ftw/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coreruleset%2Fgo-ftw/sbom","scorecard":{"id":304855,"data":{"date":"2025-08-13T05:38:47Z","repo":{"name":"github.com/coreruleset/go-ftw","commit":"6422f0a0c0f1bbbfff4a9839a84c7bba841713ed"},"scorecard":{"version":"v5.2.1","commit":"ab2f6e92482462fe66246d9e32f642855a691dc1"},"score":8.5,"checks":[{"name":"Code-Review","score":10,"reason":"all changesets reviewed","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#binary-artifacts"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":10,"reason":"GitHub workflow tokens follow principle of least privilege","details":["Info: jobLevel 'actions' permission set to 'read': .github/workflows/codeql-analysis.yml:29","Info: jobLevel 'contents' permission set to 'read': .github/workflows/codeql-analysis.yml:30","Info: jobLevel 'actions' permission set to 'read': .github/workflows/codeql.yml:29","Info: jobLevel 'contents' permission set to 'read': .github/workflows/codeql.yml:30","Info: jobLevel 'contents' permission set to 'read': .github/workflows/sonar.yaml:12","Info: jobLevel 'pull-requests' permission set to 'read': .github/workflows/sonar.yaml:13","Info: topLevel 'contents' permission set to 'read': .github/workflows/codeql-analysis.yml:24","Info: topLevel 'contents' permission set to 'read': .github/workflows/codeql.yml:24","Info: topLevel permissions set to 'read-all': .github/workflows/release.yml:9","Info: topLevel permissions set to 'read-all': .github/workflows/scorecard.yml:18","Info: topLevel 'contents' permission set to 'read': .github/workflows/sonar.yaml:7","Info: topLevel 'contents' permission set to 'read': .github/workflows/test.yml:10","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#token-permissions"}},{"name":"Maintained","score":10,"reason":"23 commit(s) and 5 issue activity found in the last 90 days -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#maintained"}},{"name":"Security-Policy","score":10,"reason":"security policy file detected","details":["Info: security policy file detected: SECURITY.md:1","Info: Found linked content: SECURITY.md:1","Info: Found disclosure, vulnerability, and/or timelines in security policy: SECURITY.md:1","Info: Found text in security policy: SECURITY.md:1"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#security-policy"}},{"name":"Dependency-Update-Tool","score":10,"reason":"update tool detected","details":["Info: detected update tool: RenovateBot: renovate.json:1"],"documentation":{"short":"Determines if the project uses a dependency update tool.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#dependency-update-tool"}},{"name":"Pinned-Dependencies","score":10,"reason":"all dependencies are pinned","details":["Info:  16 out of  16 GitHub-owned GitHubAction dependencies pinned","Info:   6 out of   6 third-party GitHubAction dependencies pinned","Info:   1 out of   1 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#pinned-dependencies"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#license"}},{"name":"CII-Best-Practices","score":2,"reason":"badge detected: InProgress","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#cii-best-practices"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":10,"reason":"SAST tool is run on all commits","details":["Info: SAST configuration detected: CodeQL","Info: SAST configuration detected: CodeQL","Info: all commits (30) are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#sast"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact v1.3.0 not signed: https://api.github.com/repos/coreruleset/go-ftw/releases/200508223","Warn: release artifact v1.2.0 not signed: https://api.github.com/repos/coreruleset/go-ftw/releases/193219292","Warn: release artifact v1.1.2 not signed: https://api.github.com/repos/coreruleset/go-ftw/releases/187743166","Warn: release artifact v1.1.1 not signed: https://api.github.com/repos/coreruleset/go-ftw/releases/183364184","Warn: release artifact v1.1.0 not signed: https://api.github.com/repos/coreruleset/go-ftw/releases/180305910","Warn: release artifact v1.3.0 does not have provenance: https://api.github.com/repos/coreruleset/go-ftw/releases/200508223","Warn: release artifact v1.2.0 does not have provenance: https://api.github.com/repos/coreruleset/go-ftw/releases/193219292","Warn: release artifact v1.1.2 does not have provenance: https://api.github.com/repos/coreruleset/go-ftw/releases/187743166","Warn: release artifact v1.1.1 does not have provenance: https://api.github.com/repos/coreruleset/go-ftw/releases/183364184","Warn: release artifact v1.1.0 does not have provenance: https://api.github.com/repos/coreruleset/go-ftw/releases/180305910"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#signed-releases"}},{"name":"Packaging","score":10,"reason":"packaging workflow detected","details":["Info: Project packages its releases by way of GitHub Actions.: .github/workflows/release.yml:12"],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#packaging"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#branch-protection"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#fuzzing"}},{"name":"CI-Tests","score":10,"reason":"30 out of 30 merged PRs checked by a CI test -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project runs tests before pull requests are merged.","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#ci-tests"}},{"name":"Contributors","score":10,"reason":"project has 13 contributing companies or organizations","details":["Info: found contributions from: MuNuChapterHKN, SeasideSt, apache, corazawaf, coreruleset, earth, facultad de ingeniería universidad de la república, https://www.netnea.com, owasp-modsecurity, pharo-vcs, tetrateio, tetratelabs, xovis.com"],"documentation":{"short":"Determines if the project has a set of contributors from multiple organizations (e.g., companies).","url":"https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#contributors"}}]},"last_synced_at":"2025-08-17T21:47:01.489Z","repository_id":39619304,"created_at":"2025-08-17T21:47:01.489Z","updated_at":"2025-08-17T21:47:01.489Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28436268,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T21:32:52.117Z","status":"ssl_error","status_checked_at":"2026-01-14T21:32:33.442Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["testing","testing-tools","waf"],"created_at":"2025-12-24T01:00:50.272Z","updated_at":"2026-01-14T22:18:01.511Z","avatar_url":"https://github.com/coreruleset.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Go-FTW - Framework for Testing WAFs in Go!\n\n[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white)](https://github.com/pre-commit/pre-commit)\n[![Go Report Card](https://goreportcard.com/badge/github.com/coreruleset/go-ftw)](https://goreportcard.com/report/github.com/coreruleset/go-ftw)\n[![Go Doc](https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square)](http://godoc.org/github.com/coreruleset/go-ftw)\n[![PkgGoDev](https://pkg.go.dev/badge/github.com/coreruleset/go-ftw)](https://pkg.go.dev/github.com/coreruleset/go-ftw)\n[![Release](https://img.shields.io/github/v/release/coreruleset/go-ftw.svg?style=flat-square)](https://github.com/coreruleset/go-ftw/releases/latest)\n[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=coreruleset_go-ftw\u0026metric=coverage)](https://sonarcloud.io/dashboard?id=coreruleset_go-ftw)\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=coreruleset_go-ftw\u0026metric=alert_status)](https://sonarcloud.io/dashboard?id=coreruleset_go-ftw)\n[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/coreruleset/go-ftw/badge)](https://securityscorecards.dev/viewer/?uri=github.com/coreruleset/go-ftw)\n\nGo-FTW is a replacement for [FTW](https://pypi.org/project/ftw/) which had reached its limits in terms of maintainability and performance.\n\nFeatures of Go-FTW include:\n  - fully customizable HTTP traffic\n  - CI/CD friendly\n  - fast\n  - syntax checking of test files\n\n## Install\n\nGo to the [releases](https://github.com/coreruleset/go-ftw/releases) page and get a binary release that matches your OS (scroll down to **Assets**).\n\nIf you have Go installed and configured to run Go binaries from your shell you can also run\n```bash\ngo install github.com/coreruleset/go-ftw@latest\n```\n\n## Example Usage\n\nThe go-ftw is designed to run Web Application Firewall (WAF) unit tests. The primary focus is the [OWASP ModSecurity Core Rule Set](https://github.com/coreruleset/coreruleset).\n\nIn order to run the tests, you need to prepare the following:\n\n1. Active WAF\n2. Log where the WAF writes the alert messages\n3. go-ftw config file `.ftw.yaml` in the local folder or in your home folder (see [YAML Config file](https://github.com/coreruleset/go-ftw#yaml-config-file) for more information).\n4. At least one unit test in (go)-ftw's yaml format.\n\n### YAML Config file\n\nWith the configuration, you can set paths for your environment, enable and disable features and you can also use it to alter the test results.\n\nThe config file has six basic settings:\n\n* `logfile` : path to WAF log with alert messages, relative or absolute\n* `testoverride` : a list of things to override (see [Overriding tests](https://github.com/coreruleset/go-ftw#overriding-tests) below)\n* `mode` : \"default\" or \"cloud\" (only change it if you need \"cloud\")\n* `logmarkerheadername` : name of an HTTP header used for marking log messages, usually `X-CRS-TEST` (see [How log parsing works](https://github.com/coreruleset/go-ftw#how-log-parsing-works) below)\n* `maxmarkerretries` : the maximum number of times the search for log markers will be repeated; each time an additional request is sent to the web server, eventually forcing the log to be flushed\n* `maxmarkerloglines` the maximum number of lines to search for a marker before aborting\n\nYou can probably leave the last three alone, they are set to sane defaults.\n\n__Example with absolute logfile__:\n\n```yaml\nlogfile: /apache/logs/error.log\nlogmarkerheadername: X-CRS-TEST\ntestoverride:\nmode: \"default\"\n```\n\n__Example with relative logfile__:\n\n```yaml\nlogfile: ../logs/error.log\nlogmarkerheadername: X-CRS-TEST\ntestoverride:\nmode: \"default\"\n```\n\n__Example with minimal definitions__:\n\nThe minimal requirement for go-ftw is to have a logfile when running in default mode:\n\n```yaml\nlogfile: ../logs/error.log\n```\n\nBy default, _go-ftw_ looks for a file in `$PWD` / local folder with the name `.ftw.yaml`. If this can not be found, it will look in the user's HOME folder. You can pass the `--config \u003cconfig file name\u003e` to point it to a different file.\n\n### WAF Server\n\nI normally perform my testing using the [Core Rule Set](https://github.com/coreruleset/coreruleset/).\n\nYou can start the containers from that repo using `docker compose`:\n\n```bash\ngit clone https://github.com/coreruleset/coreruleset.git\ncd coreruleset\ndocker compose -f tests/docker-compose.yml up -d modsec2-apache\n```\n\n### Logfile\n\nRunning in default mode implies you have access to a logfile for checking the WAF behavior against test results. For this example, assuming you are in the base directory of the coreruleset repository, these are the configurations for `apache` and `nginx`:\n\n```yaml\n---\nlogfile: 'tests/logs/modsec2-apache/error.log'\n```\n\n```yaml\n---\nlogfile: 'tests/logs/modsec3-nginx/error.log'\n```\n\n## Running\n\nThis is the help for the `run` command:\n```bash\n./ftw run --help\nRun all tests below a certain subdirectory. The command will search all y[a]ml files recursively and pass it to the test engine.\n\nUsage:\n  ftw run [flags]\n\nFlags:\n      --connect-timeout duration               timeout for connecting to endpoints during test execution (default 3s)\n  -d, --dir string                             recursively find yaml tests in this directory (default \".\")\n  -e, --exclude string                         exclude tests matching this Go regular expression (e.g. to exclude all tests beginning with \"91\", use \"^91.*\").\n                                               If you want more permanent exclusion, check the 'exclude' option in the config file.\n      --fail-fast                              Fail on first failed test\n  -f, --file string                            output file path for ftw tests. Prints to standard output by default.\n  -h, --help                                   help for run\n  -i, --include string                         include only tests matching this Go regular expression (e.g. to include only tests beginning with \"91\", use \"^91.*\").\n                                               If you want more permanent inclusion, check the 'include' option in the config file.\n  -T, --include-tags string                    include tests tagged with labels matching this Go regular expression (e.g. to include all tests being tagged with \"cookie\", use \"^cookie$\").\n  -l, --log-file string                        path to log file to watch for WAF events\n      --max-marker-log-lines int               maximum number of lines to search for a marker before aborting (default 500)\n      --max-marker-retries int                 maximum number of times the search for log markers will be repeated.\n                                               Each time an additional request is sent to the web server, eventually forcing the log to be flushed (default 20)\n  -o, --output string                          output type for ftw tests. \"normal\" is the default. (default \"normal\")\n  -r, --rate-limit duration                    Limit the request rate to the server to 1 request per specified duration. 0 is the default, and disables rate limiting.\n      --read-timeout duration                  timeout for receiving responses during test execution (default 10s)\n      --show-failures-only                     shows only the results of failed tests\n      -skip-tls-verification                   Skips TLS certificate checks. Useful for testing domains with self-signed TLS ceritificates.\n  -t, --time                                   show time spent per test\n      --wait-delay duration                    Time to wait between retries for all wait operations. (default 1s)\n      --wait-for-connection-timeout duration   Http connection timeout, The timeout includes connection time, any redirects, and reading the response body. (default 3s)\n      --wait-for-expect-body-json string       Expect response body JSON pattern.\n      --wait-for-expect-body-regex string      Expect response body pattern.\n      --wait-for-expect-body-xpath string      Expect response body XPath pattern.\n      --wait-for-expect-header string          Expect response header pattern.\n      --wait-for-expect-status-code int        Expect response code e.g. 200, 204, ... .\n      --wait-for-host string                   Wait for host to be available before running tests.\n      --wait-for-no-redirect                   Do not follow HTTP 3xx redirects.\n      --wait-for-timeout duration              Sets the timeout for all wait operations, 0 is unlimited. (default 10s)\n\nGlobal Flags:\n      --cloud              cloud mode: rely only on HTTP status codes for determining test success or failure (will not process any logs)\n      --config string      specify config file (default is $PWD/.ftw.yaml)\n      --debug              debug output\n      --overrides string   specify file with platform specific overrides\n      --trace              trace output: really, really verbose\n```\nAll the wait for flags are implemented using the [wait4x](https://github.com/atkrad/wait4x#http) library.\nSee their examples on how to use them. In our flags we added the prefix `--wait-for` but they behave similarly.\n\nNote: Duration flags above accept any input valid for [`time.ParseDuration`](https://pkg.go.dev/time#ParseDuration).\n\nHere's an example on how to run your tests recursively in the folder `tests`:\n\n```bash\nftw run -d tests -t\n```\n\nAnd the result should be similar to:\n\n```bash\n❯ ./ftw run -d tests -t\n\n🛠️  Starting tests!\n🚀 Running!\n👉 executing tests in file 911100.yaml\n\trunning 911100-1: ✔ passed 6.382692ms\n\trunning 911100-2: ✔ passed 4.590739ms\n\trunning 911100-3: ✔ passed 4.833236ms\n\trunning 911100-4: ✔ passed 4.675082ms\n\trunning 911100-5: ✔ passed 3.581742ms\n\trunning 911100-6: ✔ passed 6.426949ms\n...\n\trunning 944300-322: ✔ passed 13.292549ms\n\trunning 944300-323: ✔ passed 8.960695ms\n\trunning 944300-324: ✔ passed 7.558008ms\n\trunning 944300-325: ✔ passed 5.977716ms\n\trunning 944300-326: ✔ passed 5.457394ms\n\trunning 944300-327: ✔ passed 5.896309ms\n\trunning 944300-328: ✔ passed 5.873305ms\n\trunning 944300-329: ✔ passed 5.828122ms\n➕ run 2354 total tests in 18.923445528s\n⏭ skipped 7 tests\n🎉 All tests successful!\n```\nHappy testing!\n\n## Output\n\nNow you can choose how the output of the test session is shown by passing the `-o` flag. The default output is `-o normal`,\nand it will show the emojis in all the supported terminals. If yours doesn't support emojis, or you want a plain format,\nyou can use `-o plain`:\n```shell\n./ftw run -d tests -o plain -i 932240\n\n** Running go-ftw!\n\tskipping 920360-1 - (enabled: false) in file.\n\tskipping 920370-1 - (enabled: false) in file.\n\tskipping 920380-1 - (enabled: false) in file.\n\tskipping 920390-1 - (enabled: false) in file.\n=\u003e executing tests in file 932240.yaml\n\trunning 932240-1: + passed in 39.928201ms (RTT 67.096865ms)\n\trunning 932240-2: + passed in 29.299056ms (RTT 65.650821ms)\n\trunning 932240-3: + passed in 30.426324ms (RTT 63.173202ms)\n\trunning 932240-4: + passed in 29.111381ms (RTT 66.593728ms)\n\trunning 932240-5: + passed in 30.627351ms (RTT 67.101436ms)\n\trunning 932240-6: + passed in 40.735442ms (RTT 79.628474ms)\n+ run 6 total tests in 200.127755ms\n\u003e\u003e skipped 3322 tests\n\\o/ All tests successful!\n```\n\nTo support automation for processing the test results, there is also a new JSON output available using `-o json`:\n```json\n{\n  \"run\": 8,\n  \"success\": [\n    \"911100-1\",\n    \"911100-2\",\n    \"911100-3\",\n    \"911100-4\",\n    \"911100-5\",\n    \"911100-6\",\n    \"911100-7\",\n    \"911100-8\"\n  ],\n  \"failed\": null,\n  \"skipped\": [\n    \"913100-1\",\n    \"913100-2\",\n    \"913100-3\",\n    \"...\",\n    \"980170-2\"\n  ],\n  \"ignored\": null,\n  \"forced-pass\": null,\n  \"forced-fail\": null,\n  \"runtime\": {\n    \"911100-1\": 20631077,\n    \"911100-2\": 14112617,\n    \"911100-3\": 14524897,\n    \"911100-4\": 14699391,\n    \"911100-5\": 16137499,\n    \"911100-6\": 16589660,\n    \"911100-7\": 16741235,\n    \"911100-8\": 20658905\n  },\n  \"TotalTime\": 134095281\n}\n```\n\nThen it is easy to use your `jq` skils to get the information you want.\n\nThe list of supported outputs is:\n- \"normal\"\n- \"quiet\"\n- \"github\"\n- \"json\"\n- \"plain\"\n\n#### Only show failures\n\nIf you are only interested to see when tests fail, there is a new flag `--show-failures-only` that does exactly that.\nThis is helpful when running in CI/CD systems like GHA to get shorter outputs.\n\n## Additional features\n\n- templates with the power of Go [text/template](https://golang.org/pkg/text/template/). Add your template to any `data:` sections and enjoy!\n- [Sprig functions](https://masterminds.github.io/sprig/) can be added to templates as well.\n- Override test results.\n- Cloud mode! This new mode will ignore log files and rely solely on the HTTP status codes of the requests for determining success and failure of tests.\n\nWith templates and functions, you can simplify bulk test writing, or even read values from the environment while executing. These features allow you to write tests like this:\n\n```yaml\ndata: 'foo=%3d{{ \"+\" | repeat 34 }}'\n```\n\nWill be expanded to:\n\n```yaml\ndata: 'foo=%3d++++++++++++++++++++++++++++++++++'\n```\n\nBut also, you can get values from the environment dynamically when the test is run:\n\n```yaml\ndata: 'username={{ env \"USERNAME\" }}\n```\n\nWill give you, as you expect, the username running the tests:\n\n```yaml\ndata: 'username=fzipi\n```\n\nOther interesting functions you can use are: `randBytes`, `htpasswd`, `encryptAES`, etc.\n\n## Overriding tests\n\nSometimes you have tests that work well for some platform combinations, e.g. Apache + ModSecurity 2, but fail for others, e.g. NGiNX + ModSecurity 3. Taking that into account, you can override test results using the `testoverride` config param. The test will be skipped, and the result forced as configured.\n\nTests can be altered using four lists:\n- `input` allows you to override global parameters in tests. The following ones can be overridden:\n  - `dest_addr`: overrides the destination address (accepts IP or hostname)\n  - `override_empty_host_header`: if true and `dest_addr` override is _also_ set, empty `Host` headers will be replaced with `dest_addr`\n  - `port`: overrides the port number\n  - `protocol`: overrides the protocol\n  - `uri`: overrides the uri\n  - `version`: overrides the HTTP version. E.g. \"HTTP/1.1\"\n  - `ordered_headers`: overrides headers, the format is a list of `name` / `value` pairs\n  - `method`: overrides the method used to perform the request\n  - `data`: overrides data sent in the request\n  - `autocomplete_headers`: overrides header autocompletion (currently sets `Connection: close` and `Content-Length` and `Content-Type` for requests with body data)\n  - `encodedrequest`: overrides base64 encoded request\n  - `virtual_host_mode`: set the `Host` header specified in tests for internal requests as well (overrides sending internal requests to `localhost`)\n- `ignore` is for tests you want to ignore. You should add a comment on why you ignore the test\n- `forcepass` is for tests you want to pass unconditionally. You should add a comment on why you force to pass the test\n- `forcefail` is for tests you want to fail unconditionally. You should add a comment on why you force to fail the test\n\nEach list is populated by regular expressions (see https://pkg.go.dev/regexp), which match against test IDs.\nThe following is an example using all the lists mentioned above:\n\n```yaml\n...\ntestoverride:\n  input:\n    dest_addr: \"192.168.1.100\"\n    port: 8080\n    protocol: \"http\"\n  ignore:\n    # text comes from our friends at https://github.com/digitalwave/ftwrunner\n    '941190-3$': 'known MSC bug - PR #2023 (Cookie without value)'\n    '941330-1$': 'know MSC bug - #2148 (double escape)'\n    '942480-2$': 'known MSC bug - PR #2023 (Cookie without value)'\n    '944100-11$': 'known MSC bug - PR #2045, ISSUE #2146'\n    '^920': 'All the tests about Protocol Attack (rules starting with \"920\") will be ignored'\n  forcefail:\n    '123456-01$': 'I want this specific test to fail, even if passing'\n  forcepass:\n    '123456-02$': 'This test will always pass'\n    '123457-.*': 'All the tests about rule 123457 will always pass'\n```\n\nYou can combine any of `ignore`, `forcefail` and `forcepass` to make it work for you.\n\n## ☁️ Cloud mode\n\nMost of the tests rely on having access to a logfile to check for success or failure. Sometimes that is not possible, for example, when testing cloud services or servers where you don't have access to logfiles and/or logfiles won't have the information you need to decide if the test was good or bad.\n\nWith cloud mode, we move the decision on test failure or success to the HTTP status code received after performing the test. The general idea is that you set up your WAF in blocking mode, so anything matching will return a block status (e.g. 403), and if not we expect a 2XX return code.\n\nYou will also want to override the IP configured in the tests, and use the one from your cloud provider instead.\n\nAn example config file for this is:\n```yaml\n---\nmode: 'cloud'\ntestoverride:\n  input:\n    dest_addr: \"\u003cyour cloud WAF IP\u003e\"\n    port: 80\n```\nSave this file as `cloud-test.yaml` and edit the WAF IP.\n\nThen run: `./ftw run --config cloud-test.yaml`\n\n## How log parsing works\n\nThe WAF's log file with the alert messages is parsed and compared to the expected output defined in the unit test under `log_contains` or `no_log_contains`.\nNote that the expected output may contain multiple checks (E.g. `log_contains` and `status`). If any of the checks fail, the test will fail.\n\nThe problem with log files is that `go-ftw` is very, very fast and the log files are not updated in real time. Frequently, the\nweb server / WAF is not syncing the file fast enough. That results in a situation where `go-ftw` won't find the log messages it has triggered.\n\nTo make log parsing consistent and guarantee that we will see output when we need it, `go-ftw` will send a request that is meant to write a marker into the log file before the individual test and another marker after the individual test.\n\nIf `go-ftw` does not see the finishing marker after executing the request, it will send the marker request again until the webserver is forced to write the log file to the disk and the marker can be found.\n\nThe [container images for Core Rule Set](https://github.com/coreruleset/modsecurity-crs-docker) can be configured to write these marker log lines by setting\nthe `CRS_ENABLE_TEST_MARKER` environment variable. If you are testing a different test setup, you will need to instrument it with a rule that generated the marker in the log file via a rule alert (unless you are using \"cloud mode\").\n\nThe rule for CRS looks like this:\n\n```\n# Write the value from the X-CRS-Test header as a marker to the log\nSecRule REQUEST_HEADERS:X-CRS-Test \"@rx ^.*$\" \\\n  \"id:999999,\\\n  pass,\\\n  phase:1,\\\n  log,\\\n  msg:'X-CRS-Test %{MATCHED_VAR}',\\\n  ctl:ruleRemoveById=1-999999\"\n```\n\nThe rule looks for an HTTP header named `X-CRS-Test` and writes its value to the log, the value being the UUID of a test stage. If the header does not exist, the rule will be skipped and no marker will be written. If the header is found, the rule will also disable all further matching against the request to ensure that reported matches only concern actual test requests.\n\nYou can configure the name of the HTTP header by setting the `logmarkerheadername` option in the configuration to a custom value (the value is case-insensitive).\n\n## Wait for backend service to be ready\n\nSometimes you need to wait for a backend service to be ready before running the tests. For example, you may need to wait for an additional container to be ready before running the tests.\nNow you can do that by passing the `--wait-for-host` flag. The value of this option is a URL that will be requested, and you can configure the expected result using the following additional flags:\n- `--wait-for-host`:                     Wait for host to be available before running tests.\n- `--wait-for-connection-timeout`        Http connection timeout, The timeout includes connection time, any redirects, and reading the response body. (default 3s)\n- `--wait-for-expect-body-json`          Expect response body JSON pattern.\n- `--wait-for-expect-body-regex`         Expect response body pattern.\n- `--wait-for-expect-body-xpath`         Expect response body XPath pattern.\n- `--wait-for-expect-header`             Expect response header pattern.\n- `--wait-for-expect-status-code`        Expect response code e.g. 200, 204, ... .\n- `--wait-for-insecure-skip-tls-verify`  Skips tls certificate checks for the HTTPS request.\n- `--wait-for-no-redirect`               Do not follow HTTP 3xx redirects.\n- `--wait-for-timeout`                   Sets the timeout for all wait operations, 0 is unlimited. (default 10s)\n\n## Quantitative testing\n\n### What is the idea behind quantitative tests?\n\nQuantitative testing mode provides a means to to quantify the amount of false positives to be expected in production for a given rule.\nWe use well-known corpora of texts to generate plausible, non-malicious payloads. Whenever such a payload is blocked by the WAF, the detection is considered to be a false positive.\n\nAnyone can create their own corpora of texts and use them to test their WAF. Each corpus essentially consists of a list of strings, which may be sent to the WAF, depending on the configuration of the run.\n\nThe result of a test run is a percentage of false positives. The lower the percentage, the better the WAF is at not blocking benign payloads for a given rule. However, since we use generic corpora in our tests, the strings in those corpora will not necessarily be representative of the domain of a specific site. This means that a rule with a low false positive rate can still produce many false positives in specific contexts, e.g., when a website contains programming language code.\n\n### What is a corpus? Why do I need one?\n\nA corpus is a collection of texts that is used to generate payloads.\nThe texts can contain anything, from news articles to books. The idea is to have a large collection of texts that can be used to generate payloads. Well-known corpora usually have a domain or context, e.g., news headlines, or English books of the 18th century.\n\nThe default corpus is the [Leipzig Corpora Collection](https://wortschatz.uni-leipzig.de/en/download/), which is a collection of texts from the web.\n\n### How to create a corpus?\n\nYou can create your own corpus by collecting texts from the web, or from books, articles, etc.\nYou could even use the contents of your own website as a corpus! What you will need to do is to implement the following interfaces:\n- `corpus.Corpus`\n- `corpus.File`\n- `corpus.Iterator`\n- `corpus.Payload`\n\nYou can see an example of how to implement the `corpus.Corpus` interface in the `corpus/leipzig` package.\n\n### How to run quantitative tests?\n\nTo run quantitative tests, you just need to pass the `quantitative` flag to `ftw`.\n\nThe corpus will be downloaded and cached locally for future use. You can also specify the size of the corpus,\nthe language, the source, and the year of the corpus. The bare minimum parameter that you must specify is the\ndirectory where the CRS rules are stored.\n\nHere is the help for the `quantitative` command:\n\n```bash\n❯ ./go-ftw quantitative -h\nRun all quantitative tests\n\nUsage:\n  ftw quantitative [flags]\n\nFlags:\n  -c, --corpus string              Corpus to use for the quantitative tests. (default \"leipzig\")\n  -L, --corpus-lang string         Corpus language to use for the quantitative tests. (default \"eng\")\n  -n, --corpus-line int            Number is the payload line from the corpus to exclusively send.\n      --corpus-local-path string   Path to store the local corpora. Defaults to .ftw folder under user's home directory.\n  -s, --corpus-size string         Corpus size to use for the quantitative tests. Most corpora will have sizes like \"100K\", \"1M\", etc. (default \"100K\")\n  -S, --corpus-source string       Corpus source to use for the quantitative tests. Most corpus will have a source like \"news\", \"web\", \"wikipedia\", etc. (default \"news\")\n  -y, --corpus-year string         Corpus year to use for the quantitative tests. Most corpus will have a year like \"2023\", \"2022\", etc. (default \"2023\")\n  -C, --crs-path string            Path to top folder of local CRS installation. (default \".\")\n  -f, --file string                Output file path for quantitative tests. Prints to standard output by default.\n  -h, --help                       help for quantitative\n  -l, --lines int                  Number of lines of input to process before stopping.\n      --max-concurrency int        maximum number of goroutines. Defaults to 10, or 1 if log level is debug/trace. (default 10)\n  -o, --output string              Output type for quantitative tests. (default \"normal\")\n  -P, --paranoia-level int         Paranoia level used to run the quantitative tests. (default 1)\n  -p, --payload string             Payload is a string you want to test using quantitative tests. Will not use the corpus.\n  -r, --rule int                   Rule ID of interest: only show false positives for specified rule ID.\n\nGlobal Flags:\n      --cloud              cloud mode: rely only on HTTP status codes for determining test success or failure (will not process any logs)\n      --config string      specify config file (default is $PWD/.ftw.yaml)\n      --debug              debug output\n      --overrides string   specify file with platform specific overrides\n      --trace              trace output: really, really verbose\n```\n\n\n### Example of running quantitative tests\n\nThis will run with the default leipzig corpus and size of 10K payloads.\n```bash\n❯ ./go-ftw quantitative -C ../coreruleset -s 10K\nRunning quantitative tests\nRun 10000 payloads in 18.482979709s\nTotal False positive ratio: 408/10000 = 0.0408\nFalse positives per rule:\n  Rule 920220: 198 false positives\n  Rule 920221: 198 false positives\n  Rule 932235: 4 false positives\n  Rule 932270: 2 false positives\n  Rule 932380: 2 false positives\n  Rule 933160: 1 false positives\n  Rule 942100: 1 false positives\n  Rule 942230: 1 false positives\n  Rule 942360: 1 false positives\n```\n\nThis will run with the default leipzig corpus and size of 10K payloads, but only for the rule 920350.\n```bash\n❯ ./go-ftw quantitative -C ../coreruleset -s 10K -r 932270\nRunning quantitative tests\nRun 10000 payloads in 15.218343083s\nTotal False positive ratio: 2/10000 = 0.0002\nFalse positives per rule:\n  Rule 932270: 2 false positives\n```\n\nIf you add `--debug` to the command, you will see the payloads that cause false positives.\n```bash\n❯ ./go-ftw quantitative -C ../coreruleset -s 10K --debug\nRunning quantitative tests\n12:32PM DBG Preparing download of corpus file from https://downloads.wortschatz-leipzig.de/corpora/eng_news_2023_10K.tar.gz\n12:32PM DBG filename eng_news_2023_10K-sentences.txt already exists\n12:32PM DBG Using paranoia level: 1\n\n12:32PM DBG False positive with string: And finally: \"I'd also say temp nurses make a lot.\n12:32PM DBG **\u003e rule 932290 =\u003e Matched Data: \"I'd found within ARGS:payload: And finally: \"I'd also say temp nurses make a lot.\n12:32PM DBG False positive with string: But it was an experience Seguin said she \"wouldn't trade for anything.\"\n12:32PM DBG **\u003e rule 932290 =\u003e Matched Data: \"wouldn't found within ARGS:payload: But it was an experience Seguin said she \"wouldn't trade for anything.\"\n12:32PM DBG False positive with string: Consolidated Edison () last issued its earnings results on Thursday, November 3rd.\n12:32PM DBG **\u003e rule 932235 =\u003e Matched Data: () last  found within ARGS:payload: Consolidated Edison () last issued its earnings results on Thursday, November 3rd.\n```\n\nThe default language for the corpus is English, but you can change it to German using the `-L` flag.\n```bash\n❯ ./go-ftw quantitative -C ../coreruleset -s 10K -L deu\nRunning quantitative tests\n4:18PM INF Downloading corpus file from https://downloads.wortschatz-leipzig.de/corpora/deu_news_2023_10K.tar.gz\nMoved /Users/fzipitria/.ftw/extracted/deu_news_2023_10K/deu_news_2023_10K-sentences.txt to /Users/fzipitria/.ftw/deu_news_2023_10K-sentences.txt\nRun 10000 payloads in 25.169846084s\nTotal False positive ratio: 44/10000 = 0.0044\nFalse positives per rule:\n  Rule 920220: 19 false positives\n  Rule 920221: 19 false positives\n  Rule 932125: 1 false positives\n  Rule 932290: 5 false positives\n```\n\nResults can be shown in JSON format also, to be processed by other tools.\n```bash\n❯ ./go-ftw quantitative -C ../coreruleset -s 10K -o json\n\n{\"count\":10000,\"falsePositives\":408,\"falsePositivesPerRule\":{\"920220\":198,\"920221\":198,\"932235\":4,\"932270\":2,\"932380\":2,\"933160\":1,\"942100\":1,\"942230\":1,\"942360\":1},\"totalTime\":15031086083}%\n```\n\n### Future work for quantitative tests\n\nThis feature will enable us to compare between two different versions of CRS (or any two rules) and see, for example,\nif any modification to the rule has caused more false positives.\n\nIntegrating it to the CI/CD pipeline will allow us to check every PR for false positives before merging.\n\n## Library usage\n\n`go-ftw` can be used as a library also. Just include it in your project:\n```sh\ngo get github.com/coreruleset/go-ftw\n```\n\nThen, for the example below, import at least these:\n```go\npackage main\n\nimport (\n    \"net/url\"\n    \"os\"\n    \"path/filepath\"\n    \"strconv\"\n\n    \"github.com/bmatcuk/doublestar/v4\"\n    \"github.com/coreruleset/go-ftw/config\"\n    \"github.com/coreruleset/go-ftw/output\"\n    \"github.com/coreruleset/go-ftw/runner\"\n    \"github.com/coreruleset/go-ftw/test\"\n    \"github.com/rs/zerolog\"\n)\n```\n\nAnd a sample code:\n```go\n     // sample from https://github.com/corazawaf/coraza/blob/v3/dev/testing/coreruleset/coreruleset_test.go#L215-L251\n    var tests []test.FTWTest\n    err = doublestar.GlobWalk(crsReader, \"tests/regression/tests/**/*.yaml\", func(path string, d os.DirEntry) error {\n        yaml, err := fs.ReadFile(crsReader, path)\n        if err != nil {\n            return err\n        }\n        t, err := test.GetTestFromYaml(yaml)\n        if err != nil {\n            return err\n        }\n        tests = append(tests, t)\n        return nil\n    })\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    u, _ := url.Parse(s.URL)\n    host := u.Hostname()\n    port, _ := strconv.Atoi(u.Port())\n    zerolog.SetGlobalLevel(zerolog.InfoLevel)\n    cfg, err := config.NewConfigFromFile(\".ftw.yml\")\n    if err != nil {\n        log.Fatal(err)\n    }\n    cfg.LogFile = errorPath\n    cfg.TestOverride.Input.DestAddr = \u0026host\n    cfg.TestOverride.Input.Port = \u0026port\n    runnerConfig := config.NewRunnerConfiguration(cfg)\n    runnerConfig.ShowTime = false\n\n    res, err := runner.Run(runnerConfig, tests, output.NewOutput(\"quiet\", os.Stdout))\n    if err != nil {\n        log.Fatal(err)\n    }\n\n\n    if len(res.Stats.Failed) \u003e 0 {\n\t\tlog.Errorf(\"failed tests: %v\", res.Stats.Failed)\n\t}\n```\n\n\n## License\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fcoreruleset%2Fgo-ftw.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fcoreruleset%2Fgo-ftw?ref=badge_large)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoreruleset%2Fgo-ftw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoreruleset%2Fgo-ftw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoreruleset%2Fgo-ftw/lists"}