{"id":20038522,"url":"https://github.com/yahoo/proxy-verifier","last_synced_at":"2026-03-27T20:58:30.187Z","repository":{"id":38998629,"uuid":"236102958","full_name":"yahoo/proxy-verifier","owner":"yahoo","description":"Proxy Verifier is an HTTP replay tool designed to verify the behavior of HTTP proxies. It builds a verifier-client binary and a verifier-server binary which each read a set of YAML or JSON files that specify the HTTP traffic for the two to exchange.","archived":false,"fork":false,"pushed_at":"2024-10-04T19:17:50.000Z","size":1337,"stargazers_count":44,"open_issues_count":17,"forks_count":26,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-09T15:05:00.037Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yahoo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"Code-of-conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-24T23:49:56.000Z","updated_at":"2025-03-22T10:52:22.000Z","dependencies_parsed_at":"2023-10-05T02:11:01.193Z","dependency_job_id":"c711d0e2-a867-4528-bc17-29a59a7be91f","html_url":"https://github.com/yahoo/proxy-verifier","commit_stats":{"total_commits":295,"total_committers":9,"mean_commits":32.77777777777778,"dds":0.5389830508474576,"last_synced_commit":"be9057da76b70041db1c535ef79222c456762e1c"},"previous_names":[],"tags_count":46,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2Fproxy-verifier","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2Fproxy-verifier/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2Fproxy-verifier/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2Fproxy-verifier/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yahoo","download_url":"https://codeload.github.com/yahoo/proxy-verifier/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248055284,"owners_count":21040157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T10:29:46.905Z","updated_at":"2026-03-27T20:58:30.107Z","avatar_url":"https://github.com/yahoo.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!---\nThe following toc was autogenerated from the shell using:\nhttps://github.com/ekalinin/github-markdown-toc\n\ncd ~/bin\nwget https://raw.githubusercontent.com/ekalinin/github-markdown-toc/master/gh-md-toc\nchmod 755 gh-md-toc\ncd \u003cpath/to/proxy-verifier/repo\u003e\ngh-md-toc README.md\n\nThen copy and paste the output here.\n--\u003e\n\nTable of Contents\n=================\n\n* [Proxy Verifier](#proxy-verifier)\n * [Traffic Replay Specification](#traffic-replay-specification)\n * [HTTP Specification](#http-specification)\n * [Server Response Lookup](#server-response-lookup)\n * [HTTP/2 Specification](#http2-specification)\n * [HEADERS and DATA frame](#headers-and-data-frame)\n * [RST_STREAM frame](#rst_stream-frame)\n * [GOAWAY frame](#goaway-frame)\n * [Await](#await)\n * [Protocol Specification](#protocol-specification)\n * [PROXY protocol support](#proxy-protocol-support)\n * [Session and Transaction Delay Specification](#session-and-transaction-delay-specification)\n * [Keep Connection Open](#keep-connection-open)\n * [Traffic Verification Specification](#traffic-verification-specification)\n * [Field Verification](#field-verification)\n * [URL Verification](#url-verification)\n * [Status Verification](#status-verification)\n * [Body Verification](#body-verification)\n * [Example Replay File](#example-replay-file)\n * [Installing](#installing)\n * [Prebuilt Binaries](#prebuilt-binaries)\n * [Building from Source](#building-from-source)\n * [Prerequisites](#prerequisites)\n * [Build](#build)\n * [Using Prebuilt Libraries](#using-prebuilt-libraries)\n * [ASan Instrumentation](#asan-instrumentation)\n * [Debug Build](#debug-build)\n * [Statically Link](#statically-link)\n * [Running the Tests](#running-the-tests)\n * [Unit Tests](#unit-tests)\n * [Gold Tests](#gold-tests)\n * [Usage](#usage)\n * [Required Arguments](#required-arguments)\n * [Optional Arguments](#optional-arguments)\n * [--format \u0026lt;format-specification\u0026gt;](#--format-format-specification)\n * [--keys \u0026lt;key1 key2 ... keyn\u0026gt;](#--keys-key1-key2--keyn)\n * [--verbose](#--verbose)\n * [--interface \u0026lt;interface\u0026gt;](#--interface-interface)\n * [--no-proxy](#--no-proxy)\n * [--strict](#--strict)\n * [--rate \u0026lt;requests/second\u0026gt;](#--rate-requestssecond)\n * [--repeat \u0026lt;number\u0026gt;](#--repeat-number)\n * [--run-continuously](#--run-continuously)\n * [--thread-limit \u0026lt;number\u0026gt;](#--thread-limit-number)\n * [--qlog-dir \u0026lt;directory\u0026gt;](#--qlog-dir-directory)\n * [--tls-secrets-log-file \u0026lt;secrets_log_file_name\u0026gt;](#--tls-secrets-log-file-secrets_log_file_name)\n * [--send-buffer-size \u0026lt;size\u0026gt;](#--send-buffer-size-size)\n * [--poll-timeout \u0026lt;timeout_ms\u0026gt;](#--poll-timeout-timeout_ms)\n * [Tools](#tools)\n * [Replay Gen \u003ca href=\"tools/replay_gen.py\"\u003ereplay_gen.py\u003c/a\u003e](tools/replay_gen.py)\n * [-n,--number \u0026lt;NUMBER\u0026gt;](#-n--number-number)\n * [-tl,--trans-lower \u0026lt;TRANS_LOWER\u0026gt;](#-tl--trans-lower-trans_lower)\n * [-tu,--trans-upper \u0026lt;TRANS_UPPER\u0026gt;](#-tu--trans-upper-trans_upper)\n * [-sl,--sess-lower \u0026lt;SESS_LOWER\u0026gt;](#-sl--sess-lower-sess_lower)\n * [-su,--sess-upper \u0026lt;SESS_UPPER\u0026gt;](#-su--sess-upper-sess_upper)\n * [-tp,--trans-protocols \u0026lt;TRANS_PROTOCOLS\u0026gt;](#-tp--trans-protocols-trans_protocols)\n * [-u,--url-file \u0026lt;URL_FILE\u0026gt;](#-u--url-file-url_file)\n * [-o,--output \u0026lt;OUTPUT\u0026gt;](#-o--output-output)\n * [-p,--prefix \u0026lt;PREFIX\u0026gt;](#-p--prefix-prefix)\n * [-j,--out-json](#-j--out-json)\n * [Remap Config to URL List \u003ca href=\"tools/remap_config_to_url_list.py\"\u003eremap_config_to_url_list.py\u003c/a\u003e](tools/remap_config_to_url_list.py)\n * [-o,--output \u0026lt;OUTPUT_FILE\u0026gt;](#-o--output-output_file)\n * [--no-ip](#--no-ip)\n * [Contribute](#contribute)\n * [License](#license)\n\n\u003c!-- Created by https://github.com/ekalinin/github-markdown-toc --\u003e\n\n# Proxy Verifier\n\nProxy Verifier is an HTTP replay tool designed to verify the behavior of HTTP\nproxies. It builds a verifier-client binary and a verifier-server binary which\neach read a set of YAML files (or JSON files, since JSON is YAML) that specify\nthe HTTP traffic for the two to exchange.\n\nProxy Verifier supports the HTTP replay of the following protocols:\n\n* HTTP/1.x and HTTP/2\n* HTTP and HTTP over TLS\n* IPv4 and IPv6\n\nBroadly speaking, Proxy Verifier is designed to address two proxy testing\nneeds:\n\n* **Traffic correctness testing**: In this context Proxy Verifier can be used\n in manual or automated end to end tests to verify correct behavior of the\n details of generally a small number of transactions. [Transaction\n Box](https://github.com/SolidWallOfCode/txn_box) is an example of a tool that\n relies entirely on Proxy Verifier for its automated end to end tests (see its\n [autest](https://github.com/SolidWallOfCode/txn_box/tree/master/test/autest)\n directory).\n* **Production simulation testing**: In this context, Proxy Verifier is used in\n an isolated lab environment configured as much like a production environment\n as possible. The Verifier client and server are provided replay files as\n input that are either auto generated or collected via a tool like [Traffic\n Dump](https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/traffic_dump.en.html)\n from an actual production environment. Proxy verifier then replays this\n production-like traffic against the proxy under test. Proxy Verifier can\n replay such traffic at rates over 10k requests per second. Here are some\n examples where this use of Proxy Verifier can be helpful:\n\n * Safely testing experimental versions of a patch.\n * Running diagnostic versions of the proxy that may not be performant enough\n for the production environment. Debug, Valgrind, and ASan builds are\n examples of build configurations whose resultant proxy performance may be\n impossible to run in production but can be run safely in the production\n simulation via Proxy Verifier.\n * Performance comparison across versions of the proxy. Since Proxy Verifier\n replays the traffic for the same set of replay files consistently across\n runs, different versions of the proxy can be compared with regard to their\n performance against the same replay dataset. This affords the ability to\n compare detect performance improvement or degradation across versions of the\n proxy.\n\nIt should be noted that when using Proxy Verifier in a production simulation\nenvironment, the proxy will be receiving traffic from the client that looks\nlike production traffic. Thus, by design, the HTTP request targets and Host\nheader fields will reference actual production servers. It will be important to\nconfigure the proxy under test to direct these requests not to the actual\nproduction servers but to the Proxy Verifier server or servers. The way this\nis achieved will depend upon the proxy. One way to accomplish this is to\nconfigure the production environment with a test DNS server such as\n[MicroDNS](https://bitbucket.org/autestsuite/microdns/src/master/) configured\nto resolve all host names to the Proxy Verifier server or servers in the\nproduction simulation environment.\n\n## Traffic Replay Specification\n\n### HTTP Specification\n\nProxy Verifier traffic behavior is specified via YAML files. The behavior for\neach connection is specified under the top-most `sessions` node, the value of\nwhich is a sequence where each item in the sequence describes the\ncharacteristics of each connection. For each session the protocol stack can be\nspecified via the `protocol` node. This node describes the protocol\ncharacteristics of the connection such as what version of HTTP to use, whether\nto use TLS and what the characteristics of the TLS handshake should be. In the\nabsence of a `protocol` node the default protocol is HTTP/1 over TCP. See\n[Protocol Specification](#protocol-specification) below for details about the\n`protocol` node. Each session is run in parallel by the verifier-client\n(although see [--thread-limit \u0026lt;number\u0026gt;](#--thread-limit-number) below for\na way to serialize them).\n\nIn addition to protocol specification, each session in the `sessions`\nsequence contains a `transactions` node. This itself is a sequence of\ntransactions that should be replayed for the associated session. Within each\ntransaction, Proxy Verifier's traffic replay behavior is specified in the\n`client-request` and `server-response` nodes. `client-request` nodes are used\nby the Proxy Verifier client and tell it what kind of HTTP requests it should\ngenerate. `server-response` nodes are used by the Proxy Verifier server to\nindicate what HTTP responses it should generate. Proxy traffic verification\nbehavior is described in the `proxy-request` and `proxy-response` nodes which\nwill be covered in [Traffic Verification\nSpecification](#traffic-verification-specification). For HTTP/1, these\ntransactions are run in sequence for each session; for HTTP/2, the transactions\nare run in parallel.\n\nFor HTTP/1 requests, `client-request` has a map as a value which contains each of\nthe following key/value YAML nodes:\n\n1. `method`: This takes the HTTP method as the value, such as `GET` or `POST`.\n1. `url`: This takes the request target as the value, such as `/a/path.asp` or\n `https://www.example.com/a/path.asp`.\n1. `version`: This takes the HTTP version, such as `1.1` or `2`.\n1. `headers`: This takes a `fields` node which has, as a value, a sequence of\n HTTP fields. Each field is itself a sequence of two values: the name of the\n field and the value for the field.\n1. `content`: This specifies the body to send. It takes a map as a value. The\n user can specify a `size` integer value in which an automated body of that\n size will be generated. Otherwise a `data` string value can be provided in\n which the specified body will be sent and an `encoding` taking `plain` or\n `uri` to specify that the string is raw or URI encoded.\n\nHere's an example of a `client-request` node describing an HTTP/1.1 POST\nrequest with a request target of `/pictures/flower.jpeg`\nand containing four header fields with a body size of 399 bytes:\n\n```YAML\n client-request:\n method: POST\n url: /pictures/flower.jpeg\n version: '1.1'\n headers:\n fields:\n - [ Host, www.example.com ]\n - [ Content-Type, image/jpeg ]\n - [ Content-Length, '399' ]\n - [ uuid, 1234 ]\n content:\n size: 399\n```\n\nFor convenience, if Proxy Verifier detects a Content-Length header, then the\n`content` node stating the size of the body is not required. Thus this can be\nsimplified to:\n\n```YAML\n client-request:\n method: POST\n url: /pictures/flower.jpeg\n version: '1.1'\n headers:\n fields:\n - [ Host, www.example.com ]\n - [ Content-Type, image/jpeg ]\n - [ Content-Length, '399' ]\n - [ uuid, 1234 ]\n```\n\n`server-response` nodes indicate how the Proxy Verifier server should respond\nto HTTP requests. In place of `method`, `url`, and `version`, HTTP/1 responses\ntake the following nodes:\n\n1. `status`: This takes an integer corresponding to the HTTP response status,\n such as `200` or `404`.\n1. `reason`: This takes a string that describes the status, such as `\"OK\"` or\n `\"Not Found\"`.\n\nHere's an example of an HTTP/1 `server-response` with a status of 200, four\nfields, and a body of size 3,432 bytes:\n\n```YAML\n server-response:\n status: 200\n reason: OK\n headers:\n fields:\n - [ Date, \"Sat, 16 Mar 2019 03:11:36 GMT\" ]\n - [ Content-Type, image/jpeg ]\n - [ Transfer-Encoding, chunked ]\n - [ Connection, keep-alive ]\n content:\n size: 3432\n```\n\nObserve in this case that the response contains the `Transfer-Encoding:\nchunked` header field, indicating that the body should be chunk encoded. Proxy\nVerifier supports chunk encoding for both requests and responses and will use\nthis when it detects the `Transfer-Encoding: chunked` header field. In this\ncase, the 3,432 bytes will be the size of the chunked body payload without the\nbytes for the chunk protocol (chunk headers, etc.).\n\nFinally, here is an example of a response with specific body content sent (YAML\nin this case) as opposed to the generated content specified by the\n`content:size` nodes above:\n\n```YAML\n server-response:\n status: 200\n reason: OK\n headers:\n fields:\n - [ Date, \"Sat, 16 Mar 2019 03:11:36 GMT\" ]\n - [ Content-Type, text/yaml ]\n - [ Transfer-Encoding, chunked ]\n - [ Connection, keep-alive ]\n content:\n encoding: plain\n data: |\n ### Heading\n\n * Bullet\n * Points\n```\n\n#### Server Response Lookup\n\nThe `client-request` and `server-response` nodes are all that is required to\ndirect Proxy Verifier's replay of a transaction. The next section will describe\nhow to specify proxy transaction verification behavior via the `proxy-request` and\n`proxy-response` nodes. Before proceeding to that, however, it is valuable to\nunderstand how the Verifier server decides which response to send for any given\nrequest it receives. Consider again the `client-request` node in the preceding\nexample:\n\n```YAML\n client-request:\n method: POST\n url: /pictures/flower.jpeg\n version: '1.1'\n headers:\n fields:\n - [ Host, www.example.com ]\n - [ Content-Type, image/jpeg ]\n - [ Content-Length, '399' ]\n - [ uuid, 1234 ]\n content:\n size: 399\n```\n\nNote the presence of the `uuid` field. A field such as this is sent by the\nclient and used by the server. When a server receives a request, keep in mind\nthat all it has to direct its behavior is what it has parsed from the replay\nfile or files and what it sees in the request. There may be hundreds of\nthousands of parsed `server-response` nodes, each describing a unique response\nwith which the server may reply for any given incoming request. The server does\nnot talk directly to the client, so it cannot communicate with it and say, \"Hey\nVerifier client, I just read such and such request off the wire. Which response\nwould you like me to send for this?\" When it receives an HTTP request,\ntherefore, it only has the contents of that request from which to choose its\nresponse. To facilitate the server's behavior, a unique *key* is associated\nwith every request. During the YAML parsing phase, the Verifier server\nassociates each parsed `server-response` with a key it derives from the\nassociated `client-request` for the transaction. During the traffic replay\nphase, when it reads a request off the wire, it derives a key from the HTTP\nrequest using the same process for generating keys from `client-request` nodes\nused in the parsing phase. It then looks up the proper YAML-specified response\nusing that key. By default, Proxy Verifier uses the `uuid` HTTP header field\nvalues as the key, as utilized in the examples in this document, but this is\nconfigurable via the `--format` command line argument. For details see the\nsection describing this argument below.\n\nBoth the client and the server will fail the YAML parsing phase and exit with a\nnon-zero return if either parses a transaction for which they cannot derive a\nkey. If during the traffic processing phase the Verifier server somehow\nreceives a request for which it cannot derive a key, it will return a *404 Not\nFound* response and close the connection upon which it received the request.\n\n\n### HTTP/2 Specification\n\nFor HTTP/2, the protocol describes the initial request line values with pseudo\nheader fields. For that protocol, therefore, the user need not specify the\n`method`, `url`, and `version` nodes and would instead specify these values\nmore naturally as pseudo headers in the `fields` sequence. Here is an example of\nan HTTP/2 `client-request` analogous to the HTTP/1 request above (note that the\n`Content-Length` header field is optional in HTTP/2 and as such is omitted here):\n\n```YAML\n client-request:\n headers:\n fields:\n - [ :method, POST ]\n - [ :scheme, https ]\n - [ :authority, www.example.com ]\n - [ :path, /pictures/flower.jpeg ]\n - [ Content-Type, image/jpeg ]\n - [ uuid, 1234 ]\n content:\n size: 399\n```\n\nThe status code is described in the `:status` pseudo header field.\nAlso, HTTP/2 does not allow for chunk encoding nor does it use the `Connection`\nheader field, using instead its framing mechanism to describe the body and\nsession lifetime. An analogous HTTP/2 response to the HTTP/1 request above,\ntherefore, would look like the following:\n\n```YAML\n server-response:\n headers:\n fields:\n - [ :status, 200 ]\n - [ Date, \"Sat, 16 Mar 2019 03:11:36 GMT\" ]\n - [ Content-Type, image/jpeg ]\n content:\n size: 3432\n```\n\nTrailer response headers are useful for transmitting additional fields after the\nmessage body, providing dynamically generated metadata such as integrity checks\nor post-processing status. Proxy Verifier supports sending and receiving trailer\nheaders, as demonstrated below:\n\n```YAML\n server-response:\n status: 200\n headers:\n fields:\n # Some fields...\n content:\n # Some data...\n trailers:\n encoding: esc_json\n fields:\n - [ x-test-trailer-1, one ]\n - [ x-test-trailer-2, two ]\n```\n\nIt is also possible to specify everything as a sequence of frames. The available\noptions for the frame sequence are:\n* `DATA`\n* `HEADERS`\n* `RST_STREAM`\n* `GOAWAY`\n\nHere's an example replay file:\n\n```YAML\n client-request:\n frames:\n - HEADERS:\n headers:\n fields:\n - [ :method, POST ]\n - [ :scheme, https ]\n - [ :authority, www.example.com ]\n - [ :path, /pictures/flower.jpeg ]\n - [ Content-Type, image/jpeg ]\n - [ uuid, 1234 ]\n - DATA:\n content:\n size: 399\n - RST_STREAM:\n error-code: STREAM_CLOSED\n\n server-response:\n frames:\n - HEADERS:\n headers:\n fields:\n - [ :status, 200 ]\n - [ Date, \"Sat, 16 Mar 2019 03:11:36 GMT\" ]\n - [ Content-Type, image/jpeg ]\n - DATA:\n content:\n size: 3432\n```\n\n#### HEADERS and DATA frame\n\nThe `HEADERS` and `DATA` frame nodes are designed to be specified in a way that is consistent\nwith their HTTP/1 counterparts. From a parsing perspective, this means they simply wrap the\n`headers` and `content` nodes that are used for HTTP/1 specification as described above. For\nan example, see the `RST_STREAM` frame section below.\n\nNote that multiple `DATA` frames can be specified for requests and responses. The replay follows\nthe same order the `DATA` frames are listed. See the example below:\n\n```YAML\n client-request:\n frames:\n - HEADERS:\n headers:\n fields:\n - [:method, POST]\n - [:scheme, https]\n - [:authority, example.data.com]\n - [:path, /a/path]\n - [Content-Type, text/html]\n - [uuid, 1]\n - DATA:\n content:\n encoding: plain\n data: client_data_1\n - DATA:\n content:\n encoding: plain\n data: client_data_2\n```\n\n#### RST_STREAM frame\n\nIn some cases, there might be a need to test the behavior of the proxy when the\nclient or server terminates the stream, either because of an unexpected error or intentionally\ncancelling the stream. In HTTP/2, peers terminate a transaction by sending a `RST_STREAM` frame\nwhich includes an error code. In the replay file, this is specified via a `RST_STREAM` frame\nnode which is ordered within the stream to indicate when it should be sent and includes an\n`error-code` node to specify which error code should be used.\n\nThe following sample replay file snippet demonstrate how to specify such a test scenario:\n\n```YAML\nsessions:\n- protocol:\n - name: http\n version: 2\n - name: tls\n sni: test_sni\n - name: tcp\n - name: ip\n version: 4\n transactions:\n - client-request:\n frames:\n - HEADERS:\n headers:\n fields:\n - [:method, POST]\n - [:scheme, https]\n - [:authority, example.data.com]\n - [:path, /a/path]\n - [Content-Type, text/html]\n - [Content-Length, '11']\n - [uuid, 1]\n - RST_STREAM:\n error-code: INTERNAL_ERROR\n - DATA:\n content:\n encoding: plain\n data: client_test\n size: 11\n```\n\nNote that this example specifies the following:\n* A sequence of frames to be sent under the `client-request` node. The order\n of the frames listed is the order of the frames that Proxy Verifier will send during\n the traffic replay of the stream.\n* Thus, in this case, the client terminates the stream after sending the `HEADERS` frame since\n the `RST_STREAM` is specified after the `HEADERS` frame in the sequence.\n* The client terminates with the error code `INTERNAL_ERROR`, specified by `error-code` under\n the `RST_STREAM` frame.\n\nBe aware that when Proxy Verifier generates HTTP/2 frames, it determines from the frames elements where to\ninsert the `END_STREAM` flag. If a request or response just has a `HEADERS` frame, then the `END_STREAM`\nflag will be added to the end of the `HEADERS` frame. If there are both `HEADERS` and DATA frames, then the\n`END_STREAM` will be placed after the `DATA` frame. Proxy Verifier does not, however, use `RST_STREAM`\nframes to influence the `END_STREAM` flag. Thus in the above example, having the `RST_STREAM` between the\n`HEADERS` and `DATA` frames means that the `HEADERS` will not have an `END_STREAM` flag before the\n`RST_STREAM` is sent. For this reason, you must include a `DATA` frame after a `RST_STREAM` frame, even\nthough the `DATA` frame will not be sent, in order to keep the stream open at the time the `RST_STREAM` is\nsent.\n\nThe server side stream termination can be set in the same way, but under the `server-response` node.\n\nThe available options for `error-code` are:\n* NO_ERROR\n* PROTOCOL_ERROR\n* INTERNAL_ERROR\n* FLOW_CONTROL_ERROR\n* SETTINGS_TIMEOUT\n* STREAM_CLOSED\n* FRAME_SIZE_ERROR\n* REFUSED_STREAM\n* CANCEL\n* COMPRESSION_ERROR\n* CONNECT_ERROR\n* ENHANCE_YOUR_CALM\n* INADEQUATE_SECURITY\n* HTTP_1_1_REQUIRED\n\nFiner grained frame ordering can also be specified via a `delay` node which specifies the time\ndelay before sending the `RST_STREAM` frame. For example, to delay sending a `RST_STREAM` frame\nfor 1 second, specify a `delay: 1s` directive like so:\n\n```YAML\n - RST_STREAM:\n error-code: INTERNAL_ERROR\n delay: 1s\n```\n\nA detailed description of the `delay` node can be found [here](#session-and-transaction-delay-specification).\n\n#### GOAWAY frame\n\nThe `GOAWAY` frame acts similar to the `RST_STREAM` frame shown above. However, rather than terminating\na stream, `GOAWAY` frame terminates the connection. It supports `error-code` and `delay` as described\nfor `RST_STREAM` frame above.\n\nHTTP/2 sessions also have a `close-on-goaway` directive. This boolean\nconfiguration informs how the Verifier client should behave when there are more\nstreams configured in the replay YAML file after a `GOAWAY` frame is received\n(see [RFC 7540 section\n6.8](https://datatracker.ietf.org/doc/html/rfc7540#section-6.8) for details\nabout the `GOAWAY` frame). When set to `true`, on receipt of a `GOAWAY` frame,\nthe client terminates the connection after processing the streams that are\ncurrently being processed. When set to `false`, the client does not terminate\nthe connection and continues with subsequent streams specified in the replay\nfile. The default value of this option is `true` since that is in keeping with\nthe RFC. Here's an example session configured with `close-on-goaway` set to\n`false`:\n\n```YAML\nsessions:\n- protocol:\n - name: http\n version: 2\n - name: tls\n sni: test_sni\n - name: tcp\n - name: ip\n version: 4\n close-on-goaway: false\n transactions:\n```\n\n#### Await\n\nBy protocol specification, HTTP/1 transactions are serialized. That is, apart\nfrom the rarely used pipelined request protocol feature (which is not supported\nby Proxy Verifier), each HTTP/1 transaction in a connection is not exchanged\nuntil the request and response of the previous one is completed. By contrast,\nthe HTTP/2 protocol specifies that streams (i.e., HTTP/2 transactions) are\nmultiplexed within their sessions (i.e., HTTP/2 connections). For the Verifier\nclient, this means that, by default, requests for streams within a session are\neach sent immediately in the order that they are specified in the replay file\nwithout waiting for their respective responses. That is, if a replay file\nspecifies three streams within a session, then the client will send the request\nfor each stream as quickly as it can serialize it on the socket for the\nsession before waiting for any responses.\n\nThe timing for sending the stream requests can be modified from this default\nbehavior in several ways. As with replaying HTTP/1 traffic, if the streams\ncontain timing specification nodes, then the client can be configured to replay\nthe streams at a rate scaled to that timing information. See the documentation\nfor the [--rate](#--rate-requestssecond) argument for a description of how\nthis works. Also, the user can include\n[delay](#session-and-transaction-delay-specification) nodes to space out the\nreplay of each stream.\n\nIn addition to these configurations supported in HTTP/1 traffic replay, for\nHTTP/2 and HTTP/3 the Verifier client supports the `await` node to control when\nit starts streams. This node takes a transaction key or a sequence of\ntransaction keys, the responses of which are dependencies for sending the request\nfor the associated stream. That is, a stream with an `await` node will not be\nsent until all the responses are received for the listed keys.\n\nAs an example, consider the specification of the following session with two\nstreams:\n\n```YAML\n\n# Specify an HTTP/2 session\n- protocol:\n - name: http\n version: 2\n - name: tls\n sni: test_sni\n - name: tcp\n - name: ip\n\n transactions:\n\n # Stream 1\n - client-request:\n headers:\n fields:\n - [ :method, GET ]\n - [ :scheme, https ]\n - [ :authority, www.example.com ]\n - [ :path, /pictures/flower.jpeg ]\n - [ Content-Type, image/jpeg ]\n - [ uuid, first-stream ]\n\n server-response:\n headers:\n fields:\n - [ :status, 200 ]\n - [ Content-Type, image/jpeg ]\n - [ X-Response, first-response ]\n content:\n size: 3432\n\n # Stream 2\n - client-request:\n\n # This await will cause the Verifier client to hold off on sending this\n # request until the response to first-stream is received.\n await: first-stream\n\n headers:\n fields:\n - [ :method, GET ]\n - [ :scheme, https ]\n - [ :authority, www.example.com ]\n - [ :path, /pictures/flower.jpeg ]\n - [ Content-Type, image/jpeg ]\n - [ uuid, second-stream ]\n\n server-response:\n headers:\n fields:\n - [ :status, 200 ]\n - [ Content-Type, image/jpeg ]\n - [ X-Response, second-response ]\n content:\n size: 3432\n```\n\nNotice that the second transaction with key `second-stream` contains an\n`await` node. Without this node, the Verifier client would send both requests\nback to back. That is, the `second-stream` request would be sent immediately\nafter the `first-stream` request was sent. With the `await` node specifying a\ndependency upon the `first-stream` transaction, however, the Verifier client\nwill instead delay sending the `second-stream` request until the response to\n`first-stream` was received from the proxy.\n\nThere are a few things related to `await` nodes to be cognizant of:\n\n* `await` nodes can be used in conjunction with `delay` nodes for\n `client-request` specifications. When both are used, the Verifier client will\n first hold off on sending the request until all dependant responses specified\n by the keys in `await` are processed. Once the dependant responses are\n received, then the `delay` node is processed and the request is further\n delayed for the amount of time specified by the value of the `delay` node.\n Then the request is sent after the delay.\n* As stated before, the Verifier client is designed to processes streams and\n their directives in the order that they are specified in the `transactions`\n sequence for their stream. This means that any `await` or `delay` nodes for\n one transaction causes a corresponding delay for later nodes before they are\n processed. That is to say, if a session is specified with five streams, and\n the third stream has an `await` specified for the first two streams, then the\n fourth and fifth streams will also be delayed behind the third stream while\n it awaits the responses for the first and second streams, even if streams\n four and five contain no `await` nor `delay` nodes themselves.\n* Transaction keys specified via `await` can only be for earlier transactions\n in the same session that the `client-request` node is in. That is, the\n Verifier client does not support a stream in one session to `await` the\n response to a stream in a different session.\n\n### Protocol Specification\n\nThe above discussed the replay YAML nodes that describe how Proxy Verifier will\ncraft HTTP layer traffic. This section discusses how the user specifies the\nlower layer protocols used to transport this HTTP traffic.\n\nAs stated above, each HTTP session is described as an item under the `sessions`\nnode sequence. Each session takes a map. HTTP transactions are described under\nthe `transactions` key described above. In addition to `transactions`, a\nsession also takes an optional `protocol` node. This node takes an ordered\nsequence of maps, where each item in the sequence describes the characteristics\nof a protocol layer. The sequence is expected to be ordered from higher layer\nprotocols (such as HTTP and TLS) to lower layer protocols (such as IP).\n\nHere is an example protocol node along with `sessions` and `transactions`\nnodes provided to give some context:\n\n```YAML\nsessions:\n\n- protocol:\n - name: http\n version: 2\n - name: tls\n sni: test_sni\n - name: tcp\n - name: ip\n\n transactions:\n # ...\n```\n\nNote again how the `protocol` node is under the `sessions` node which takes a\nsequence of sessions. This sample shows the start of a single session that, in\nthis case, provides a protocol description via a `protocol` key. This same\nsession also has a truncated set of transactions that will be specified under\nthe `transactions` key. Looking further at the `protocol` node, observe that\nthis session has four layers described for it: http, tls, tcp, and ip. The\n`http` node specifies that the session should use the HTTP/2 protocol. The\n`tls` node specifies that the client should use an SNI of \"test\\_sni\" in the\nTLS client hello handshake. Further, this should be transported over TCP on IP.\n\nThe following nodes are supported for `protocol`:\n\n| Name | Node | Supported Values | Description\n| ----- |-------- | ---------------- | -----------\n| http | | |\n| | version | {1, 2} | Whether to use HTTP/1 or HTTP/2.\n| tls | | |\n| | sni | string | The SNI to send in the TLS handshake.\n| | request-certificate | boolean | Whether the client or server should request a certificate from the proxy.\n| | proxy-provided-certificate | boolean | This directs the same behavior as the request-certificate directive. This alias is helpful when the node describes what happened in the past, such as in the context of a replay file specified by [Traffic Dump](https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/traffic_dump.en.html).\n| | verify-mode | {0-15} | The value to pass directly to OpenSSL's `SSL_set_verify` to control peer verification in the TLS handshake. This allows fine grained control over TLS verification behavior. `0` corresponds with SSL_VERIFY_NONE, `1` corresponds with SSL_VERIFY_PEER, `2` corresponds with SSL_VERIFY_FAIL_IF_NO_PEER_CERT, `4` corresponds with SSL_VERIFY_CLIENT_ONCE, and `8` corresponds with SSL_VERIFY_POST_HANDSHAKE. Any bitwise OR'd value of these values can be provided. For details about their behavior, see OpenSSL's [SSL_verify_cb](https://www.openssl.org/docs/man1.1.1/man3/SSL_verify_cb.html) documentation.\n| | alpn-protocols | sequence of strings | This specifies the server's protocol list used in ALPN selection. See OpenSSL's [SSL_select_next_proto](https://www.openssl.org/docs/man1.0.2/man3/SSL_select_next_proto.html) documentation for details.\n| proxy-protocol | | |\n| | version | {1, 2} | Whether to use PROXY header v1 or v2\n| | src-addr | string | The source address and port in the PROXY header. Specified in the format of `111.111.111.111:11111`.\n| | dst-addr | string | The destination address and port in the PROXY header. Specified in the same format of `src-addr`.\n| tcp | | |\n| ip | | |\n\n\nThe following protocol specification features are not currently implemented:\n\n* HTTP/2 is only supported over TLS. Proxy Verifier uses ALPN in the TLS\n handshake to negotiate HTTP/2 with the proxy. HTTP/2 upgrade from HTTP/1\n without TLS is not supported.\n* The user cannot supply a TLS version to negotiate for the handshake.\n Currently, if the TLS node is present, Proxy Verifier will use the highest\n TLS version it can negotiate with the peer. This is OpenSSL's default\n behavior. An enhancement request to support A TLS version specification\n feature request is recorded in issue\n [101](https://github.com/yahoo/proxy-verifier/issues/101).\n* Similarly, the user cannot specify whether to use IPv4 or IPv6 via an\n `ip:version` node. Proxy Verifier can test IPv6, but it does so via the user\n passing IPv6 addresses on the commandline. An IP version feature request is\n recorded in issue [100](https://github.com/yahoo/proxy-verifier/issues/100).\n* Only TCP is supported. There have been recent discussions about adding\n HTTP/3 support, which is over UDP, but work for that has not yet started.\n* The PROXY protocol support is limited to the `PROXY` command via TCP over IPv4 or IPv6. Therefore,\n * No support for `AF_UNIX` socket address as either source and destination address.\n * No support for UDP transport type.\n * No support for `LOCAL` command(in v2 header).\n\nIf there is no `protocol` node specified, then Proxy Verifier will default to\nestablishing an HTTP/1 connection over TCP (no TLS).\n\n#### PROXY protocol support\nGenerally speaking, a server sitting downstream from a proxy does not have\nvisibility into the client's network socket information that lies behind the\nproxy. PROXY Protocol is a mechanism to provide visibility for this. PROXY\nprotocol is a network protocol that communicates a client's source and\ndestination IP and port information via a set of bytes at the start of a TCP\nconnection from a proxy. Here is a link to the protocol description:\n\nhttps://github.com/haproxy/haproxy/blob/master/doc/proxy-protocol.txt\n\nProxy Verifier supports sending and receiving the PROXY protocol, which can be\nhelpful to verify the PROXY protocol behavior of the proxy under test.\nThe feature can be enabled by specifying the `proxy-protocol` protocol node as\noutlined above. If enabled, the Verifier client would send out the PROXY\nprotocol header at the beginning of the connection. Upon receiving a PROXY\nprotocol header, the Verifier server would display it in the human-readable v1\nformat. Note that the specification of source and destination addresses are\nsupported but not required; if not specified, the Proxy Verifier client would\nsend out PROXY message with the source and destination addresses matching the\nunderlying socket, similar to how `curl --haproxy-protocol` behaves.\n\n### Session and Transaction Delay Specification\n\nA user can also stipulate per session and/or per transaction delays to be\ninserted by the Verifier client and server during the replay of traffic. This\nis done via the `delay` node which takes a unit-specified duration for the\nassociated delay. During traffic replay, the delay is inserted before the\nassociated session is established or before the client request or server\nresponse is sent. Proxy Verifier recognizes the following time duration units\nfor the `delay` node:\n\nUnit Suffix | Meaning\n----------- | -------\ns | seconds\nms | milliseconds\nus | microseconds\n\nHere is a sample replay file snippet that demonstrates the specification of a\nset of delays:\n\n```YAML\nsessions:\n- delay: 2s\n\n transactions:\n\n client-request:\n delay: 15ms\n\n method: POST\n url: /a/path.jpeg\n version: '1.1'\n headers:\n fields:\n - [ Content-Length, '399' ]\n - [ Content-Type, image/jpeg ]\n - [ Host, example.com ]\n - [ uuid, 1 ]\n\n server-response:\n delay: 17000 us\n\n status: 200\n reason: OK\n headers:\n fields:\n - [ Date, \"Sat, 16 Mar 2019 03:11:36 GMT\" ]\n - [ Content-Type, image/jpeg ]\n - [ Transfer-Encoding, chunked ]\n - [ Connection, keep-alive ]\n content:\n size: 3432\n```\n\nNote that this example specifies the following delays:\n\n* The client delays 2 seconds before establishing the session.\n* The client also delays 15 milliseconds before sending the client\n request.\n* The server delays 17 milliseconds (17,000 microseconds) before sending the\n corresponding response after receiving the request.\n\nBe aware of the following characteristics of the `delay` node:\n\n* The Verifier client interprets and implements `delay` for sessions in the\n `sessions` node and for transactions in the `client-request` node. The\n Verifier server interprets delay only for transactions in the\n `server-response` node and ignores `sessions` delays. Since the server is\n passive in receiving connections, it's not obvious what a server-side session\n delay would mean in this context.\n* Notice that Proxy Verifier supports microsecond level delay granularity, and\ndoes indeed faithfully insert delays at the appropriate times during replay\nwith that precision of time. Be aware, however, that for the vast majority of\nnetworks anything more precise than a millisecond will not generally be useful.\n\nSee also [--rate \u0026lt;requests/second\u0026gt;](#--rate-requestssecond) below for\nrate specification of transactions.\n\n### Keep Connection Open\n\nIn certain special situations, a user might need to keep the connection open\nafter the final transaction in a session is done. To specify how long the connection\nneeds to be kept open, the user can specify the duration as follows (value format is\nthe same as [Session and Transaction Delay Specification](#session-and-transaction-delay-specification)):\n\n```YAML\nsessions:\n- protocol:\n - name: http\n version: 2\n - name: tls\n sni: test_sni\n - name: tcp\n - name: ip\n version: 4\n\n keep-connection-open: 2s\n\n transactions:\n\n client-request:\n delay: 15ms\n\n method: POST\n url: /a/path.jpeg\n version: '1.1'\n headers:\n fields:\n - [ Content-Length, '399' ]\n - [ Content-Type, image/jpeg ]\n - [ Host, example.com ]\n - [ uuid, 1 ]\n\n server-response:\n delay: 17000 us\n\n status: 200\n reason: OK\n headers:\n fields:\n - [ Date, \"Sat, 16 Mar 2019 03:11:36 GMT\" ]\n - [ Content-Type, image/jpeg ]\n - [ Transfer-Encoding, chunked ]\n - [ Connection, keep-alive ]\n content:\n size: 3432\n```\n\n## Traffic Verification Specification\n\nIn addition to replaying HTTP traffic as described above, Proxy Verifier also\nimplements proxy traffic verification. For any given transaction, Proxy\nVerifier can optionally verify characteristics of an HTTP request line (for\nHTTP/1 requests - HTTP/2 requests do not have a request line), the response\nstatus, and the content of HTTP fields for requests and responses. Proxy\nVerifier also supports field verification of HTTP/2 pseudo header fields.\nWhereas `client-request` and `server-response` nodes direct the Verifier client\nand server (respectively) on how to send traffic, `proxy-request` and\n`proxy-response` nodes direct the server and client (respectively) on how to\nverify the traffic it receives from the proxy. Thus:\n\n* `client-request` nodes are used by the Verifier client to direct how it will\n generate requests it will send to the proxy.\n* `server-response` nodes are used by the Verifier server to direct how it will\n generate responses to send to the proxy.\n* `proxy-request` nodes are used by the Verifier server to direct how it will\n verify requests received from the proxy.\n* `proxy-response` nodes are used by the Verifier client to direct how it will\n verify responses received from the proxy.\n\nIn the event that the proxy does not produce the stipulated traffic according\nto a verification directive, the Proxy Verifier client or server that detects\nthe violation will emit a log indicating the violation and will, upon process\nexit, return a non-zero status to the shell.\n\nTraffic verification is an optional feature. Thus if Proxy Verifier is being\nused simply to replay traffic and the verification features are not helpful,\nthen the user can simply omit the `proxy-request` and `proxy-response` nodes\nand no verification will be performed.\n\nThe following sections describe how to specify traffic verification in the YAML\nreplay file.\n\n### Field Verification\n\nRecall that in `client-request` and `server-response` nodes, fields are\nspecified via a sequence of two items: the field name followed by the field\nvalue. For instance, the following incomplete `client-request` node contains\nthe specification of a single `Content-Length` field (it is incomplete because\nit does not specify the method, request target, etc., but this snippet\nsufficiently demonstrates a typical description of an HTTP field):\n\n```YAML\n client-request:\n headers:\n fields:\n - [ Content-Length, 399 ]\n```\n\nFor this client request, Proxy Verifier is directed to create a `Content-Length`\nHTTP field with a value of `399`.\n\nField verification is specified in a similar manner, but the second item in the\nsequence is a map describing how the field should be verified. The map takes a\n`value` item describing the characteristics of the value to verify, if\napplicable, and an `as` item providing a directive describing how the field\nshould be verified. Here is an example of a `proxy-request` node directing the\nVerifier server to verify the characteristics of the `Content-Length` field\nreceived from the proxy:\n\n```YAML\n proxy-request:\n headers:\n fields:\n - [ Content-Length, { value: 399, as: equal } ]\n```\n\nObserve the following from this verification example:\n\n* As described above, notice that the second entry in the field specification\n is a map instead of a scalar field value. The Proxy Verifier parser\n recognizes this map type as providing a verification specification.\n* As with `client-request` and `server-response` field specifications, the\n first item in the list describes the field name, in this case\n \"Content-Length\". This indicates that this verification rule applies to the\n \"Content-Length\" HTTP field from the proxy.\n* The directive is specified via the `as` key. In this example, `equal` is the\n directive for this particular field verification, indicating that the\n Verifier server should verify that the proxy's request for this transaction\n contains a `Content-Length` field with the exact value of `399`.\n\nProxy Verifier supports six field verification directives:\n\nDirective | Description\n--------- | ------------\nabsent | The *absence* of a field with the specified field name.\npresent | The *presence* of a field with the specified field name and having any or no value.\nequal | The presence of a field with the specified field name and a value *equal* to the specified value.\ncontains | The presence of a field with the specified name with a value *containing* the specified value.\nprefix | The presence of a field with the specified name with a value *prefixed* with the specified value.\nsuffix | The presence of a field with the specified name with a value *suffixed* with the specified value.\nincludes | Helpful for multi-value fields. Specifies that the set of value strings exist in the header field values for a particular header name in any order. Note that each value is matched against the set like ``contains``, so each value is a substring match.\n\nFor all of these field verification behaviors, field names are matched case\ninsensitively while field values are matched case sensitively.\n\nThus the following field specification requests no field verification because it does not include a directive and the second item in the sequence is a scarlar:\n\n```YAML\n - [ X-Forwarded-For, 10.10.10.2 ]\n```\n\nSuch non-operative fields can also be specified using the map syntax without an\n`as` item:\n\n```YAML\n - [ X-Forwarded-For, { value: 10.10.10.2 } ]\n```\n\nFields like this in `proxy-request` and `proxy-response` nodes are permissible\nby Proxy Verifier's parser even though they have no functional impact (i.e.,\nthey do not direct Proxy Verifier's traffic behavior because they are not in\n`client-request` nor `server-response` nodes, and they do not describe any\nverification behavior). Allowing such fields affords the user the ability to\nrecord the proxy's traffic behavior in situations where field verification is\nnot required or desired. For example, the [Traffic\nDump](https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/traffic_dump.en.html)\nTraffic Server plugin records HTTP traffic and uses these proxy HTTP fields to\nindicate what fields were sent by the Traffic Server proxy to the client and\nthe server. This can be helpful for analyzing the proxy's behavior via these\nreplay files. Thus this proxy traffic recording function can be helpful to the\nuser even though Proxy Verifier treats the fields as non-operable.\n\n\nThe following demonstrates the `absent` directive which specifies that the HTTP field `X-Forwarded-For` _with any value_ should not have been sent by the proxy:\n\n```YAML\n - [ X-Forwarded-For, { as: absent } ]\n```\n\nThe following demonstrates the `present` directive which specifies that the HTTP field `X-Forwarded-For` _with any value_ should have been sent by the proxy:\n\n```YAML\n - [ X-Forwarded-For, { as: present } ]\n```\n\nNotice that for both the `absent` and the `present` directives, the `value` map item is not relevant and need not be provided and, in fact, will be ignored by Proxy Verifier if it is provided.\n\nThe following demonstrates the `equal` directive which specifies that `X-Forwarded-For` should have been received from the proxy with the exact value \"10.10.10.2\":\n\n```YAML\n - [ X-Forwarded-For, { value: 10.10.10.2, as: equal } ]\n```\n\nThe following demonstrates the `contains` directive which specifies that `X-Forwarded-For` should have been received from the proxy containing the value \"10\" at any position in the field value:\n\n```YAML\n - [ X-Forwarded-For, { value: 10, as: contains } ]\n```\n\nThe following demonstrates the `prefix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value starting with \"1\":\n\n```YAML\n - [ X-Forwarded-For, { value: 1, as: prefix } ]\n```\n\nThe following demonstrates the `suffix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value ending with \"2\":\n\n```YAML\n - [ X-Forwarded-For, { value: 2, as: suffix } ]\n```\n\nThe following demonstrates the `includes` directive which specifies that `Set-Cookie` should have been received from the proxy including the values \"A\" and \"B\" at any position in the field value:\n\n```YAML\n - [ Set-Cookie, { value: [A, B] , as: includes } ]\n```\n\nProxy Verifier supports inverting the result of any rule by using `not` instead of `as`. The following demonstrates the `prefix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value not starting with \"a\":\n\n```YAML\n - [ X-Forwarded-For, { value: a, not: prefix } ]\n```\n\nProxy Verifier also supports ignoring the upper/lower case distinction with another directive: `case: ignore`. The following demonstrates the `suffix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value starting with \"a\" or \"A\":\n\n```YAML\n - [ X-Forwarded-For, { value: a, as: prefix, case: ignore } ]\n```\n\nThe `not` and `case: ignore` directives can both be applied on the same rule. The following demonstrates the `suffix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value not starting with \"a\" nor \"A\":\n\n```YAML\n - [ X-Forwarded-For, { value: a, not: prefix, case: ignore } ]\n```\n\nIn addition to HTTP/2 trailer header replay discussed above, Proxy Verifier also supports trailer header verification, as demonstrated below:\n\n```YAML\nproxy-response:\n # Other verifications...\n trailers:\n fields:\n # Verify the client receives the response trailers.\n - [ x-test-trailer-1, { value: one, as: equal } ]\n - [ x-test-trailer-2, { value: two, as: equal } ]\n```\n\nTo perform multi-value field verification, a specific format must be adhered to.\nThis format involves specifying each value within a sequence, ensuring that each\nvalue is individually verified according to the defined rules.\n\nBe aware that field verification is order sensitive. That is, the field values\nwill be verified in the order that they are specified in the verification rule.\n\nSee example below, value `B1` and `B2` are specified in the same order in the\nheader fields and verification rule:\n\n```YAML\nserver-response:\n headers:\n fields:\n - [:status, 200]\n - [Content-Type, text/html]\n - [Content-Length, '11']\n - [Set-Cookie, \"A1=111\"]\n - [Set-Cookie, \"A2=222\"]\n - [Set-Cookie, \"B1=333\"]\n content:\n encoding: plain\n data: server_test\n size: 11\n\nproxy-response:\n headers:\n fields:\n - [Set-Cookie, { value: [A1=111, A2=222, B1=333] , as: equal }]\n```\n\nThere is an exception for the `includes` check, where it is not order sensitive.\nSee exammple below:\n\n```YAML\nserver-response:\n headers:\n fields:\n - [:status, 200]\n - [Content-Type, text/html]\n - [Content-Length, '11']\n - [Set-Cookie, \"A1=111\"]\n - [Set-Cookie, \"A2=222\"]\n - [Set-Cookie, \"B1=333\"]\n - [Set-Cookie, \"B2=444\"]\n - [Set-Cookie, \"C1=555\"]\n - [Set-Cookie, \"C2=666\"]\n - [Set-Cookie, \"D1=777\"]\n - [Set-Cookie, \"D2=888\"]\n content:\n encoding: plain\n data: server_test\n size: 11\n\nproxy-response:\n headers:\n fields:\n - [Set-Cookie, { value: [B2=, A2=, C2=, D1=, C1=] , as: includes }]\n```\n\n### URL Verification\n\nIn a manner similar to field verification described above, a mechanism exists\nto verify the parts of URLs in the request line being received from the proxy\nby the server. This mechanism is useful for verifying the request targets of\nHTTP/1 requests. For HTTP/2 requests, the analogous verification is done via\npseudo header field verification of the `:scheme`, `:authority`, and\n`:path` fields using the above described field verification.\n\nRequest target verification rules are stipulated via a sequence value for the\n`url` node rather than the scalar value used in `client-request` nodes. The\nverifiable parts of the request target follow the URI specification (see\n[RFC 3986 section 3](https://tools.ietf.org/html/rfc3986#section-3) for the formal\ndefinition of these various terms):\n\n* `scheme`\n* `host`\n* `port`\n* `authority` (also known as `net-loc`, the combination of `host` and `port`),\n* `path`\n* `query`\n* `fragment`\n\nFor example, consider the following request line:\n\n```\n GET http://example.one:8080/path?query=q#Frag HTTP/1.1\n```\n\nThe request URL in this case is case is\n`http://example.one:8080/path?query=q#Frag`. The Verifier server can be\nconfigured to verify the various parts of such URLs using the same map sytax\nexplained above for field verification using any of those same directives\n(`equal`, `contains`, etc.). Continuing with this example URL, the following\nuses the `equal` directive for each part of the URL, thus verifying that the\nrequest URL exactly matches the URL given in this example and emitting a\nverification error message if any parts of the URL do not match:\n\n```YAML\n proxy-request:\n url:\n - [ scheme, { value: http, as: equal } ]\n - [ host, { value: example.one, as: equal } ]\n - [ port, { value: 8080, as: equal } ]\n - [ path, { value: /path, as: equal } ]\n - [ query, { value: query=q, as: equal } ]\n - [ fragment, { value: Frag, as: equal } ]\n```\n\nAlternatively to `host` and `port`, `authority`, with an alias of `net-loc`, is\nsupported, which is the combination of the two:\n\n```YAML\n - [ authority, { value: example.one:8080, as: equal } ]\n```\n\nAs another example using other directives, consider a request URL of\n`/path/x/y?query=q#Frag`. Verification of this can be specified with the\nfollowing:\n\n```YAML\n proxy-request:\n url:\n - [ scheme, { value: http, as: absent } ]\n - [ authority, { value: foo, as: absent } ]\n - [ path, { value: /path/x/y, as: equal } ]\n - [ query, { value: query=q, as: equal } ]\n - [ fragment, { value: foo, as: present } ]\n```\n\nNote that `scheme` and `authority` parts both use `absent` directives because\nthis particular URL just has path, query, and fragment components. Thus, with\nthis verification specification, if the proxy includes scheme or authority in\nthe request target, it will result in a verification failure. The `path` and\n`query` components must match `/path/x/y` and `query=q` exactly because they\nuse the `equal` directive. The `fragment` of the URL is verified with the\n`present` directive in this case, indicating that the received URL from the\nproxy only needs to have some fragment of any value to pass this specified\nverification.\n\n### Status Verification\n\nProxy HTTP response status verification is specified in `proxy-response`\nnodes. In this case, the response status that the Verifier client should expect\nfrom the proxy is specified in the same way that directs the Verifier server\nin what response status should be sent for a given request. Both response code,\nsuch as \"403\", and HTTP/1 response reason string, such as \"Forbidden\", are\nsupported for verification.\n\nFor example, the following complete `proxy-response` node directs the Proxy\nVerifier client to verify that the proxy replies to the associated HTTP request\nwith a `404` status:\n\n```YAML\n proxy-response:\n status: 404\n reason: Not Found\n```\n\nThis verification directive applies to HTTP/1 transactions. For HTTP/2, status\nverification is specified via `:status` pseudo header field verification using\nthe field verification mechanism described above.\n\n### Body Verification\n\nIn a manner similar to field verification described above, a mechanism exists\nto verify the body content of a request or response. To specify a rule for\nverifying body content, a new `verify` node should be added under the `content`\nnode. The rules follow the same map syntax as described for field verification.\n\n```YAML\n proxy-request:\n content:\n verify: {value: test1, as: equal}\n\n proxy-response:\n content:\n verify: {value: test2, as: contains}\n```\n\nThe `value` node in the `verify` node can be ommited if the `data` node is used\nto specify the content, since body content can get very long, and/or multi-lined.\nHowever, `value` node has priority over the `data` node, meaning if there is a\n`value` node, then the `data` will be ignored.\n\n```YAML\n proxy-request:\n content:\n encoding: plain\n data: test1\n verify: {as: equal}\n\n proxy-response:\n content:\n encoding: plain\n data: test2\n verify: {as: contains}\n```\n\nNote that only one body verification node is needed, even if multiple `DATA` frames\nare specified as described in [HEADERS and DATA frame](#headers-and-data-frame). The\nverification node need to combine all the values specified for the `DATA` frames.\nSee the example below:\n\n```YAML\n client-request:\n frames:\n - HEADERS:\n headers:\n fields:\n - [:method, POST]\n - [:scheme, https]\n - [:authority, example.data.com]\n - [:path, /a/path]\n - [Content-Type, text/html]\n - [uuid, 1]\n - DATA:\n content:\n encoding: plain\n data: client_data_1\n - DATA:\n content:\n encoding: plain\n data: client_data_2\n\n proxy-request:\n content:\n encoding: plain\n data: client_data_1client_data_2\n verify: {as: equal}\n```\n\n### Example Replay File\n\nThe sections leading up to this one have described each of the major components\nof a YAML Proxy Verifier replay file. Putting these components together, the\nfollowing complete sample replay file demonstrates the description of the\nreplay of two sessions: one an HTTP/1.1 session, the second an HTTP/2 session.\nEach session contains a single transaction with verification of certain parts\nof the HTTP messages.\n\n\u003c!---\nNOTE: doctest.test.py runs with this exact replay file and should be maintained\nas such to verify that this sample replay file is valid. It would be really\nannoying to users, and wasteful of their time, if this file, as is, does not\nrun correctly.\n--\u003e\n\n```YAML\nmeta:\n version: '1.0'\n\nsessions:\n\n#\n# First session: since there is no \"protocol\" node for this session,\n# HTTP/1.1 over TCP (no TLS) is assumed.\n#\n- transactions:\n\n #\n # Direct the Proxy Verifier client to send a POST request with a body of\n # 399 bytes.\n #\n - client-request:\n method: POST\n url: /pictures/flower.jpeg\n version: '1.1'\n headers:\n fields:\n - [ Host, www.example.com ]\n - [ Content-Type, image/jpeg ]\n - [ Content-Length, '399' ]\n - [ uuid, first-request ]\n # A \"content\" node is not needed if a Content-Length field is specified.\n\n #\n # Direct the Proxy Verifier server to verify that the request received from\n # the proxy has a path in the request target that contains \"flower.jpeg\",\n # has a path that is not prefixed with \"JPEG\" (case insensitively),\n # and has the Content-Length field of any value.\n #\n proxy-request:\n url:\n - [ path, { value: flower.jpeg, as: contains } ]\n - [ path, { value: JPEG, not: prefix, case: ignore } ]\n\n headers:\n fields:\n - [ Content-Length, { value: '399', as: present } ]\n\n #\n # Direct the Proxy Verifier server to reply with a 200 OK response with a body\n # of 3,432 bytes.\n #\n server-response:\n status: 200\n reason: OK\n headers:\n fields:\n - [ Date, \"Sat, 16 Mar 2019 03:11:36 GMT\" ]\n - [ Content-Type, image/jpeg ]\n - [ Transfer-Encoding, chunked ]\n - [ Connection, keep-alive ]\n # Unlike the request which contains a Content-Length, this response\n # will require a \"content\" node to specify the size of the body.\n # Otherwise Proxy Verifier has no way of knowing how large the response\n # should be.\n content:\n size: 3432\n\n #\n # Direct the Proxy Verifier client to verify that it receives a 200 OK from\n # the proxy with a `Transfer-Encoding: chunked` header field.\n #\n proxy-response:\n status: 200\n headers:\n fields:\n - [ Transfer-Encoding, { value: chunked, as: equal } ]\n\n#\n# For the second session, we use a protocol node to configure HTTP/2 using an\n# SNI of # test_sni in the TLS handshake.\n#\n- protocol:\n - name: http\n version: 2\n - name: tls\n sni: test_sni\n - name: tcp\n - name: ip\n\n transactions:\n\n #\n # Direct the Proxy Verifier client to send a POST request with a body of\n # 399 bytes.\n #\n - client-request:\n headers:\n fields:\n - [ :method, POST ]\n - [ :scheme, https ]\n - [ :authority, www.example.com ]\n - [ :path, /pictures/flower.jpeg ]\n - [ Content-Type, image/jpeg ]\n - [ uuid, second-request ]\n content:\n size: 399\n\n #\n # Direct the Proxy Verifier server to verify that the request received from\n # the proxy has a path pseudo header field that contains \"flower.jpeg\"\n # and has a field \"Content-Type: image/jpeg\".\n #\n proxy-request:\n url:\n - [ path, { value: flower.jpeg, as: contains } ]\n\n headers:\n fields:\n - [ :method, POST ]\n - [ :scheme, https ]\n - [ :authority, www.example.com ]\n - [ :path, { value: flower.jpeg, as: contains } ]\n - [ Content-Type, { value: image/jpeg, as: equal } ]\n\n #\n # Direct the Proxy Verifier server to reply with a 200 OK response with a body\n # of 3,432 bytes.\n #\n server-response:\n headers:\n fields:\n - [ :status, 200 ]\n - [ Date, \"Sat, 16 Mar 2019 03:11:36 GMT\" ]\n - [ Content-Type, image/jpeg ]\n content:\n size: 3432\n\n #\n # Direct the Proxy Verifier client to verify that it receives a 200 OK from\n # the proxy.\n #\n proxy-response:\n status: 200\n\n#\n# For the third session, we demonstrate how body verification should be specified.\n#\n- protocol:\n - name: http\n version: 1.1\n - name: tls\n sni: test_sni\n - name: tcp\n - name: ip\n version: 4\n\n transactions:\n\n #\n # Direct the Proxy Verifier client to send a POST request with a body of\n # 11 bytes.\n #\n - client-request:\n method: POST\n url: /a/path\n version: '1.1'\n headers:\n fields:\n - [ Host, example.data.com ]\n - [ Content-Type, text/html ]\n - [ Content-Length, '11' ]\n - [ uuid, third-request ]\n content:\n encoding: plain\n data: client_test\n size: 11\n\n #\n # Direct the Proxy Verifier server to verify that the request received from\n # the proxy has body content \"client_test\".\n #\n proxy-request:\n content:\n verify: { value: client_test, as: equal }\n\n #\n # Direct the Proxy Verifier server to reply with a 200 OK response with a body\n # of 11 bytes.\n #\n server-response:\n status: 200\n reason: OK\n headers:\n fields:\n - [ Content-Type, text/html ]\n - [ Content-Length, '11' ]\n content:\n encoding: plain\n data: |-\n server\n test\n size: 11\n\n #\n # Direct the Proxy Verifier client to verify that the response received from\n # the proxy has body content \"server\\ntest\".\n #\n proxy-response:\n content:\n encoding: plain\n data: |-\n server\n test\n verify: { as: equal }\n```\n\n## Installing\n\n### Prebuilt Binaries\n\nStarting with the v2.2.0 release, statically linked binaries for Linux and Mac\nare provided with the release in the\n[Releases](https://github.com/yahoo/proxy-verifier/releases) page. If you do not\nneed your own customized build of Proxy Verifier, the easiest way to start\nusing it is to simply download the proxy-verifier `tar.gz` for the desired\nrelease, untar it on the desired box, and copy the `verifier-client` and\n`verifier-server` binaries to a convenient location from which to run them.\nThe Linux binaries should run on Ubuntu, Alma/CentOS/Fedora/RHEL, FreeBSD, and\nother Linux flavors.\n\n### Building from Source\n\nThese instructions describe how to build a copy of the Proxy Verifier project\non your local machine for development and testing purposes.\n\n#### Prerequisites\n\nProxy Verifier is built using [SCons](https://scons.org). Scons is a Python\nmodule, so installing it is as straightforward as installing any Python\npackage. A top-level\n[Pipfile](https://github.com/yahoo/proxy-verifier/tree/master/Pipfile) is\nprovided to install Scons and its use is described and assumed in these\ninstructions, but it can also be installed using pip if preferred.\n\nScons will clone and build the dependent libraries using Automake. Thus\nbuilding will require the installation of the following system packages:\n\n* git\n* pipenv\n* autoconf\n* libtool\n* pkg-config\n\nFor system-specific commands to install these packages (Ubuntu, CentOS, etc.),\none can view the\n[Dockerfile](https://docs.docker.com/engine/reference/builder/) documents\nprovided under\n[docker](https://github.com/yahoo/proxy-verifier/tree/master/docker). These\ndemonstrate, for each system, what commands are used to install these package\ndependencies. Naturally, performing a `docker build` against these Dockerfiles\ncan also be used to create Docker images, containers from which builds can be\nperformed.\n\nIn addition to the above system package dependencies, Proxy Verifier utilizes\nthe following C++ libraries:\n\n* [OpenSSL](https://www.openssl.org) is used to implement TLS encryption.\n Proxy Verifier requires the version of OpenSSL that supports QUIC.\n* [Nghttp2](https://nghttp2.org) is used for parsing HTTP/2 traffic.\n* [ngtcp2](https://github.com/ngtcp2/ngtcp2) is used for parsing QUIC\n traffic.\n* [nghttp3](https://github.com/ngtcp2/nghttp3) is used for parsing HTTP/3\n traffic.\n* [yaml-cpp](https://github.com/jbeder/yaml-cpp) is used for parsing the YAML\n replay files.\n* [libswoc](https://github.com/SolidWallOfCode/libswoc) are a set of C++\n library extensions to support string parsing, memory management, logging, and\n other features.\n\n*Note*: None of these libraries need to be explicitly installed before you\nbuild. By default, Scons will fetch and build each of these libraries as a\npart of building the project.\n\n#### Build\n\nOnce the above-listed system packages (git, autoconf, etc.) are installed on\nyour system, you can build Proxy Verifier using Scons. This involves first\ncreating the Python virtual environment and then running the `scons` command\nto build Proxy Verifier. Here is an example invocation:\n\n```\n# Install scons and any of its Python requirements. This only needs to be\n# done once before the first invocation of scons.\n#\n# Note: for older RHEL/CentOS systems, you will have to souce the appropriate\n# Python 3 enable script to initialize the correct Python 3 environment. For\n# example:\n# source /opt/rh/rh-python38/enable\npipenv install\n\n# Now run scons to build proxy-verifier.\npipenv run scons -j4\n```\n\nThis will build and install `verifier-client` and `verifier-server` in the\n`bin/` directory at the root of the repository. `-j4` directs Scons to build\nwith 4 threads. Adjust according to the capabilities of your build system.\n\n#### Using Prebuilt Libraries\n\nAs mentioned above, Scons will by default fetch the various library\ndependencies (OpenSSL, Nghttp2, etc.), build, and manage those for you. If you\ndo not change the fetched source code for these libraries, they will not be\nrebuilt after the first `scons` build invocation. This behavior is convenient\nas it relieves the burden of fetching and building these libraries from the\ndeveloper. However, Scons will rescan the fetched source trees for these\nlibraries on every call of `scons` to inspect them for any changes. For\nlong-term development projects, a developer may find it more efficient to build\nthese libraries externally and relieve Scons from managing them. To\nconveniently support this, the\n[build_library_dependencies.sh](https://github.com/yahoo/proxy-verifier/blob/master/tools/build_library_dependencies.sh)\nscript is provided to build these libraries. To build and install the\nlibraries, run that script, passing as an argument the desired install location\nfor the various libraries. Then point Scons to those libraries using various\n`--with` directives.\n\nHere's an example invocation of `scons` along with the use of the library build\nscript:\n\n```\n# Alter this to your desired library location.\nhttp3_libs_dir=${HOME}/src/http3_libs\n\nbash ./tools/build_http3_dependencies.sh ${http3_libs_dir}\n\npipenv install\npipenv run scons \\\n -j4 \\\n --with-ssl=${http3_libs_dir}/openssl \\\n --with-nghttp2=${http3_libs_dir}/nghttp2 \\\n --with-ngtcp2=${http3_libs_dir}/ngtcp2 \\\n --with-nghttp3=${http3_libs_dir}/nghttp3\n```\n\nThe [Dockerfile](https://github.com/yahoo/proxy-verifier/tree/master/docker)\ndocuments run this build script, installing the HTTP packages in `/opt`.\nTherefore, if you are developing in a container made from images generated from\nthese Dockerfile documents, you can use the following `scons` command to build\nProxy Verifier:\n\n```\npipenv install\npipenv run scons \\\n -j4 \\\n --with-ssl=/opt/openssl \\\n --with-nghttp2=/opt/nghttp2 \\\n --with-ngtcp2=/opt/ngtcp2 \\\n --with-nghttp3=/opt/nghttp3\n```\n\nAs a further convenience, if these libraries (`openssl`, `nghttp2`, `ngtcp2`,\nand `nghttp3`, with those exact names) exist under a single directory, such as\nis the case with images built from the provided\n[Dockerfile](https://github.com/yahoo/proxy-verifier/tree/master/docker)\ndocumemnts, then you can specify the location of these libraries with a single\n`--with-libs` argument. Thus the previous command can be expressed like so:\n\n```\npipenv install\npipenv run scons -j4 --with-libs=/opt\n```\n\n#### ASan Instrumentation\n\nThe local Sconstruct file is configured to take an optional `--enable-asan`\nparameter. If this is passed to the `scons` build line then the Proxy Verifier\nobjects and binaries will be compiled and linked with the flags that instrument\nthem for [AddressSanatizer](https://clang.llvm.org/docs/AddressSanitizer.html).\nThis assumes that the system has the AddressSanatizer library installed on the\nsystem. Thus the above invocation would look like the following to compile it\nwith AddressSanitizer instrumentation:\n\n```\npipenv install\npipenv run scons \\\n -j4 \\\n --with-ssl=/path/to/openssl \\\n --with-nghttp2=/path/to/nghttp2 \\\n --with-ngtcp2=/path/to/ngtcp2 \\\n --with-nghttp3=/path/to/nghttp3 \\\n --enable-asan\n```\n\n#### Debug Build\n\nBy default, Scons will build the Proxy Verifier project in `release` mode. This\nmeans that the binaries will compiled with optimization. If an unoptimized\ndebug build is desired, then pass the `--cfg=debug` option to `scons`:\n\n```\npipenv run scons -j4 --cfg=debug\n```\n\n#### Statically Link\n\nThe current Scons configuration dynamically links the binaries with the various\nOpenSSL and HTTP build libraries. This is fine for local testing and execution,\nbut can be inconvenient when copying the binaries to other machines. Ideally\nthe\n[Sconstruct](https://github.com/yahoo/proxy-verifier/blob/master/Sconstruct)\nfile can be updated to support an option to link the binaries statically. This\ncurrently does not exist and is not trivial to create. Future updates to\n`scons-parts` may help with this. As a current stopgap measure,\n[tools/build_static](https://github.com/yahoo/proxy-verifier/blob/master/tools/build_static)\nis provided which automatically links the Verifier binaries statically. It is\nrun from the root directory of your repository like so:\n\n```\n./tools/build_static\n```\n\nBy default this builds Proxy Verifier with the following invocation:\n\n```\npipenv run scons -j$(nproc)\n```\n\nAny arguments passed to `build_static` will be passed through to the scons\ncommand. Thus, if you desire to build Proxy Verifier with `--with-libs=/opt`,\nrun the script like so:\n\n```\n./tools/build_static --with-libs=/opt`\n```\n\n### Running the Tests\n\n#### Unit Tests\n\nTo build and run the unit tests, use the `run_utest` Scons target (this assumes\nyou previously ran `pipenv install`, see above):\n\n```\npipenv run scons \\\n -j4 \\\n --with-ssl=/path/to/openssl \\\n --with-nghttp2=/path/to/nghttp2 \\\n --with-ngtcp2=/path/to/ngtcp2 \\\n --with-nghttp3=/path/to/nghttp3 \\\n run_utest::\n```\n\n#### Gold Tests\nProxy Verifier ships with a set of automated end to end tests written using the\n[AuTest](https://bitbucket.org/autestsuite/reusable-gold-testing-system/src/master/)\nframework. To run them, simply run the `autest.sh` script:\n\n```\ncd test/autests\n./autest.sh\n```\n\nWhen doing development for which a particular AuTest is relevant, the `-f`\noption can be used to run just a particular test (or set of tests, specified\nin a space-separated list). For instance, the following invocation runs\njust the http and https tests:\n\n```\n./autest.sh -f http https\n```\n\nAuTest supports a variety of other options. Run `./autest.sh --help` to get a\nquick description of the various command-line options. See the [AuTest\nDocumentation](https://autestsuite.bitbucket.io) for further details about the\nframework.\n\n**A note for macOS**: The Python virtual environment for these gold tests\nrequires the [cryptograpy](https://github.com/pyca/cryptography) package as a\ndependency of the [pyOpenSSL](https://www.pyopenssl.org/en/stable/) package.\nPipenv will install this automatically, but the installation of the\n`cryptography` package will require compiling certain c files against OpenSSL.\nmacOS has its own SSL libraries which brew's version of OpenSSL does not\nreplace, for understandable reasons. The building of `cryptography` will\nfail against the system's SSL libraries. To point the build to brew's OpenSSL\nlibraries, the `autest.sh` script exports the following variables before\nrunning `pipenv install`:\n\n```\nexport LDFLAGS=\"-L/usr/local/opt/openssl/lib\"\nexport CPPFLAGS=\"-I/usr/local/opt/openssl/include\"\nexport PKG_CONFIG_PATH=\"/usr/local/opt/openssl/lib/pkgconfig\"\n```\n\nThus if you stick with using the `autest.sh` script you do not need to worry\nabout this. But if you install pipenv by hand rather than through the\n`autest.sh` script on macOS, then keep this in mind and export those variables\nbefore running `pipenv install`.\n\n## Usage\n\nThis section describes how to run the Proxy Verifier client and server at\nthe command line.\n\n### Required Arguments\n\nAt a high level, Proxy Verifier is run in the following manner:\n\n1. Run the verifier-server with the set of HTTP and HTTPS ports to listen on\n configured though the command line. The directory containing the replay file\n is also configured through a command line argument.\n1. Configure and run the proxy to listen on a set of HTTP and HTTPS ports and\n to proxy those connections to the listening verifier-server ports.\n1. Run the verifier-client with the sets of HTTP and HTTPS ports on which to\n connect configured though the command line. The directory containing the\n replay file is also configured through a command line argument.\n\nHere's an example invocation of the verifier-server, configuring it to listen on\nlocalhost port 8080 for HTTP connections and localhost port 4443 for HTTPS\nconnections:\n\n```\nverifier-server \\\n run \\\n --listen-http 127.0.0.1:8080 \\\n --listen-https 127.0.0.1:4443 \\\n --server-cert \u003cserver_cert\u003e \\\n --ca-certs \u003cfile_or_directory_of_ca_certs\u003e \\\n \u003creplay_file_or_directory\u003e\n```\n\nHere's an example invocation of the verifier-client, configuring it to connect to\nthe proxy which has been configured to listen on localhost port 8081 for HTTP\nconnections and localhost port 4444 for HTTPS connections:\n\n```\nverifier-client \\\n run \\\n --client-cert \u003cclient_cert\u003e \\\n --ca-certs \u003cfile_or_directory_of_ca_certs\u003e \\\n --connect-http 127.0.0.1:8081 \\\n --connect-https 127.0.0.1:4444 \\\n \u003creplay_file_or_directory\u003e\n```\n\nWith these two invocations, the verifier-client and verifier-server will replay the\nsessions and transactions in `\u003creplay_file_or_directory\u003e` and perform any field\nverification described therein.\n\nOn the server either `--listen-http` or `--listen-https` or both must be\nprovided. That is, for example, if you are only testing HTTPS traffic, you may\nonly specify `--listen-https`. The same is true on the client: either\n`--connect-http` or `--connect-https` or both must be provided. These address\narguments take a comma-separated list of address/port pairs to specify multiple\nlistening or connecting sockets. The processing of these arguments\nautomatically detects any IPv6 addresses if provided. The client's processing\nof `--connect-http` and `--connect-https` arguments will resolve fully\nqualified domain names.\n\nNote that the `--client-cert` and `--server-cert` both take either a\ncertificate file containing the public and private key or a directory\ncontaining pem and key files. Similarly, the `--ca-certs` takes either a file\ncontaining one or more certificates or a directory with separate certificate\nfiles. For convenience, the\n[test/keys](https://github.com/bneradt/proxy-verifier/tree/expand_readme/test/keys)\ndirectory contains key files which can be used for testing. These certificate\narguments are only required if HTTPS traffic will be replayed.\n\n### Optional Arguments\n\n#### --format \\\u003cformat-specification\\\u003e\n\nEach transaction has to be uniquely identifiable by the client and server in a\nway that is consistent across both replay file parsing and traffic replay\nprocessing. Whatever attributes we use from the messages to uniquely identify\ntransactions is called the \"key\" for the dataset. The ability to uniquely\nidentify these messages is important for at least the following reasons:\n\n* When the Verifier server receives a request, it has to know from which of the\n set of parsed transactions it should generate a response. At the time of\n processing an incoming message, all it has to go on is the request header\n line and the request header fields. From these, it has to be able to identify\n which of the potentially thousands of parsed transactions from the replay input\n files it should generate a response.\n* When the client and server perform field verification, they need to know what\n particular verification rules specified in the replay files should be applied\n to the given incoming message.\n* If the client and server are processing many transactions, generic log\n messages could be near useless if there was not a way for the logs to\n identify individual transactions to the user somehow.\n\nBy default the Verifier client and server both expect a `uuid` header field\nvalue to function as the key.\n\nIf the user would like to use other attributes as a key, they can specify\nsomething else via the `--format` argument. The format argument currently\nsupports generating a key on arbitrary field values and the `URL` of the\nrequest. Some example `--format` expressions include:\n\n* `--format \"{field.uuid}\"`: This is the default key format. It treats the UUID\n header field value as the transaction key.\n* `--format \"{url}\"`: Treat the request `URL` as the key.\n* `--format \"{field.host}\"`: Treat the `Host` header field value as the key.\n* `--format \"{field.host}/{url}\"`: Treat the combination of the `Host` header\n field and the request `URL` as the key.\n\n#### --keys \\\u003ckey1 key2 ... keyn\\\u003e\n\n`--keys` can be passed to the verifier-client to specify a subset of keys from\nthe replay file to run. Only the transactions from the space-separated list of\nkeys will be replayed. For example, the following invocation will only run the\ntransactions with keys whose values are 3 and 5:\n\n```\nverifier-client \\\n run \\\n \u003creplay_file_diretory\u003e \\\n 127.0.0.1:8082 \\\n 127.0.0.1:4443 \\\n --keys 3 5\n```\n\nThis is a client-side only option.\n\n#### --verbose\n\nProxy Verifier has four levels of verbosity that it can run with:\n\n| Verbosity | Description |\n| --------- | ----------- |\n| error | Transactions either failed to run or failed verification. |\n| warning | A non-failing problem occurred but something is likely to go wrong in the future. |\n| info | High level test execution information. |\n| diag | Low level debug information. |\n\n\nEach level implies the ones above it. Thus, if a user specifies a verbosity\nlevel of `warning`, then both warning and error messages are reported.\n\nBy default, Proxy Verifier runs at `info` verbosity, only producing summary\noutput by both the client and the server along with any warnings and errors it\nfound. This can be tweaked via the `--verbose` flag. Here's an example of requesting\nthe most verbose level of logging (`diag`):\n\n```\nverifier-client \\\n run \\\n \u003creplay_file_diretory\u003e \\\n 127.0.0.1:8082 \\\n 127.0.0.1:4443 \\\n --verbose diag\n```\n\n#### --interface \\\u003cinterface\\\u003e\n\nInitiate connections from the specified interface, such as eth0:1.\n\nThis is a client-side only option.\n\n#### --no-proxy\n\nAs explained above, replay files contain traffic information for both client to\nproxy traffic and proxy to server traffic. Under certain circumstances it may\nbe helpful to run the Verifier client directly against the Verifier server.\nThis can be useful while developing Proxy Verifier itself, for example,\nallowing the developer to do some limited testing without requiring the setup\nof a test proxy.\n\nTo support this, the Verifier client has the `--no-proxy` option. If this\noption is used, then the client has its expectations configured such that it\nassumes it is run against the Verifier server rather than a proxy. Effectively\nthis means that instead of trying to run the client to proxy traffic, it will\ninstead act as the proxy host for the Verifier server and will run the proxy to\nserver traffic. Concretely, this means that the Verifier client will replay the\n`proxy-request` and `proxy-response` nodes rather than the `client-request` and\n`client-response` nodes.\n\nThis is a client-side only option.\n\n#### --strict\n\nGenerally, very little about the replayed traffic is verified except what is\nexplicitly specified via field verification (see above). This is by design,\nallowing the user to replay traffic with only the requested content being\nverified. In high-volume cases, such as situations where Proxy Verifier is\nbeing used to scale test the proxy, traffic verification may be considered\nunimportant or even unnecessarily noisy. If, however, the user wants every\nfield to be verified regardless of specification, then the `--strict` option\ncan be passed to either or both the Proxy Verifier client and server to report\nany verification issues against every field specified in the replay file.\n\n#### --rate \\\u003crequests/second\\\u003e\n\nBy default, the client will replay the session and transactions in the replay\nfiles as fast as possible. If the user desires to configure the client to\nreplay the transactions at a particular rate, they can provide the `--rate`\nargument. This argument takes the number of requests per second the client will\nattempt to send requests at.\n\nNote session and transaction timing data can be specified in the replay files.\nThese are provided via `start-time` nodes for each `session` and `transaction`.\n`start-time` takes as a value the number of nanoseconds since Unix epoch (or\nwhatever other time of reference observed by all `start-time` nodes in the set\nof replay files being run) associated for that session or transaction. With\nthis timing information, if `--rate` is provided, Proxy Verifier simply scales\nthe relative time deltas between sessions and transactions that appropriately\nachieves the desired transaction rate. [Traffic\nDump](https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/traffic_dump.en.html)\nrecords such timing information when it writes replay files. In the absence of\n`start-time` nodes, Proxy Verifier will attempt to apply an appropriate uniform\ndelay across the sessions and transactions to achieve the specified `--rate`\nvalue.\n\nThis is a client-side only option.\n\n#### --repeat \\\u003cnumber\\\u003e\n\nBy default, the client will replay all the transactions once in the set of\ninput replay files. If the user would like the client to automatically repeat\nthis set a number of times, they can provide the `--repeat` option. The\nargument takes the number of times the client should replay the entire dataset.\n\nThis is a client-side only option.\n\n#### --run-continuously\n\nRun the set of sessions in the input replay files in an infinite loop. Passing\nthis option can be thought of as `--repeat` with a value of infinity.\n\nThis is a client-side only option.\n\n#### --thread-limit \\\u003cnumber\\\u003e\n\nEach connection, corresponding to a `session` in a replay file, is dispatched\non the client in parallel. Likewise, each accepted connection on the server is\nhandled in parallel. Each of these sessions is handled via a single thread of\nexecution. By default, Proxy Verifier limits the number of threads for handling\nthese connections to 2,000. This limit can be changed via the `--thread-limit`\noption. Setting a value of 1 on the client will effectively cause sessions\nto be replayed in serial.\n\n#### --qlog-dir \\\u003cdirectory\\\u003e\n\nProxy Verifier supports logging of replayed QUIC traffic information conformant\nto the qlog format. If the `--qlog-dir` option is provided, then qlog files for all\nreplayed QUIC traffic will be written into the specified directory. qlog\ndiagnostic logging is disabled by default.\n\n#### --tls-secrets-log-file \\\u003csecrets_log_file_name\\\u003e\n\nTo facilitate debugging, Proxy Verifier supports logging TLS keys for encrypted\nreplayed traffic. If this option is used, TLS key logging will be appended to\nthe specified filename. This file can then be provided to protocol analyzers\nsuch as Wireshark to decrypt the traffic. TLS key logging is disabled by\ndefault.\n\n#### --send-buffer-size \\\u003csize\\\u003e\n\nConfigure a `SO_SNDBUF` value to set on the server socket. This can be a\nhelpful parameter to tune when dealing with the replay of large response\nbodies. If this optional value is not set, then the `SO_SNDBUF` option is not\nset on the socket and, thus, the system default socket buffer size will be\nused.\n\nThis is a server-side only option.\n\n#### --poll-timeout \\\u003ctimeout_ms\\\u003e\n\nWhen Proxy Verifier performs read and write operations, it does so using\nnon-blocking sockets with a timeout. By default, this timeout is set to 5\nseconds (5,000 milliseconds). This optional argument provides a way to specify a\ndifferent timeout in milliseconds for these operations.\n\n## Tools\nThis section describes how to use some of the scripts under the [tools](tools) directory.\n\n### Replay Gen [replay_gen.py](tools/replay_gen.py)\nThis tool is used to generate mock replay files for easy testing.\nListed below are the available arguments for this script.\n\n#### -n,--number \\\u003cNUMBER\\\u003e\nNumber of total transactions.\n\n#### -tl,--trans-lower \\\u003cTRANS_LOWER\\\u003e\nThe lower limit of transactions per session.\n\n#### -tu,--trans-upper \\\u003cTRANS_UPPER\\\u003e\nThe upper limit of transactions per session.\n\n#### -sl,--sess-lower \\\u003cSESS_LOWER\\\u003e\nThe lower limit of sessions per file.\n\n#### -su,--sess-upper \\\u003cSESS_UPPER\\\u003e\nThe upper limit of sessions per file.\n\n#### -tp,--trans-protocols \\\u003cTRANS_PROTOCOLS\\\u003e\nA comma separated list of protocols that are allowed to be generated.\nAvailable options are: http, tls, h2, all.\n\n#### -u,--url-file \\\u003cURL_FILE\\\u003e\nPath to a file with the list of URLs that can be used.\nThe URL list file can be acquired by running the [Remap Config to URL List]() script described below.\n\n#### -o,--output \\\u003cOUTPUT\\\u003e\nPath to a directory where the replay files are generated.\n\n#### -p,--prefix \\\u003cPREFIX\\\u003e\nPrefix for the replay file names.\n\n#### -j,--out-json\nDump replay files in JSON format. By default replay files will be formatted as YAML.\n\n### Remap Config to URL List [remap_config_to_url_list.py](tools/remap_config_to_url_list.py)\nThis tool converts a `remap.config` file to a URL list file that is used by [Replay Gen](#replay-gen-replay_genpytoolsreplay_genpy)\nListed below are the available arguments for this script.\n\n#### -o,--output \\\u003cOUTPUT_FILE\\\u003e\nA filename to which to write the list of URLs. Defaults to `stdout`.\n\n#### --no-ip\nIgnore ip address (in the \"replacement\" section) in the `remap.config` file.\n\n## Contribute\n\nPlease refer to [CONTRIBUTING](CONTRIBUTING.md) for information about how to get involved. We welcome issues, questions, and pull requests.\n\n## License\n\nThis project is licensed under the terms of the Apache 2.0 open source license.\nPlease refer to [LICENSE](LICENSE) for the full terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyahoo%2Fproxy-verifier","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyahoo%2Fproxy-verifier","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyahoo%2Fproxy-verifier/lists"}