{"id":28533980,"url":"https://github.com/vectordotdev/vector-test-harness","last_synced_at":"2025-09-02T03:35:18.854Z","repository":{"id":46676008,"uuid":"179370871","full_name":"vectordotdev/vector-test-harness","owner":"vectordotdev","description":"End-to-end test harness for the Vector observability data router ","archived":true,"fork":false,"pushed_at":"2022-01-05T11:08:54.000Z","size":1124,"stargazers_count":32,"open_issues_count":13,"forks_count":8,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-07-07T21:32:33.411Z","etag":null,"topics":["benchmarking","fluentbit","fluentd","logstash","management","metallica","terraform","test-harness","testing","vector"],"latest_commit_sha":null,"homepage":"https://vector.dev","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vectordotdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-03T21:09:58.000Z","updated_at":"2025-06-11T19:25:11.000Z","dependencies_parsed_at":"2022-09-11T23:41:11.996Z","dependency_job_id":null,"html_url":"https://github.com/vectordotdev/vector-test-harness","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vectordotdev/vector-test-harness","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vectordotdev%2Fvector-test-harness","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vectordotdev%2Fvector-test-harness/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vectordotdev%2Fvector-test-harness/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vectordotdev%2Fvector-test-harness/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vectordotdev","download_url":"https://codeload.github.com/vectordotdev/vector-test-harness/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vectordotdev%2Fvector-test-harness/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273225502,"owners_count":25067323,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","fluentbit","fluentd","logstash","management","metallica","terraform","test-harness","testing","vector"],"created_at":"2025-06-09T17:09:25.395Z","updated_at":"2025-09-02T03:35:18.837Z","avatar_url":"https://github.com/vectordotdev.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./README/header.svg\" alt=\"Vector Test Harness\"\u003e\n\u003c/p\u003e\n\u003cbr /\u003e\n\nFull end-to-end test harness for the [Vector][vector] log \u0026 metrics router. This is the\ntest framework used to generate the [performance] and [correctness] results displayed in the\n[Vector docs][vector_docs]. You can learn more about how this test harness works in the\n[How It Works][how_it_works] section, and you can begin using this test harness via the\n[Usage][usage] section.\n\n---\n\n***Contributions for additional benchmarks and tools are welcome! As required by the\n[MPL 2.0 License](LICENSE), changes to this code base, including additional benchmarks\nand tools, must be made in the open. Please be skeptical of tools making performance\nclaims without doing so in the public. The purpose of this repository is to create\ntransparency around benchmarks and the resulting performance.***\n\n### TOC\n\n* [Results][results]\n* [Directories][directories]\n* [Setup][setup]\n* [Usage][usage]\n* [Development][development]\n* [Debugging][debugging]\n* [How It Works][how_it_works]\n\n### Performance Tests\n\n* [**`disk_buffer_performance` test**](cases/disk_buffer_performance)\n* [**`file_to_tcp_performance` test**](cases/file_to_tcp_performance)\n* [**`tcp_to_blackhole_performance` test**](cases/tcp_to_blackhole_performance)\n* [**`tcp_to_tcp_performance` test**](cases/tcp_to_tcp_performance)\n* [**`tcp_to_http_performance` test**](cases/tcp_to_http_performance)\n* [**`regex_parsing_performance` test**](cases/regex_parsing_performance)\n\n### Correctness Tests\n\n* [**`disk_buffer_persistence_correctness` test**](cases/disk_buffer_persistence_correctness)\n* [**`file_rotate_create_correctness` test**](cases/file_rotate_create_correctness)\n* [**`file_rotate_truncate_correctness` test**](cases/file_rotate_truncate_correctness)\n* [**`file_truncate_correctness` test**](cases/file_truncate_correctness)\n* [**`sighup_correctness` test**](cases/sighup_correctness)\n* [**`wrapped_json_correctness` test**](cases/wrapped_json_correctness)\n\n---\n\n## Directories\n\n* [`/ansible`](/ansible) - global ansible resources and tasks\n* [`/bin`](/bin) - contains all scripts\n* [`/cases`][cases] - contains all test cases\n* [`/packer`](/packer) - packer script to build the AMIs necessart for tests\n* [`/terraform`](/terraform) - global terraform state, resources, and modules\n\n## Setup\n\n1. Ensure you have [Ansible][ansible] (2.7+) and [Terraform][terraform] (0.12.20+) installed.\n2. This step is optional, but highly recommended. Setup a [`vector`][vector] specific\n   [AWS profile][aws_profile] in your `~/.aws/credentials` file. We highly recommend running the\n   Vector test harness in a separate AWS sandbox account if possible.\n3. [Create an Amazon compatible key pair][create_keys]. This will be used for SSH access to test\n   instances.\n4. Run `cp .envrc.example .envrc`. Read through the file, update as necessary.\n5. Run `source .envrc` to prepare the environment. Alternatively install [direnv] to do this\n   automatically.\n   Note that the `.env` file, if it exists, will be automatically sourced into\n   the scripts environment - so it's another option to set the environment\n   variables for the `bin/*` commands of this repo.\n6. Run:\n\n   ```bash\n   ./bin/test -t [tcp_to_tcp_performance]\n   ```\n\n   This script will take care of running the necessary Terraform and Ansible scripts.\n\n## Usage\n\n* [`bin/build-amis`][build-amis] - builds AMIs for use in test cases\n* [`bin/compare`][compare] - compare of test results across all subjects\n* [`bin/ssh`][test] - utility script to SSH into a test server\n* [`bin/test`][test] - run a specific test\n\n## Results\n\n* High-level results can be found in the Vector [performance] and [correctness] documentation\n  sections.\n* Detailed results can be found within each [test case's][cases] README.\n* Raw performance result data can be found in our public [S3 bucket][s3_bucket].\n* You can run your own queries against the raw data. See the [Usage][usage] section.\n\n## Development\n\n### Adding a test\n\nWe recommend cloning a similar to test since it removes a lot of the boilerplate. If you prefer\nto start from scratch:\n\n1. Create a new folder in the [`/cases`][cases] directory. Your name should end with\n   `_performance` or `_correctness` to clarify the type of test this is.\n2. Add a `README.md` providing an overview of the test. See the\n   [`tcp_to_tcp_performance`][tcp_to_tcp_performance] test for an example.\n3. Add a `terraform/main.tf` file for provisioning test resources.\n4. Add a `ansible/bootstrap.yml` to bootstrap the environment.\n5. Add a `ansible/run.yml` to run the test againt each subject.\n6. Add any additional files as you see fit for each test.\n7. Run `bin/test -t \u003cname_of_test\u003e`.\n\n### Changing a test\n\nYou should _not_ be changing tests with historical test data. You can change test subject versions\nsince test data is [partitioned by version][data_location], but you _cannot_ change a test's\nexecution strategy as this would corrupt historical test data. If you need to change the test\nin such a way that would violate historical data we recommend creating an entirely new test.\n\n### Deleting a test\n\nSimply delete the folder and any data in the [s3 bucket][s3_bucket].\n\n## Debugging\n\n### On a VM end\n\nIf you encounter an error it's likely you'll need to SSH onto the server to\ninvestigate.\n\n#### SSHing\n\n```bash\nssh  -o 'IdentityFile=\"~/.ssh/vector_management\"' ubuntu@51.5.210.84\n```\n\nWhere:\n\n* `~/.ssh/vector_management` = the `VECTOR_TEST_SSH_PRIVATE_KEY` value provided in your `.envrc` file.\n* `ubuntu` = the default root username for the instance.\n* `51.5.210.84` = the _public_ IP address of the instance.\n\nWe provide a command that wraps the system `ssh` and provides the same\ncredentials that ansible uses when connecting to the VM:\n\n```bash\n./bin/ssh 51.5.210.84\n```\n\n#### Viewing logs\n\nAll services are configured with systemd where their logs can be accessed with\n`journalctl`:\n\n```shell\nsudo journactl -fu \u003cservice\u003e\n```\n\n#### Failed services\n\nIf you find that the service failed to start, I find it helpful to manually\nattempt to start the service by inspecting the command in the `.service` file:\n\n```shell\ncat /etc/systemd/system/\u003cname\u003e.service\n```\n\nThen copy the command specified in `ExecStart` and run it manually. Ex:\n\n```shell\n/usr/bin/vector\n```\n\n### On your end\n\nThings can go wrong on your end (i.e. on the local system you're running the\ntest harness) too.\n\n#### Ansible Task Debugger\n\n```bash\nexport ANSIBLE_ENABLE_TASK_DEBUGGER=True\n```\n\nSet the environment variable above, and Ansible will drop you in a debug mode on any task failure.\n\nSee Ansible documentation on [Playbook Debugger](https://docs.ansible.com/ansible/latest/user_guide/playbooks_debugger.html) to learn more.\n\nSome useful commands:\n\n```python\npprint task_vars['hostvars'][str(host)]['last_message']\n```\n\n#### Verbose Ansible Execution\n\n```bash\nexport ANSIBLE_EXTRA_ARGS=-vvv\n```\n\nSet the environment variable above, and Ansible will print verbose debug\ninformation for every task it executes.\n\n## How It Works\n\n### Design\n\nThe Vector test harness is a mix of [bash][bin], [Terraform][terraform], and [Ansible][ansible]\nscripts. Each test case lives in the [`/cases`][cases] directory and has full reign of it's\nbootstrap and test process via it's own [Terraform][terraform] and [Ansible][ansible] scripts.\nThe location of these scripts is dictated by the [`test`][test] script and is outlined in more\ndetail in the [Adding a test][adding_a_test] section. Each test falls into one of 2 categories:\nperformance tests and correctness tests:\n\n#### Performance tests\n\nPerformance tests measure performance and MUST capture detailed performance data as outlined\nin the [Performance Data][performance_data] and [Rules][rules] sections.\n\nIn addition to the `test` script, there is a [`compare`][compare] scripts.\nThis script analyzes the performance data captured when executing a test. More information\non this data and how it's captured and analyzed can be found in the\n[Performance Data][performance_data] section. Finally, each [script][bin] includes a usage\noverview that you can access with the `--help` flag.\n\n##### Performance data\n\nPerformance test data is captured via [`dstat`][dstat], which is a lightweight utility that\ncaptures a variety of system statistics in 1-second snapshot intervals. The final result is a CSV\nwhere each row represents a snapshot. You can see the [`dstat` command][dstat_command] used in the\n[`ansible/roles/profiling/start.yml`][profiling_start] file.\n\n###### Performance data schema\n\nThe performance data schema is reflected in the [Athena table definition][athena_table] as\nwell as the CSV itself. The following is an ordered list of columns:\n\n| Name            |   Type    |\n|:----------------|:---------:|\n| `epoch`         | `double`  |\n| `cpu_usr`       | `double`  |\n| `cpu_sys`       | `double`  |\n| `cpu_idl`       | `double`  |\n| `cpu_wai`       | `double`  |\n| `cpu_hiq`       | `double`  |\n| `cpu_siq`       | `double`  |\n| `disk_read`     | `double`  |\n| `disk_writ`     | `double`  |\n| `io_read`       | `double`  |\n| `io_writ`       | `double`  |\n| `load_avg_1m`   | `double`  |\n| `load_avg_5m`   | `double`  |\n| `load_avg_15m`  | `double`  |\n| `mem_used`      | `double`  |\n| `mem_buff`      | `double`  |\n| `mem_cach`      | `double`  |\n| `mem_free`      | `double`  |\n| `net_recv`      | `double`  |\n| `net_send`      | `double`  |\n| `procs_run`     | `double`  |\n| `procs_bulk`    | `double`  |\n| `procs_new`     | `double`  |\n| `procs_total`   | `double`  |\n| `sys_init`      | `double`  |\n| `sys_csw`       | `double`  |\n| `sock_total`    | `double`  |\n| `sock_tcp`      | `double`  |\n| `sock_udp`      | `double`  |\n| `sock_raw`      | `double`  |\n| `sock_frg`      | `double`  |\n| `tcp_lis`       | `double`  |\n| `tcp_act`       | `double`  |\n| `tcp_syn`       | `double`  |\n| `tcp_tim`       | `double`  |\n| `tcp_clo`       | `double`  |\n\n###### Performance data location\n\nAll performance data is made public via the [`vector-tests` S3 bucket][s3_bucket] in the\n`us-east-1` region. The partitioning structure follows the Hive partitioning structure with\nvariable names in the path. For example:\n\n```\nname=tcp_to_tcp_performance/configuration=default/subject=vector/version=v0.2.0-dev.1-20-gae8eba2/timestamp=1559073720\n```\n\nAnd the same in a tree form:\n\n```\nname=tcp_to_tcp_performance/\n  configuration=default/\n    subject=vector/\n      version=v0.2.0-dev.1-20-gae8eba2/\n        timestamp=1559073720\n```\n\n* `name` = the test name.\n* `configuration` = refers to the test's specific configuration (tests can have multiple configurations\n  if necessary).\n* `subject` = the test subject, such as `vector`.\n* `version` = the version fo the test subject.\n* `timestamp` = when the test was executed.\n\n###### Performance data analysis\n\nAnalysis of this data is performed through the [AWS Athena service][aws_athena]. This allows us\nto execute complex queries on the [performance data][performance_data] stored in S3. You can see\nthe queries ran in the [`compare`][compare] script.\n\n#### Correctness tests\n\nCorrectness tests simply verify behavior. These tests are not required to capture or to persist\nany data. The results can be manually verified and placed in the test's README.\n\n##### Correctness data\n\nSince correctness tests are pass/fail there is no data to capture other than the successful\nrunning of the test.\n\n##### Correctness output\n\nGenerally, correctness tests verify the output. Because of the various test subjects, we use a variety\nof output methods to capture output (tcp, http, and file). This is highly dependent on the test\nsubject and the methods available. For example, the Splunk Forwarders only support TCP and\nSplunk specific outputs.\n\nTo make capturing this data easy, we created a [`test_server` Ansible role][test_servers_role]\nthat spins up various test servers and provides a simple way to capture summary output.\n\n### Environments\n\nTests must operate in isolated reproducible environments, they must _never_ run locally.\nThe obvious benefit is that it removes variables across tests, but it also improves collaboration\nsince remote environments are easily accessible and reproducible by other engineers.\n\n### Rules\n\n1. ALWAYS filter to resources specific to your `test_name`, `test_configuration`, and\n   `user_id` (ex: ansible host targeting)\n2. ALWAYS make sure the initial instance state is identical across test subjects. We recommend explicitly\n   stopping all test subjects to properly handle the case of preceding failure and the situation where a\n   subject was not cleanly shutdown.\n3. ALWAYS use the [`profile` ansible role][profiling_role] to capture data. This ensures a consistent\n   data structure across tests.\n4. ALWAYS run performance tests for at least 1 minute to calculate a 1m CPU load average.\n5. Use [ansible roles][roles] whenever possible.\n6. If you are not testing local data collection we recommend using TCP as a data source since\n   it is a lightweight source that is more likely to be consistent, performance wise, across\n   subjects.\n\n\n[adding_a_test]: #adding-a-test\n[ansible]: https://www.ansible.com/\n[athena_table]: /terraform/data.tf\n[aws_athena]: https://aws.amazon.com/athena/\n[aws_cli]: https://aws.amazon.com/cli/\n[aws_profile]: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html\n[bin]: /bin\n[build-amis]: bin/build-amis\n[cases]: /cases\n[compare]: bin/compare\n[correctness]: https://vector.dev/#correctness\n[create_keys]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html\n[data_location]: #performance-data-location\n[debugging]: #debugging\n[development]: #development\n[directories]: #directories\n[direnv]: https://direnv.net/\n[dstat]: http://dag.wiee.rs/home-made/dstat/\n[dstat_command]: /ansible/roles/profiling/tasks/start.yml\n[how_it_works]: #how-it-works\n[performance]: https://vector.dev/#performance\n[performance_data]: #performance-data\n[profiling_role]: /ansible/roles/profiling\n[profiling_start]: /ansible/roles/profiling/tasks/start.yml\n[results]: #results\n[roles]: /ansible/roles\n[rules]: #rules\n[s3_bucket]: https://s3.console.aws.amazon.com/s3/buckets/vector-tests/?region=us-east-1\n[setup]: #setup\n[test_servers_role]: /ansible/roles/test_servers\n[tcp_to_tcp_performance]: cases/tcp_to_tcp_performance\n[terraform]: https://www.terraform.io/\n[test]: bin/test\n[usage]: #usage\n[vector]: https://github.com/timberio/vector\n[vector_docs]: https://docs.vector.dev/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvectordotdev%2Fvector-test-harness","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvectordotdev%2Fvector-test-harness","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvectordotdev%2Fvector-test-harness/lists"}