{"id":18897344,"url":"https://github.com/tempesta-tech/tempesta-test","last_synced_at":"2025-04-15T02:09:15.297Z","repository":{"id":37539562,"uuid":"127786214","full_name":"tempesta-tech/tempesta-test","owner":"tempesta-tech","description":"Test suite for Tempesta FW","archived":false,"fork":false,"pushed_at":"2025-04-14T14:08:10.000Z","size":2981,"stargazers_count":12,"open_issues_count":124,"forks_count":6,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-15T02:08:59.504Z","etag":null,"topics":["automated-tests","continuous-integration","stress-testing"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tempesta-tech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-04-02T17:02:17.000Z","updated_at":"2025-04-14T11:18:47.000Z","dependencies_parsed_at":"2024-02-26T12:25:05.533Z","dependency_job_id":"281c6c7b-7545-4a7f-a815-400b19af386a","html_url":"https://github.com/tempesta-tech/tempesta-test","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tempesta-tech%2Ftempesta-test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tempesta-tech%2Ftempesta-test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tempesta-tech%2Ftempesta-test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tempesta-tech%2Ftempesta-test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tempesta-tech","download_url":"https://codeload.github.com/tempesta-tech/tempesta-test/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248991544,"owners_count":21194894,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automated-tests","continuous-integration","stress-testing"],"created_at":"2024-11-08T08:37:02.562Z","updated_at":"2025-04-15T02:09:15.285Z","avatar_url":"https://github.com/tempesta-tech.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Functional Tests for TempestaFW\n\n## Recommended configuration\n\nRunning tests during development process can cause crashes to TempestaFW.\nSince TempestaFW is implemented as a set of kernel modules it is not convenient\nto run testing framework on the same host. It is recommended to run TempestaFW \non a separated host. Please use a separate computer or a virtual machine (containerization is not suitable).\n\nRecommended test-beds:\n\n- Local testing. All parts of the testing framework are running on the same\nhost. The simplest configuration to check that current revision of TempestaFW\npasses all the functional tests. It is default configuration.\n```\n    ┌──────────────────────────────────────────────────────┐\n    │ Testing Framework + Client + TempestaFW + Web Server │\n    └──────────────────────────────────────────────────────┘\n```\n\n- With isolated testing framework. 2 different hosts with their own roles are used.\nThis configuration generates real network traffic. Handy for stress and performance \ntesting but require a lot of resources.\nThis configuration is recommended for TempestaFW developers.\n```\n    ┌───────────────────┐\n    │    TempestaFW     ├────┐\n    └──────┬────────────┘    │ Management over SSH\n           │              ┌──┴──────────────────────────────────────┐\n           │              │ Testing Framework + Client + Web Server │\n           │              └───────────────┬─────────────────────────┘\n           └──────────────────────────────┘\n          Separated network for test traffic\n```\n\n## Setup\n\nTo run requirements auto installation run setup.py from `tempesta-test` directopy. Some operations must be performed with root privileges. At the start, the script will ask for the sudo user’s password.\n\n```sh\npython3 setup.py\n```\n\n## Requirements\n\n- Ubuntu 24.04 \n- Python version 3.10. Ubuntu 24.04 uses Python 3.12. Please install Python 3.10 and use a virtual environment (also we use [asyncore](https://docs.python.org/3.11/library/asyncore.html) that was removed in 3.12)\n```sh\nadd-apt-repository ppa:deadsnakes/ppa\napt update\napt install python3.10\napt install python3.10-venv\n```\n- ClickHouse Database 25 (Optional, used for storing logs in the database). See installation [here](https://clickhouse.com/docs/en/install#quick-install)\n- Host for testing framework: `python3`, `wrk`, `ab`, `nghttp2`, `h2spec`, \n`curl`, `h2load`, `tls-perf`, `netstat`, `lxc`, `nginx`, `docker.io`, web content \ndirectory accessible by nginx, nginx should not be running before the tests start.\nSee Python libraries in `requirements.txt`\n- All hosts except previous one: `sftp-server`\n- Host for running TempestaFW: Linux kernel with Tempesta, TempestaFW sources,\n  `systemtap`, `tcpdump`, `bc`\n\n`wrk` is an HTTP benchmarking tool, available from [Github](https://github.com/wg/wrk).\n\n`ab` is Apache benchmark tool, that can be found in `apache2-utils` package.\n\n`h2spec` is HTTP/2 conformance test suite. Can't be installed from package\nmanager and must be [compiled from sources](https://github.com/tempesta-tech/h2spec#build).\n\n`tls-perf` can be downloaded from our GitHub [repository](https://github.com/tempesta-tech/tls-perf).\n\nLinux kernel for TempestaFW recommended to install using [kernel_installer.py](https://github.com/tempesta-tech/tempesta-ci/tree/master/scripts)\n\nTesting framework manages other hosts via SSH protocol, so the host running\ntesting framework must be able to be authenticated on other hosts by the key.\nThat can be done using `ssh-copy-id`. Root access is required on all hosts.\n\nRequirements can be checked with `check_deps/check_dependencies.sh`. It should\nbe ran from `check_deps` directory.\n\n## Run tests\n\n### Configuration\n\nTesting framework is configured via `tests_config.ini` file. Example\nconfiguration is described in `tests_config.ini.sample` file.\nYou can also create default tests configuration \n(see `TestFrameworkCfg.defaults` method from `helpers/tf_cfg.py`) by calling:\n\n```sh\nenv/bin/python3 run_tests.py -d local\n```\n\nThere is 5 sections in configuration: `General`, `Client`, `Tempesta`, `Server`, `TFW_Logger`.\n\n### Run tests\nThe tests work with Ubuntu settings, please use the root user directly.\n\nIt's important that all tests are run from the Python 3.10 virtual environment. If the tests are executed from the tempesta-test folder, the easiest way is:\n```sh\nenv/bin/python3 run_test.py\n```\n\nTo run all the tests simply run:\n```sh\nenv/bin/python3 run_tests.py\n```\n\nTo run individual tests, name them in the arguments to the `run_tests.py` script\nin dot-separated format (as if you were importing them as python modules,\nalthough it is also possible to run specific testcases or even methods inside a\ntestcase):\n```sh\nenv/bin/python3 run_tests.py cache.test_cache\nenv/bin/python3 run_tests.py cache.test_cache.TestCacheDisabled.test_cache_fullfill_all\n```\n\nOr you can run all tests from a file:\n```sh\nenv/bin/python3 run_tests.py selftests/test_deproxy.py \n```\n\nOr you can run individual tests (or test class) using `-H` options:\n\n```sh\nenv/bin/python3 run_tests.py -H selftests/test_deproxy.py \n[?] Select test class: DeproxyTestH2\n   DeproxyChunkedTest\n   DeproxyClientTest\n   DeproxyDummyTest\n   DeproxyTest\n   DeproxyTestFailOver\n \u003e DeproxyTestH2\n   ProtocolError\n\n[?] Select test method: ALL\n \u003e ALL\n   test_bodyless\n   test_bodyless_multiplexed\n   test_disable_huffman_encoding\n   test_get_4xx_response\n   test_make_request\n   test_no_parsing_make_request\n   test_parsing_make_request\n   test_with_body\n```\n\nTo ignore specific tests, specify them in the arguments prefixed with `-`\n(you may need to use `--` to avoid treating that as a flag):\n```sh\nenv/bin/python3 run_tests.py cache -cache.test_purge # run cache.*, except cache.test_purge.*\nenv/bin/python3 run_tests.py -- -cache # run everything, except cache.*\n```\n\nIf the testsuite was interrupted or aborted, next run will continue from the\ninterruption point. The resumption information is stored in the\n`tests_resume.txt` file in the current working directory. It is also possible\nto resume the testsuite from a specific test:\n```sh\nenv/bin/python3 run_tests.py --resume flacky_net\nenv/bin/python3 run_tests.py --resume-after cache.test_purge\n```\n\nIn all cases, prefix specifications are allowed, i. e. `cache.test_cache` will\nmatch all tests in `cache/test_cache.py`, but `test_cache` will not match\nanything. When resuming, execution will continue from (after) the first test\nthat matches the specified string.\n\n## Adding new tests\n\n**WARNING**: there are 2 testing frameworks in directories `testers` and `framework`.\nPlease use only `framework.testet.TempestaTest` for the new tests. \n`testers.functional.FunctionalTest` and `testers.stress.StressTest` are deprecated and \nmust be removed in https://github.com/tempesta-tech/tempesta-test/issues/56 .\n\n### Requirements to adding new tests:\n1. Name of the test directory must be started with `t_` prefix;\n2. Name of the file must be started with `test_` prefix;\n\n```sh\nmkdir t_new_directory\ntouch t_new_directory/test_some_feature.py\necho \"__all__ = [ 'test_some_feature' ]\" \u003e\u003e my_test/__init.py__\n```\n\n3. Name of the test class must be started with `Test` prefix;\n\n```python3\nfrom test_suite import tester\n\nclass TestCases(tester.TempestaTest):\n    ...\n```\n\n4. Test class must contain TempestaFW, backend and client configuration. Each config is \na structure, containing item id, type, and other options, depending on item type. \nTempestaFW config, deproxy response, addr and port can use templates in format \n`${part_variable}` where `part` is one of 'server', 'tempesta', 'client' or 'backend'. \nFor example:\n\n```python3\nbackends = [\n    {\n        \"id\": \"deproxy\",\n        \"type\": \"deproxy\",\n        \"port\": \"8000\",\n        \"response\": \"static\",\n    },\n    {\n        \"id\": \"nginx\",\n        \"type\": \"nginx\",\n        \"status_uri\": \"http://${server_ip}:8000/nginx_status\",\n        \"config\": \"nginx config as string\",\n    }\n]\n\nclients = [\n    {\n        \"id\": \"deproxy\",\n        \"type\": \"deproxy_h2\",  # \"deproxy\" for HTTP/1\n        \"addr\": \"${tempesta_ip}\",\n        \"port\": \"443\",\n        \"ssl\": True,\n    },\n    {\"id\": \"wrk\", \"type\": \"wrk\", \"addr\": \"${server_ip}:8000\"},\n    {\n        \"id\": \"external\",\n        \"type\": \"external\",\n        \"binary\": \"curl\",\n        \"cmd_args\": \"-Ikf http://${tempesta_ip}:80/\",\n    },\n    {\"id\": \"curl\", \"type\": \"curl\", \"h2\": True},\n]\n\ntempesta = {\n    \"config\": \"\"\"\n        listen 80;\n        server ${server_ip}:8000;\n    \"\"\"\n}\n```\n\n5. We use decorators to parameterize the tests (please don't use inheritance):\n\n```python3\nfrom test_suite import tester\nfrom test_suite import marks\n\n\n@marks.parameterize_class(\n  [\n    {\"name\": \"Http\", \"clients\": [\"http_config\"]},\n    {\"name\": \"H2\", \"clients\": [\"h2_config\"]},\n  ]\n)\nclass TestExample(tester.TempestaTest):\n  @marks.Parameterize.expand(\n    [\n      marks.Param(name='1', key_1=\"value_1\"),\n      marks.Param(name='2', key_1=\"value_2\"),\n    ]\n  )\n  def test_request(self, name, key_1):\n    ...\n\n# we will get 4 tests:\n# TestExampleHttp.test_request_1\n# TestExampleHttp.test_request_2\n# TestExampleH2.test_request_1\n# TestExampleH2.test_request_2\n```\n\nExample tests can be found in `selftests/test_framework.py`\n\nTests can be skipped or marked as expected to fail.\nMore info at [Python documentation](https://docs.python.org/3/library/unittest.html).\n\n### Testing with chunked messages\n\nWARNING: only for deproxy client and server, and it only works on local configuration.\n\nSome tests require division of request or response into small TCP segments (\"chunks\").\nThis division is controlled by `segment_size` parameter of the client or the backend\n(see above). Usualy better to set this parameter programmaticaly rather than in client\nor backend configuration. You can run any test with TCP segmentation using `-T` option:\n\n```shell\nenv/bin/python3 run_tests.py -T 10 selftests/test_deproxy.py \n```\n\n## Internal structure and motivation of user configured tests\n\nUser configured tests have very flexible structure. They allow arbitrary\nclients and server start, stop, making requests. This leads to several\npoints in internal structure.\n\n### Using separate thread for polling cycle\n\nNow, deproxy client and deproxy server, all of them use the single polling\ncycle. We have 3 cases of using deproxy clients and server:\n\n1) both deproxy client and server are used\n\n2) only deproxy client is used\n\n3) only deproxy server is used\n\nAnd case 3 leads to instant running of polling cycle in separate thread.\nIndeed, let's consider case of wrk client and deproxy server without instant\nrunning of this cycle. We start deproxy server, then we start wrk client\nand after this we start polling cycle. The time before starting cycle,\nwrk will get an errors. Ok, let's start polling cycle before wrk. But now it's\nimpossible to start wrk, because we are in polling cycle. This problem appeares,\nbecause with running polling cycle in the same thread, as the main procedure,\ndeproxy server can receive requests only after polling cycle starts.\n\nThe solution is to make possible handling requests exactly when server starts.\nIn this case test procedure becames simple and straightforward: start deproxy\nserver, the start wrk. And this became possible with polling cycle, running in\nseparate thread.\n\nBut using separate thread leads to requirements of using locks. It's appeared\nthat creating new connection while polling function is running in it's thread,\ncan lead to error. So we should be sure, that it won't happen. That's why locks\nare used.\n\n          Main thread                  Thread with poll loop\n\n            |                                   |\n            | ------------------------------- Lock()\n            |                                   |\n            |                                 Poll()\n            |                                   |\n            | ------------------------------ Unlock()\n            |                                   |\n            .                                   .\n            .                                   .\n            .                                   .\n            |                                   |\n            |                                 Lock()\n    client or server                            |\n         start()                               Poll()\n            \\                                   |\n             \\                               Unlock()\n            Lock() ---------------------------- |\n              |                                 |\n        create socket                           |\n              |                                 |\n        connect or listen                       |\n              |                                 |\n          Unlock() ---------------------------- |\n              |                               Lock()\n            return                              |\n             /                                Poll()\n            |                                   |\n            |                                Unlock()\n            |                                   |\n            .                                   .\n            .                                   .\n            .                                   .\n\n### Classes used\n\nCode of configurable tests located in `framework/` directory. It contains\nbasic class for configurable test and classes for items. Also it contains\nclass for deproxy management and polling cycle.\n\n#### TempestaTest\n\nBasic class for user configured tests. Contains parsing of used items\ndeclaration (clients, backends, tempesta), startup and teardown functions.\nUser configured tests should inherit it.\n\nAlso there are cases when you most likely would want to create a basic abstract\nclass for a group of tests and utilize Python's inheritance mechanism. In order\nto do that, just pass an argument `base` upon class creation, e.g.:\n\n```python\nclass HttpTablesTestBase(tester.TempestaTest, base=True):\n    ...\n\n```\nand tests won't be called for that particular class.\n\nDefault `base` value is `False`.\n\nNote that if you override `setUp` method, please, don't forget to put\n```python\n...\nsuper().setUp()\n...\n```\nin there, otherwise this feature won't work properly.\n\n#### DeproxyManager\n\nThis class is a stateful wrap for the `run_deproxy_server()` function.\nThis function contains a polling cycle. DeproxyManager creates new thread for\nthis function, and stops it, when received `stop()`. DeproxyManager starts\nin test `setUp()` and stops in `tearDown()`. So, polling cycle run all the\ntest time.\n\n#### FreePortsChecker\n\nWhen we start backend, it can appear, that specified port is already used\nby smth. So server startup will fail. We can make all servers to write about\nthis, but it simpler to check free ports before start server.\n\n#### Classes for servers and clients\n\ndeproxyclient, deproxyserver, nginx, wrk - this classes used for creating\nand handling corresponding types of items.\n\n\n## Development\n\nIn the project we use [![wemake-python-styleguide](https://img.shields.io/badge/style-wemake-000000.svg)](https://github.com/wemake-services/wemake-python-styleguide)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat\u0026labelColor=ef8336)](https://pycqa.github.io/isort/)\n\nThis project uses solutions from [MHDDoS](https://github.com/MatrixTM/MHDDoS).\n\nInstall dependencies: `pip3 install -r requirements.txt`\n\nThere may be a possible problem related to `scapy`, `paramiko`, and `cryptography`. \n`paramiko` will install `cryptography==43.0.3`, but `scapy~=2.5.0rc2` cannot work\nwith such a version of `cryptography` producing `ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.ec'`\nWe added `cryptography==38.0.2` on the top of `requirements.txt` to install it first.\n\nIf you still encounter a mentioned exception, try to run the next commands: \n```bash\npip uninstall -y cryptography\npip install cryptography==38.0.2\n```\n\n\nRun `pre-commit install` to set up the git hook script.\n\nRun `pre-commit autoupdate` for update to the latest repos' versions (optional).\n\nConfiguration files: wemake - `tox.ini`, black and isort - `pyproject.toml`, pre-commit - `.pre-commit-config.yaml`.\n\nRun formatters `isort \u003csource_file_or_directory\u003e` and `black \u003csource_file_or_directory\u003e`\n\nRun linter `flake8 \u003ctarget\u003e`:\n\n  where `tagret` is optional parameter, it defines target file to be checked,\n  if omitted, checks is going to be processed on all files in running directory.\n\nUse `git commit -v --all` to format all changed python files or just use `git commit -m \u003cmsg\u003e`.\n\n## Resources\n\nThere are not so much good references about best practices in development of\ntesting framework.\n\n* [Why Good Developers Write Bad Tests](https://www.youtube.com/watch?v=hM_ex4-xu4E)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftempesta-tech%2Ftempesta-test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftempesta-tech%2Ftempesta-test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftempesta-tech%2Ftempesta-test/lists"}