{"id":20564323,"url":"https://github.com/tarantool/test-run","last_synced_at":"2025-04-14T15:10:46.138Z","repository":{"id":30833910,"uuid":"34391331","full_name":"tarantool/test-run","owner":"tarantool","description":"Tarantool functional testing framework","archived":false,"fork":false,"pushed_at":"2024-12-26T17:26:23.000Z","size":985,"stargazers_count":13,"open_issues_count":77,"forks_count":17,"subscribers_count":30,"default_branch":"master","last_synced_at":"2025-03-28T04:02:38.404Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tarantool.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-04-22T13:17:21.000Z","updated_at":"2025-02-01T18:54:14.000Z","dependencies_parsed_at":"2024-01-19T14:25:24.422Z","dependency_job_id":"6f341b97-667d-4c50-bd16-25eaaab2e2d8","html_url":"https://github.com/tarantool/test-run","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tarantool%2Ftest-run","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tarantool%2Ftest-run/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tarantool%2Ftest-run/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tarantool%2Ftest-run/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tarantool","download_url":"https://codeload.github.com/tarantool/test-run/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248904640,"owners_count":21180835,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T04:25:39.960Z","updated_at":"2025-04-14T15:10:46.118Z","avatar_url":"https://github.com/tarantool.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tarantool Functional testing framework\n\n[![Coverage Status](https://coveralls.io/repos/github/tarantool/test-run/badge.svg)](https://coveralls.io/github/tarantool/test-run)\n\n### Test Suite\n\nBunch of tests, that lay down in the subfolder (recursively) with `suite.ini`\nfile. `suite.ini` is basic ini-file, that consists of one section `default`,\nand a number of fields:\n\n* `core`\n* `description` - Test Suite description\n* `script` - shebang file to start tarantool with\n* disables:\n    * `disabled` - tests that must be skipped\n    * `release_disabled` - tests that must be skipped when Tarantool has been\n      builded with `Release`\n    * `valgrind_disabled` - tests that must be skipped when Valgrind is enabled\n* `lua_libs` - paths for lua files, that should be copied into the folder,\n  where server is started (delimited with the space, e.g. `lua_libs=lua/1.lua\n  lua/2.lua`)\n* `long_run` - mark tests as long, enabled only with `--long` option (delimited\n  with the space, e.g. `long_run=t1.test.lua t2.test.lua`)\n* `config` - test configuration file name\n\nField `core` must be one of:\n\n* `luatest` - [luatest][luatest] compatible test suite\n* `tarantool` - Test-Suite for Functional Testing\n* `app` - Another functional Test-Suite\n* `unittest` - Unit-Testing Test Suite\n\n### Test\n\nEach test consists of files `*.test(.lua|.sql|.py)?`, `*.result`, and may have\nskip condition file `*.skipcond`.  On first run (without `.result`) `.result`\nis generated from output.  Each run, in the beggining, `.skipcond` file is\nexecuted. In the local env there's object `self`, that's `Test` object. If test\nmust be skipped - you must put `self.skip = 1` in this file. Next,\n`.test(.lua|.py)?` is executed and file `.reject` is created, then `.reject` is\ncompared with `.result`. If something differs, then 15 last string of this diff\nfile are printed and `.reject` file is saving in the `\u003cvardir\u003e/rejects/\u003csuite\u003e`\nsubfolder given in options or set localy as `var/rejects/\u003csuite\u003e` by default.\nIf not, then `.reject` file is deleted.\n\n### Test configuration\n\nTest configuration file contains config for multiple run. For each test section\nsystem runs separated test and compares result with common `.result` file. For\nexample we need to run one test for different db engines(\"*\" means default\nconfiguration):\n\n```json\n{\n    \"my.test.lua\": {\n        \"first\": {\"a\": 1, \"b\": 2},\n        \"second\": {\"a\": 1, \"b\": 3}\n    },\n    \"*\": {\n        \"memtx\": {\"engine\": \"memtx\"},\n        \"vinyl\": {\"engine\": \"vinyl\"}\n    }\n}\n```\n\nIn test case we can get configuration from inspector:\n\n```lua\nengine = test_run:get_cfg('engine')\n-- first run engine is 'memtx'\n-- second run engine is 'vinyl'\n```\n\n\"engine\" value has a special meaning for *.test.sql files: if it is \"memtx\" or\n\"vinyl\", then the corresponding default engine will be set before executing\ncommands from a test file. An engine is set with the following commands:\n\n```sql\nUPDATE \"_session_settings\" SET \"value\" = 'memtx|vinyl' WHERE \"name\" = 'sql_default_engine'\npragma sql_default_engine='memtx|vinyl'\n```\n\nIf the first fails, then the second will be executed. When both fails, fail the test.\n\n#### Python\n\nFiles: `\u003cname\u003e.test.py`, `\u003cname\u003e.result` and `\u003cname\u003e.skipcond`(optionaly).\n\nEnvironment:\n\n* `sql` - `BoxConnection` class. Convert our subclass of SQL into IProto query\n  and then decode it. Print into `.result` in YAML. Examples:\n    * `sql(\"select * from t\u003cspace\u003e where k\u003ckey\u003e=\u003cstring|number\u003e[ limit \u003cnumber\u003e]\")`\n    * `sql(\"insert into t\u003cspace\u003e values ([\u003cstring|number\u003e [, \u003cstring|number\u003e]*])\")`\n    * `sql(\"delete from t\u003cspace\u003e where k\u003ckey\u003e=\u003cstring|number\u003e\")`\n    * `sql(\"call \u003cproc_name\u003e([string|number]*)\")`\n    * `sql(\"update t\u003cspace\u003e set [k\u003cfield\u003e=\u003cstring|number\u003e [, k\u003cfield\u003e=\u003cstring|number\u003e]*] where k\u003ckey\u003e=\u003cstring|number\u003e\"\")`\n    * `sql(\"ping\")`\n* `admin` - `AdminConnection` - simply send admin query on admin port (LUA),\n  then, receive answer. Examples\n    * `admin('box.info')`\n\n**Example:**\n\n```python\nimport os\nimport time\n\nfrom lib.admin_connection import AdminConnection\nfrom lib.tarantool_server import TarantoolServer\n\nmaster = server\nadmin(\"box.info.lsn\") # equivalent to master.admin(\"box.info.lsn\") and server.admin(...)\nsql(\"select * from t0 where k0=1\")\nreplica = TarantoolServer()\nreplica.script = 'replication/replica.lua'\nreplica.vardir = os.path.join(server.vardir, \"replica\")\nreplica.deploy()\nmaster.admin(\"box.insert(0, 1, 'hello')\")\nprint('sleep_1')\ntime.sleep(0.1)\nprint('sleep_finished')\nprint('sleep_2')\nadmin(\"require('fiber').sleep(0.1)\")\nprint('sleep_finished')\nreplica.admin(\"box.select(0, 0, 1)\")\ncon2 = AdminConnection('localhost', server.admin.port)\ncon2(\"box.info.lsn\")\nreplica.stop()\nreplica.cleanup()\ncon2.disconnect()\n```\n\n**Result:**\n\n```yaml\nbox.info.lsn\n---\n- null\n...\nselect * from t0 where k0=1\n---\n- error:\n    errcode: ER_NO_SUCH_SPACE\n    errmsg: Space '#0' does not exist\n...\nbox.insert(0, 1, 'hello')\n---\n- error: '[string \"return box.insert(0, 1, ''hello'')\"]:1: attempt to call field ''insert''\n    (a nil value)'\n...\nsleep_1\nsleep_finished\nsleep_2\nrequire('fiber').sleep(0.1)\n---\n...\nsleep_finished\nbox.select(0, 0, 1)\n---\n- error: '[string \"return box.select(0, 0, 1)\"]:1: attempt to call field ''select''\n    (a nil value)'\n...\nbox.info.lsn\n---\n- null\n...\n```\n\n#### Lua\n\nFiles: `\u003cname\u003e.test.lua`, `\u003cname\u003e.result` and `\u003cname\u003e.skipcond`(optionaly).\nTests interact only with `AdminConnection`. Supports some preprocessor functions (eg `delimiter`)\n\n**Delimiter example:**\n\n```\nenv = require('test_run')\ntest_run = env.new()\nbox.schema.space.create('temp')\nt1 = box.space.temp\nt1:create_index('primary', { type = 'hash', parts = {1, 'num'}, unique = true})\nt1:insert{0, 1, 'hello'}\ntest_run:cmd(\"setopt delimiter ';'\")\nfunction test()\n    return {1,2,3}\nend;\ntest(\n);\ntest_run:cmd(\"setopt delimiter ''\");\ntest(\n);\ntest\n```\n\n**Delimiter result:**\n\n```\nenv = require('test_run')\ntest_run = env.new()\nbox.schema.space.create('temp')\n---\n- index: []\n  on_replace: 'function: 0x40e4fdf0'\n  temporary: false\n  id: 512\n  engine: memtx\n  enabled: false\n  name: temp\n  field_count: 0\n- created\n...\nt1 = box.space.temp\n---\n...\nt1:create_index('primary', { type = 'hash', parts = {1, 'num'}, unique = true})\n---\n...\nt1:insert{0, 1, 'hello'}\n---\n- [0, 1, 'hello']\n...\ntest_run:cmd(\"setopt delimiter ';'\")\nfunction test()\n    return {1,2,3}\nend;\n---\n...\ntest(\n);\n---\n- - 1\n  - 2\n  - 3\n...\ntest_run:cmd(\"setopt delimiter ''\");\ntest(\n---\n- error: '[string \"test( \"]:1: unexpected symbol near ''\u003ceof\u003e'''\n...\n);\n---\n- error: '[string \"); \"]:1: unexpected symbol near '')'''\n...\ntest\n---\n- 'function: 0x40e533b8'\n...\n```\n\nIt is possible to use backslash at and of a line to carry it.\n\n```lua\nfunction echo(...) \\\n    return ...     \\\nend\n```\n\n#### SQL\n\n*.test.sql files are just SQL statements written line-by-line.\n\nIt is possible to mix SQL and Lua commands using `\\set language lua` and `\\set\nlanguage sql` commands.\n\n##### Interaction with the test environment\n\nIn lua test you can use `test_run` module to interact with the test\nenvironment.\n\n```lua\nenv = require('test_run')\ntest_run = env.new()\ntest_run:cmd(\"\u003ccommand\u003e\")\n```\n\n__Base directives:__\n\n* `setopt delimiter '\u003cdelimiter\u003e'` - Sets delimiter to `\u003cdelimiter\u003e`\\n\n\n__Server directives:__\n\n* `create server \u003cname\u003e with ...` - Create server with name `\u003cname\u003e`, where `...`\n  may be:\n    * `script = '\u003cpath\u003e'` - script to start\n    * `rpl_master = \u003cserver\u003e` - replication master server name\n* `start server \u003cname\u003e` - Run server `\u003cname\u003e`\n* `stop server \u003cname\u003e [with signal=\u003csignal\u003e]` - Stop server `\u003cname\u003e`\n    * `\u003csignal\u003e` is a signal name (with or without 'SIG' prefix, uppercased) or\n      a signal number to use instead of default SIGTERM\n* `cleanup server \u003cname\u003e` - Cleanup (basically after server has been stopped)\n* `restart server \u003cname\u003e` - Restart server `\u003cname\u003e` (you can restart yourself\n  from lua!)\n\n__Connection switch:__\n\n* `switch \u003cname\u003e` - Switch connection to server `\u003cname\u003e` and add test run into\n  global scope\n\n__Connection directives(low level):__\n\n* `create connection \u003cname-con\u003e to \u003cname-serv\u003e` - create connection named\n  `\u003cname-con\u003e` to `\u003cname-serv\u003e` server\n* `drop connection \u003cname\u003e` - Turn connection `\u003cname\u003e` off and delete it\n* `set connection \u003cname\u003e` - Set connection `\u003cname\u003e` to be main, for next commands\n\n__Filter directives:__\n\n* `push filter '\u003cregexp_from\u003e' to '\u003cregexp_to\u003e'` - e.g. `push filter 'listen: .*' to 'listen: \u003curi\u003e'`\n\n__Set variables:__\n\n* `set variables '\u003cvariable_name\u003e' to '\u003cwhere\u003e'` - execute\n  `\u003cvariable_name\u003e = *\u003cwhere\u003e` where *\u003cwhere\u003e is value of where. Where must be\n    * `\u003cserver_name\u003e.admin` - admin port of this server\n    * `\u003cserver_name\u003e.master` - listen port of master of this replica\n    * `\u003cserver_name\u003e.listen` - listen port of this server\n\n__Dev ops features:__\n\nYou can power on any tarantool replicas in a loop.\n\n```lua\ntest_run:cmd('setopt delimiter \";\"')\nfunction join(inspector, n)\n    for i=1,n do\n        local rid = tostring(i)\n        os.execute('mkdir -p tmp')\n        os.execute('cp ../replication/replica.lua ./tmp/replica'..rid..'.lua')\n        os.execute('chmod +x ./tmp/replica'..rid..'.lua')\n        inspector:cmd(\"create server replica\"..rid..\" with rpl_master=default, script='./var/tmp/replica\"..rid..\".lua'\")\n        inspector:cmd(\"start server replica\"..rid)\n    end\nend;\ntest_run:cmd('setopt delimiter \"\"');\n\n-- create 30 replicas for current tarantool\njoin(test_run, 30)\n```\n\n### pretest_clean()\n\nNothing will be done before a Python test and for `core = unittest`\ntest suites.\n\nFor a `core = [app|tarantool]` test suites this function removes tarantool WAL\nand snapshot files before each test.\n\nThe following files will be removed:\n\n* `*.snap`\n* `*.xlog`\n* `*.vylog`\n* `*.inprogress`\n* `[0-9]*/`\n\n### Tags\n\nUsage:\n\n```sh\n./test-run.py --tags foo\n./test-run.py --tags foo,bar app/ app-tap/\n```\n\ntest-run will run only those tests, which have at least one of the\nprovided tags.\n\nShow a list of tags:\n\n```sh\n./test-run.py --tags\n./test-run.py app-tap/ --tags\n```\n\nThe tags metainfo should be placed within a first comment of a test\nfile.\n\nExamples:\n\n* .lua file:\n\n  ```lua\n  #!/usr/bin/tarantool\n\n  -- tags: foo, bar\n  -- tags: one, more\n\n  \u003c...\u003e\n  ```\n\n* .sql file:\n\n  ```sql\n  -- tags: foo\n  -- tags: bar\n  \u003c...\u003e\n  ```\n\n* .py file:\n\n  ```python\n  # tags: foo\n\n  \u003c...\u003e\n  ```\n\nUnsupported features:\n\n* Marking unit tests with tags.\n* Multiline comments (use singleline ones for now).\n\n### Using luatest\n\ntest-run supports tests written in the [luatest][luatest] format. `*_test.lua`\nfiles in a `core = luatest` test suite are run as part of `./test/test-run.py`\ninvocation: no extra actions are needed.\n\nYou can also run a particular test using a substring of its full name:\n\n```shell\n$ ./test/test-run.py foo-luatest/bar_test.lua\n$ ./test/test-run.py bar_test.lua\n$ ./test/test-run.py bar\n```\n\nIf you need to run a particular test case from a luatest compatible test, use\n`luatest` command directly. In order to use luatest, which is bundled into\ntest-run, source test-run's environment:\n\n```shell\n$ . \u003c(./test/test-run.py --env)\n$ luatest -v -p my_specific_test_case\n```\n\n### Used By\n\n- [Tarantool](https://github.com/tarantool/tarantool) - in-memory database and application server\n- [memcached](https://github.com/tarantool/memcached) - Memcached protocol 'wrapper' for Tarantool\n- [vshard](https://github.com/tarantool/vshard) - sharding based on virtual buckets\n- xsync (internal project)\n\n\u003c!-- References --\u003e\n\n[luatest]: https://github.com/tarantool/luatest\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftarantool%2Ftest-run","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftarantool%2Ftest-run","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftarantool%2Ftest-run/lists"}