https://github.com/tarantool/test-run
Tarantool functional testing framework
https://github.com/tarantool/test-run
Last synced: about 1 year ago
JSON representation
Tarantool functional testing framework
- Host: GitHub
- URL: https://github.com/tarantool/test-run
- Owner: tarantool
- Created: 2015-04-22T13:17:21.000Z (about 11 years ago)
- Default Branch: master
- Last Pushed: 2024-12-26T17:26:23.000Z (over 1 year ago)
- Last Synced: 2025-03-28T04:02:38.404Z (over 1 year ago)
- Language: Python
- Homepage:
- Size: 962 KB
- Stars: 13
- Watchers: 30
- Forks: 17
- Open Issues: 77
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Tarantool Functional testing framework
[](https://coveralls.io/github/tarantool/test-run)
### Test Suite
Bunch of tests, that lay down in the subfolder (recursively) with `suite.ini`
file. `suite.ini` is basic ini-file, that consists of one section `default`,
and a number of fields:
* `core`
* `description` - Test Suite description
* `script` - shebang file to start tarantool with
* disables:
* `disabled` - tests that must be skipped
* `release_disabled` - tests that must be skipped when Tarantool has been
builded with `Release`
* `valgrind_disabled` - tests that must be skipped when Valgrind is enabled
* `lua_libs` - paths for lua files, that should be copied into the folder,
where server is started (delimited with the space, e.g. `lua_libs=lua/1.lua
lua/2.lua`)
* `long_run` - mark tests as long, enabled only with `--long` option (delimited
with the space, e.g. `long_run=t1.test.lua t2.test.lua`)
* `config` - test configuration file name
Field `core` must be one of:
* `luatest` - [luatest][luatest] compatible test suite
* `tarantool` - Test-Suite for Functional Testing
* `app` - Another functional Test-Suite
* `unittest` - Unit-Testing Test Suite
### Test
Each test consists of files `*.test(.lua|.sql|.py)?`, `*.result`, and may have
skip condition file `*.skipcond`. On first run (without `.result`) `.result`
is generated from output. Each run, in the beggining, `.skipcond` file is
executed. In the local env there's object `self`, that's `Test` object. If test
must be skipped - you must put `self.skip = 1` in this file. Next,
`.test(.lua|.py)?` is executed and file `.reject` is created, then `.reject` is
compared with `.result`. If something differs, then 15 last string of this diff
file are printed and `.reject` file is saving in the `/rejects/`
subfolder given in options or set localy as `var/rejects/` by default.
If not, then `.reject` file is deleted.
### Test configuration
Test configuration file contains config for multiple run. For each test section
system runs separated test and compares result with common `.result` file. For
example we need to run one test for different db engines("*" means default
configuration):
```json
{
"my.test.lua": {
"first": {"a": 1, "b": 2},
"second": {"a": 1, "b": 3}
},
"*": {
"memtx": {"engine": "memtx"},
"vinyl": {"engine": "vinyl"}
}
}
```
In test case we can get configuration from inspector:
```lua
engine = test_run:get_cfg('engine')
-- first run engine is 'memtx'
-- second run engine is 'vinyl'
```
"engine" value has a special meaning for *.test.sql files: if it is "memtx" or
"vinyl", then the corresponding default engine will be set before executing
commands from a test file. An engine is set with the following commands:
```sql
UPDATE "_session_settings" SET "value" = 'memtx|vinyl' WHERE "name" = 'sql_default_engine'
pragma sql_default_engine='memtx|vinyl'
```
If the first fails, then the second will be executed. When both fails, fail the test.
#### Python
Files: `.test.py`, `.result` and `.skipcond`(optionaly).
Environment:
* `sql` - `BoxConnection` class. Convert our subclass of SQL into IProto query
and then decode it. Print into `.result` in YAML. Examples:
* `sql("select * from t where k=[ limit ]")`
* `sql("insert into t values ([ [, ]*])")`
* `sql("delete from t where k=")`
* `sql("call ([string|number]*)")`
* `sql("update t set [k= [, k=]*] where k="")`
* `sql("ping")`
* `admin` - `AdminConnection` - simply send admin query on admin port (LUA),
then, receive answer. Examples
* `admin('box.info')`
**Example:**
```python
import os
import time
from lib.admin_connection import AdminConnection
from lib.tarantool_server import TarantoolServer
master = server
admin("box.info.lsn") # equivalent to master.admin("box.info.lsn") and server.admin(...)
sql("select * from t0 where k0=1")
replica = TarantoolServer()
replica.script = 'replication/replica.lua'
replica.vardir = os.path.join(server.vardir, "replica")
replica.deploy()
master.admin("box.insert(0, 1, 'hello')")
print('sleep_1')
time.sleep(0.1)
print('sleep_finished')
print('sleep_2')
admin("require('fiber').sleep(0.1)")
print('sleep_finished')
replica.admin("box.select(0, 0, 1)")
con2 = AdminConnection('localhost', server.admin.port)
con2("box.info.lsn")
replica.stop()
replica.cleanup()
con2.disconnect()
```
**Result:**
```yaml
box.info.lsn
---
- null
...
select * from t0 where k0=1
---
- error:
errcode: ER_NO_SUCH_SPACE
errmsg: Space '#0' does not exist
...
box.insert(0, 1, 'hello')
---
- error: '[string "return box.insert(0, 1, ''hello'')"]:1: attempt to call field ''insert''
(a nil value)'
...
sleep_1
sleep_finished
sleep_2
require('fiber').sleep(0.1)
---
...
sleep_finished
box.select(0, 0, 1)
---
- error: '[string "return box.select(0, 0, 1)"]:1: attempt to call field ''select''
(a nil value)'
...
box.info.lsn
---
- null
...
```
#### Lua
Files: `.test.lua`, `.result` and `.skipcond`(optionaly).
Tests interact only with `AdminConnection`. Supports some preprocessor functions (eg `delimiter`)
**Delimiter example:**
```
env = require('test_run')
test_run = env.new()
box.schema.space.create('temp')
t1 = box.space.temp
t1:create_index('primary', { type = 'hash', parts = {1, 'num'}, unique = true})
t1:insert{0, 1, 'hello'}
test_run:cmd("setopt delimiter ';'")
function test()
return {1,2,3}
end;
test(
);
test_run:cmd("setopt delimiter ''");
test(
);
test
```
**Delimiter result:**
```
env = require('test_run')
test_run = env.new()
box.schema.space.create('temp')
---
- index: []
on_replace: 'function: 0x40e4fdf0'
temporary: false
id: 512
engine: memtx
enabled: false
name: temp
field_count: 0
- created
...
t1 = box.space.temp
---
...
t1:create_index('primary', { type = 'hash', parts = {1, 'num'}, unique = true})
---
...
t1:insert{0, 1, 'hello'}
---
- [0, 1, 'hello']
...
test_run:cmd("setopt delimiter ';'")
function test()
return {1,2,3}
end;
---
...
test(
);
---
- - 1
- 2
- 3
...
test_run:cmd("setopt delimiter ''");
test(
---
- error: '[string "test( "]:1: unexpected symbol near '''''
...
);
---
- error: '[string "); "]:1: unexpected symbol near '')'''
...
test
---
- 'function: 0x40e533b8'
...
```
It is possible to use backslash at and of a line to carry it.
```lua
function echo(...) \
return ... \
end
```
#### SQL
*.test.sql files are just SQL statements written line-by-line.
It is possible to mix SQL and Lua commands using `\set language lua` and `\set
language sql` commands.
##### Interaction with the test environment
In lua test you can use `test_run` module to interact with the test
environment.
```lua
env = require('test_run')
test_run = env.new()
test_run:cmd("")
```
__Base directives:__
* `setopt delimiter ''` - Sets delimiter to ``\n
__Server directives:__
* `create server with ...` - Create server with name ``, where `...`
may be:
* `script = ''` - script to start
* `rpl_master = ` - replication master server name
* `start server ` - Run server ``
* `stop server [with signal=]` - Stop server ``
* `` is a signal name (with or without 'SIG' prefix, uppercased) or
a signal number to use instead of default SIGTERM
* `cleanup server ` - Cleanup (basically after server has been stopped)
* `restart server ` - Restart server `` (you can restart yourself
from lua!)
__Connection switch:__
* `switch ` - Switch connection to server `` and add test run into
global scope
__Connection directives(low level):__
* `create connection to ` - create connection named
`` to `` server
* `drop connection ` - Turn connection `` off and delete it
* `set connection ` - Set connection `` to be main, for next commands
__Filter directives:__
* `push filter '' to ''` - e.g. `push filter 'listen: .*' to 'listen: '`
__Set variables:__
* `set variables '' to ''` - execute
` = *` where * is value of where. Where must be
* `.admin` - admin port of this server
* `.master` - listen port of master of this replica
* `.listen` - listen port of this server
__Dev ops features:__
You can power on any tarantool replicas in a loop.
```lua
test_run:cmd('setopt delimiter ";"')
function join(inspector, n)
for i=1,n do
local rid = tostring(i)
os.execute('mkdir -p tmp')
os.execute('cp ../replication/replica.lua ./tmp/replica'..rid..'.lua')
os.execute('chmod +x ./tmp/replica'..rid..'.lua')
inspector:cmd("create server replica"..rid.." with rpl_master=default, script='./var/tmp/replica"..rid..".lua'")
inspector:cmd("start server replica"..rid)
end
end;
test_run:cmd('setopt delimiter ""');
-- create 30 replicas for current tarantool
join(test_run, 30)
```
### pretest_clean()
Nothing will be done before a Python test and for `core = unittest`
test suites.
For a `core = [app|tarantool]` test suites this function removes tarantool WAL
and snapshot files before each test.
The following files will be removed:
* `*.snap`
* `*.xlog`
* `*.vylog`
* `*.inprogress`
* `[0-9]*/`
### Tags
Usage:
```sh
./test-run.py --tags foo
./test-run.py --tags foo,bar app/ app-tap/
```
test-run will run only those tests, which have at least one of the
provided tags.
Show a list of tags:
```sh
./test-run.py --tags
./test-run.py app-tap/ --tags
```
The tags metainfo should be placed within a first comment of a test
file.
Examples:
* .lua file:
```lua
#!/usr/bin/tarantool
-- tags: foo, bar
-- tags: one, more
<...>
```
* .sql file:
```sql
-- tags: foo
-- tags: bar
<...>
```
* .py file:
```python
# tags: foo
<...>
```
Unsupported features:
* Marking unit tests with tags.
* Multiline comments (use singleline ones for now).
### Using luatest
test-run supports tests written in the [luatest][luatest] format. `*_test.lua`
files in a `core = luatest` test suite are run as part of `./test/test-run.py`
invocation: no extra actions are needed.
You can also run a particular test using a substring of its full name:
```shell
$ ./test/test-run.py foo-luatest/bar_test.lua
$ ./test/test-run.py bar_test.lua
$ ./test/test-run.py bar
```
If you need to run a particular test case from a luatest compatible test, use
`luatest` command directly. In order to use luatest, which is bundled into
test-run, source test-run's environment:
```shell
$ . <(./test/test-run.py --env)
$ luatest -v -p my_specific_test_case
```
### Used By
- [Tarantool](https://github.com/tarantool/tarantool) - in-memory database and application server
- [memcached](https://github.com/tarantool/memcached) - Memcached protocol 'wrapper' for Tarantool
- [vshard](https://github.com/tarantool/vshard) - sharding based on virtual buckets
- xsync (internal project)
[luatest]: https://github.com/tarantool/luatest