Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/smartdevicelink/sdl_atf
Automated Tests for SmartDeviceLink
https://github.com/smartdevicelink/sdl_atf
Last synced: 4 days ago
JSON representation
Automated Tests for SmartDeviceLink
- Host: GitHub
- URL: https://github.com/smartdevicelink/sdl_atf
- Owner: smartdevicelink
- License: bsd-3-clause
- Created: 2016-03-02T01:30:48.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2022-04-13T14:38:21.000Z (over 2 years ago)
- Last Synced: 2024-04-13T03:34:30.830Z (7 months ago)
- Language: Lua
- Size: 1.47 MB
- Stars: 5
- Watchers: 30
- Forks: 35
- Open Issues: 15
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# Automated Test Framework (ATF)
## Dependencies:
Library | License
---------------------- | -------------
**Lua libs** |
liblua5.2 | MIT
json4lua | MIT
lua-stdlib | MIT
lua-lpeg |
**Qt libs** |
Qt5.9 WebSockets | LGPL 2.1
Qt5.9 Network | LGPL 2.1
Qt5.9 Core | LGPL 2.1
Qt5.9 Test | LGPL 2.1
**Other libs** |
lpthread | LGPL
OpenSSL (ssl, crypto) | OpenSSL License
libxml2 | MIT
ldoc | MIT/X11
docker-ce | Apache
screen | GNU
rsync | GNU
## Get source code
```
$ git clone --recurse-submodules https://github.com/smartdevicelink/sdl_atf.git
$ git clone https://github.com/smartdevicelink/sdl_atf_test_scripts.git
```
## Compilation
**1.** Install 3rd-parties developers libraries
- Run the following commands:
```
$ sudo apt-get install lua5.2 liblua5.2-dev libxml2-dev lua-lpeg-dev
$ sudo apt-get install openssl libssl-dev net-tools
```
**2.** Install Qt5.9+
- Run the following command:
```
$ sudo apt-get install libqt5websockets5 libqt5websockets5-dev
```
**3.** Build ATF
CMake 3.15 or newer is required to generate the build files and can be downloaded from [here](https://cmake.org/download/).
- Create a build folder outside of the `sdl_atf` folder, e.g. `atf_build`
- Assume there is the following directory structure:
```
..
├sdl_atf/
├sdl_atf_test_scripts/
└atf_build
```
- Go into `atf_build` folder and run commands:
```
$ cmake ../sdl_atf
$ make
$ make install
```
- When build process is finished navigate to `atf_build/bin` folder
**4.** Create symlinks to folders in scripts repository
```
$ ln -s ../../sdl_atf_test_scripts/files
$ ln -s ../../sdl_atf_test_scripts/test_sets
$ ln -s ../../sdl_atf_test_scripts/test_scripts
$ ln -s ../../sdl_atf_test_scripts/user_modules
```
**5.** Install dependencies for local parallel mode
Steps below are required only in case if local parallel mode is going to be used.
- Install Docker, e.g. [how-to](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04)
- Install additional tools:
```
$ sudo apt-get install screen rsync
```
- Create docker image
```
$ cd atf_parallels/docker
$ ./build.sh
```
Note: accepted values are `18` and `20` (will be processed as `18.04`, and `20.04` correspondingly).
If version is not specified `20` will be used by default.
## Settings
### Defaults
By default ATF reads the following parameters from `Default` configuration (`./modules/configuration/`).
Config File | Config Parameter | Cmd Argument | Description
----------------------|---------------------------------|-------------------|-----------------------------------------
base_config.lua | config.SDL | | SDL binary name
base_config.lua | config.reportPath | --report | Path to reports and logs
base_config.lua | config.pathToSDL | --sdl-core | Path to SDL binaries
base_config.lua | config.pathToSDLSource | --sdl-src | Path to SDL Core
base_config.lua | config.pathToSDLMobileInterface | --sdl-mobile-api | Path to SDL MOBILE API
base_config.lua | config.pathToSDLHMIInterface | --sdl-hmi-api | Path to SDL HMI API
connection_config.lua | config.remoteConnection.enabled | | Defines if remote connection is enabled
### Priorities
1. Command line argument
2. Specific config
3. Base config
Where 1 is max and 3 is min priority.
This means if some particular option is defined in all 3 places ATF will use the one from highest priority place.
E.g. if path to SDL binaries is defined through `--sdl-core` command line argument (1), in specific config (2) and also in base config (3), ATF will use the value from command line argument.
## Run
Being in `atf_build/bin` folder run the command:
```
./start.sh TEST [OPTIONS]...
```
- TEST - test target, is one of the following:
- test script
- test set
- folder with test scripts
- [OPTION] - is one or more of available options:
- --sdl-core <path> - path to SDL binaries
- --config <folder> - name of the folder with configuration
- --sdl-src <path> - path to SDL Core
- --sdl-mobile-api <path> - path to SDL MOBILE API
- --sdl-hmi-api <path> - path to SDL HMI API
- --report <path> - path to report and logs
- --sdl-log [ACTION] - how to collect SDL logs:
'yes' - (default) always save, 'no' - do not save, 'fail' - save if script failed or aborted
- --sdl-core-dump [ACTION] - how to collect SDL core dumps:
'yes' - (default) always save, 'no' - do not save, 'fail' - save if script failed or aborted
- --parallels - force to use local parallel mode
- -j|--jobs <n> - number of simultaneous jobs to start
- --third-party <path> - path to SDL third party
- --tmp <path> - path to temporary folder
- --copy-atf-ts - force copying of ATF test scripts instead of creating symlinks
In case if folder is specified:
- only scripts which name starts with number will be taken into account (e.g. 001, 002 etc.)
- if there are sub-folders scripts will be run recursively
Besides execution of .lua scripts ATF also does auxiliary actions:
- clean up SDL and ATF folders before running of each script
- backup and restore SDL important files
- create report with all required logs for each script including SDL core dumps (in local parallel mode)
### Modes
#### Local consecutive
Test scripts will be run locally one by one.
*Example 1* - Single script:
```
./start.sh ./test_scripts/Smoke/Policies/001_PTU_all_flows.lua
```
*Example 2* - Multiple scripts from some folder:
```
./start.sh ./test_scripts/Smoke/Policies
```
*Example 3* - Test set:
```
./start.sh ./test_sets/smoke_tests.txt
```
#### Local parallel
Test scripts will be run locally in isolated environments and, if required, in several threads.
Test targets described in examples 1 - 3 can be executed in parallel mode by adding --parallels option.
In addition number of parallel jobs can be defined by setting `-j` (or `--jobs`) option.
*Example 4* - Single script in parallel mode:
```
./start.sh ./test_scripts/Smoke/Policies/001_PTU_all_flows.lua --parallels
```
*Example 5* - Multiple scripts from some folder in parallel mode split into 4 threads:
```
./start.sh ./test_scripts/Smoke/Policies -j 4
```
*Example 6* - Test set in parallel mode split into 2 threads:
```
./start.sh ./test_sets/smoke_tests.txt -j 2
```
In case if number of jobs is more than one `--parallels` option can be omitted.
#### Remote consecutive
Test scripts will be run using remote connection.
In this mode `RemoteTestingAdapterServer` should be run on the same host as SDL
Test targets described in examples 1 - 3 can be executed in remote mode.
Appropriate parameters needs to be defined in configuration files:
Config Parameter | Description
------------------------------- | -----------------------------------
config.pathToSDL | Path to SDL on a remote host
config.remoteConnection.enabled | Set to `true` for remote connection
config.remoteConnection.url | SDL host URL
config.mobileHost | Mobile host URL
config.sdl_logs_host | SDL host URL to receive log
The best approach is to use predefined configuration (e.g. `remote_linux`) as base and update some parameters in it.
*Example 7* - Test set in remote connection mode:
```
./start.sh ./test_sets/smoke_tests.txt --config remote_linux
```
### Advanced options
1. For a big tests sets (>1000 scripts) Report and Logs can be very huge (>10Gb). Most of the space is occupied by SDL logs. In order to turn them off `--sdl-log` or `--sdl-core-dump` options with `no` or `fail` value can be specified.
2. Some scripts (old policy ones) create the same temporary files inside `files`, `test_scripts` or `user_modules` folders. In case of parallel mode the same temporary file can be used by different scripts at the same time. This leads to incorrect results or even aborts. In order to mitigate this issue `--copy-atf-ts` option can be specified. It tells ATF to copy mentioned folders for each job instead of creating symlinks.
3. By default scripts for vehicle data executed for all available parameters. However there is possibility to restrict parameters to be tested by defining `VD_PARAMS` environment variable. E.g. `export VD_PARAMS=gps,speed` will allow to run the tests only for `gps` and `speed` vehicle data parameters.
## Documentation generation
### Download and install [ldoc](stevedonovan.github.io/ldoc/manual/doc.md.html)
```
$ sudo apt install luarocks
$ sudo luarocks install luasec
$ sudo luarocks install penlight
$ sudo luarocks install ldoc
$ sudo luarocks install discount
```
### Generate ATF documentation
```
$ cd sdl_atf
$ ldoc -c docs/config.ld .
```
### Open documentation
```
$ chromium-browser docs/html/index.html
```
## Run Unit Tests
```
$ make test
```