Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/os-autoinst/os-autoinst

OS-level test automation
https://github.com/os-autoinst/os-autoinst

hacktoberfest perl test-automation test-execution virtualization

Last synced: 15 days ago
JSON representation

OS-level test automation

Awesome Lists containing this project

README 

        
= os-autoinst image:https://github.com/os-autoinst/os-autoinst/workflows/ci/badge.svg?branch=master[link=https://github.com/os-autoinst/os-autoinst/actions] image:https://codecov.io/gh/os-autoinst/os-autoinst/branch/master/graph/badge.svg[link=https://codecov.io/gh/os-autoinst/os-autoinst]

:author: openSUSE Team at SUSE

:toc:



The OS-autoinst project aims at providing a means to run fully

automated tests.  Especially to run tests of basic and low-level

operating system components such as bootloader, kernel, installer and

upgrade, which can not easily and safely be tested with other

automated testing frameworks.  However, it can just as well be used to

test firefox and openoffice operation on top of a newly installed OS.



os-autoinst can be executed alone, but is currently designed to be

executed together with openQA, the web user interface that allows to

run more than one os-autoinst instance at the same time.



More information on os-autoinst and openQA can be found on

http://os-autoinst.github.io/openQA/



== Getting started



Under openSUSE the `os-autoinst` package can be installed from the official

repository or from our

link:https://build.opensuse.org/project/show/devel:openQA[devel repository].

For further details, have a look at the

link:http://open.qa/docs/#_development_version_repository[openQA documentation].



For building os-autoinst manually checkout the build instructions below.



The main executable `isotovideo` can read test parameters from the command

line or read test parameters from a file named `vars.json`. This file stores

the values of the different variables that will configure the behavior of the

test execution.



A container is provided and can be pulled and the main execution can be called

in one step, for example using the `podman` container engine for tests defined

in the current directory on x86_64 if your environment supports KVM

virtualization acceleration:



[source,sh]

----

podman run --rm -it -v .:/tests registry.opensuse.org/devel/openqa/containers/isotovideo:qemu-kvm casedir=/tests

----



Use the image variant ending with `qemu-x86` on x86_64 if no KVM support is

available.



Take a look on

https://registry.opensuse.org/cgi-bin/cooverview?srch_term=project%3Ddevel%3AopenQA

for all available container images.



Additional test variables can be supplied on the command line. There are some

variables used by os-autoinst itself and other that are used by the tests. A

minimal command line can look like this:



[source,sh]

----

isotovideo distri=opensuse casedir=/full/path/for/tests iso=/full/path/for/iso

----



As alternative or completementary a corresponding `vars.json` with additional

parameters could be:



[source, javascript]

-------------------------------------------------------------------

{

   "DISTRI" :      "opensuse",

   "CASEDIR" :     "/full/path/for/tests",

   "NAME" :        "test-name",

   "ISO" :         "/full/path/for/iso",

   "VNC" :         "91",

   "BACKEND" :     "qemu",

   "DESKTOP" :     "kde"

}

-------------------------------------------------------------------



Be advised that the file `vars.json` is also modified by `os-autoinst` so make

sure to backup handcrafted versions of this file.



For more concrete instructions read on in the "How to run test cases" section

below. Find sections about "How to contribute" or "Build instructions" further

below.



== How to run test cases



This following instructions shows how to run test cases. First one needs to

clone the test distribution. Checkout

link:https://github.com/os-autoinst/os-autoinst-distri-example[os-autoinst-distri-example]

for an example of a minimal test distribution.



Example for openSUSE's tests:



-----------------------------------------------------------------------------

mkdir distri && cd distri

git clone [email protected]:os-autoinst/os-autoinst-distri-opensuse.git opensuse

cd opensuse/products/opensuse

git clone [email protected]:os-autoinst/os-autoinst-needles-opensuse.git needles

-----------------------------------------------------------------------------



Example for openQA's self-tests ("openQA-in-openQA" test):



-----------------------------------------------------------------------------

mkdir distri && cd distri

git clone [email protected]:os-autoinst/os-autoinst-distri-openQA.git openqa

cd openqa

git clone [email protected]:os-autoinst/os-autoinst-needles-openQA.git needles

-----------------------------------------------------------------------------



Then create a working directory for the test execution, e.g.:



-----------------------------------------------------

mkdir /tmp/os-autoinst-run && cd /tmp/os-autoinst-run

-----------------------------------------------------



Create a minimal `vars.json` config file within that directory, e.g.:



.vars.json

[source,json]

---------------------------------------------------------

{

   "ARCH" : "x86_64",

   "BACKEND" : "qemu",

   "CASEDIR" : "/path/to/os-autoinst-distri-opensuse",

   "DESKTOP" : "gnome",

   "DISTRI" : "opensuse",

   "ISO" : "/path/to/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20160715-Media.iso",

   "PRODUCTDIR" : "/path/to/os-autoinst-distri-opensuse/products/opensuse",

   "VNC" : 90,

}

---------------------------------------------------------



You will need to correct the file paths to point to real locations. Some of the

variables you can use are listed link:doc/backend_vars.asciidoc[here]. Test case

specific variables are listed in the distri directories e.g.

link:https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/variables.md[os-autoinst-distri-opensuse/variables].



Then you can run the `isotovideo` script within the created working directory.

When doing a manual build, that script can be found at the top-level of the

os-autoinst Git checkout.



All of these examples were using the QEMU backend which is usually the easiest

backend to handle and therefore recommended. If you need to develop and test

other backends, have a look at the

link:doc/backends.md[backend-specific documentation].



When using the QEMU backend it is possible to access the system under test via

VNC:



----------------------------------------

vncviewer localhost:91 -ViewOnly -Shared

----------------------------------------



Run `isotovideo` with the environment variable `RUN_VNCVIEWER` set to autostart

a VNC viewer on the right port.



Run `isotovideo` with the environment variable `RUN_DEBUGVIEWER` to start the

internal debug screenshot viewer updated with an always recent screenshot of the

test run.



== Develop test modules



Individual test modules are written with one test module per file using the test

API in Perl code. Experimental support for test modules in the Python

programming language is provided.



Find more details about how to write tests on

http://open.qa/docs/#_how_to_write_tests



== Verifying a runtime environment



To check if your hardware is able to successfully execute os-autoinst based

tests one can execute openQA tests, all the development tests or simply call

something like



----

podman run --pull=always --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ test-perl-testsuite TESTS=t/99-full-stack.t'

----



which only requires the container runtime environment "podman" and will run a

container based os-autoinst full-stack test, here without KVM hardware

accelerated virtualization support.



== How to contribute



If you want to contribute to this project, please clone and send pull requests

via https://github.com/os-autoinst/os-autoinst.



More information on the contribution can be found on

http://os-autoinst.github.io/openQA/contact/, too.



Issues are tracked on https://progress.opensuse.org/projects/openqav3/.



For an overview of the architecture, see

link:doc/architecture.md[doc/architecture.md].



=== Rules for commits



* Every commit is checked by our CI system as soon as you create a pull request

  but you *should* run the os-autoinst tests locally. Checkout the build

  instructions for further details.



* For git commit messages use the rules stated on

  http://chris.beams.io/posts/git-commit/[How to Write a Git Commit Message] as

  a reference



* Every pull request is reviewed in a peer review to give feedback on possible

  implications and how we can help each other to improve



If this is too much hassle for you feel free to provide incomplete pull requests

for consideration or create an issue with a code change proposal.



=== Deprecation approach



In case you want to deprecate functionality consider the use of the function

`backend::baseclass::handle_deprecate_backend`.



== Build instructions



=== Installing dependencies



On openSUSE one can install the package `os-autoinst-devel` which provides all

the dependencies to build and run os-autoinst for the corresponding version of

the sources. To build a current version of `os-autoinst` it is recommended to

install `os-autoinst-devel` from

link:https://build.opensuse.org/project/show/devel:openQA[devel:openQA] as the

distribution-provided packages might be too old or miss dependencies. This is

particularly true for openSUSE Leap. Also see

link:https://open.qa/docs/#_development_version_repository[the openQA docs].



The required dependencies are also declared in `dependencies.yaml`. (The names

listed within that file are specific to openSUSE but can be easily transferred

to other distributions.)



=== Conducting the build



Simply call



----

make

----



in the top folder which automatically creates a build directory and builds the

complete project.



Call



----

make help

----



to list all available targets.



The above commands use a convenience Makefile calling `cmake`. For packaging,

when using an IDE or to conduct the steps manually it is suggested to use CMake

directly and do the following: Create a build directory outside of the source

directory. The following commands need to be invoked within that directory.



Configure build:

----

cmake $path_to_os_autoinst_checkout

----



You can specify any of the standard CMake variables, e.g.

`-DCMAKE_BUILD_TYPE=Debug` and `-DCMAKE_INSTALL_PREFIX=/custom/install/prefix`.



The following examples assume that GNU Make is used. It is possible to generate

for a different build tool by adding e.g. `-G Ninja` to the CMake arguments.



Build executables and libraries:

----

make symlinks

----



This target also creates symlinks of the built executables and libraries within

the source directory so `isotovideo` can find them.



Run all tests:

----

make check

----



By default CTest is invoked in verbose mode because prove already provides

condensed output. Add `-DVERBOSE_CTEST=OFF` to the CMake arguments to avoid

that.



Run all Perl tests (`*.t` files found within the `t` and `xt` directories):

----

make test-perl-testsuite

----



Run individual tests by specifying them explicitly:

----

make test-perl-testsuite TESTS="t/15-logging.t t/28-signalblocker.t"

----



Run perl author tests:

----

make test-local-author-perl

----



Run all author tests:

----

make test-local

----



Notice that the user needs to include the test directory for each test (either t

for normal or xt for developer-centric tests) when specifying individual tests.



Add additional arguments to the `prove` invocation, e.g. enable verbose output:

----

make test-perl-testsuite PROVE_ARGS=-v

----



Gather coverage data while running tests:

----

make test-perl-testsuite WITH_COVER_OPTIONS=1

----



Generate a coverage report from the gathered coverage data:

----

make coverage

----



If no coverage data has been gathered so far the `coverage` target will invoke

the testsuite automatically.



Reset gathered coverage data:

----

make coverage-reset

----



Install files for packaging:

----

make install DESTDIR=…

----



Automatically tidy all perl files:

----

tools/tidyall

----



Tidy all changed perl files:

----

tools/tidyall --git

----



Further notes:



* When using the `test-perl-testsuite` target, `ctest` is not used (and

  therefore `ctest` specific tweaks have no effect).

* One can always run Perl tests manually via `prove` after the build has been

  conducted with `make symlinks`. Note that some tests need to be invoked within

  the `t` directory. An invocation like

  `prove -vI.. -I../external/os-autoinst-common/lib 28-signalblocker.t` is

  supposed to work.

* It is also possible to run `ctest` within the build directory directly instead

  of using the mentioned targets.

* All mentioned variables to influence the test execution (`TESTS`,

  `WITH_COVER_OPTIONS`, …) can be combined and can also be used with the

  `coverage` target.



== Running isotovideo as CI check

We provide a container to run `isotovideo` which can be used to run QEMU-based

tests directly in a CI runner. Checkout this

https://github.com/os-autoinst/os-autoinst-distri-example/blob/main/.github/workflows/isotovideo-action.yml[example workflow]

for how it can be used. The

https://github.com/os-autoinst/os-autoinst-distri-example/blob/main/README.md#local-testing-and-ci-environment[README of the example test distribution]

also contains further details.



== Standalone tool for image search

The script `imgsearch` in the repository's script folder allows to use the

fuzzy image comparison independently of the normal test execution. Invoke the

script with no parameters to show its usage. There is also an

link:https://github.com/os-autoinst/os-autoinst/blob/master/t/imgsearch/expected-output.json[example file]

showing what output you can expect. There is one key for each file to be

searched. The best matching image to be found will show up as `match` and the

other images under `candidates`. If no image matches well enough, `match` will

be `null`.



To use the script the previously shown build instructions need to be executed

(including the invocation of the `symlinks` target).



== History of os-autoinst



At a time Bernhard M. Wiedemann who later joined was on the openSUSE testing

team and was assigned the task of testing the installer. Which meant tedious

and dull work of waiting for 4GB ISO files to download when it's not even

clear if those things even boot. And as the Perl founder Larry Wall states,

important traits of programmers are laziness, impatience and hybris. Which

quickly led to developing os-autoinst to automate installations ;) See

https://lizards.opensuse.org/2010/04/29/making-of-the-opensuse-install-video/

and https://lizards.opensuse.org/2010/05/25/automated-opensuse-testing/ for

Bernhard's blog posts.



== Further notes



When using the QEMU backend, also ensure your user running os-autoinst has

access to `/dev/kvm`.



-----------------------------------------------------------

modprobe kvm-intel || modprobe kvm-amd

chgrp kvm /dev/kvm ; chmod g+rw /dev/kvm # maybe redundant

# optionally use a new user; just to keep things separate

useradd -m USERNAME -G kvm

passwd USERNAME # and/or add ~USERNAME/.ssh/authorized_keys

-----------------------------------------------------------