{"id":18794854,"url":"https://github.com/acerv/runltp-ng-original","last_synced_at":"2025-04-13T15:33:59.788Z","repository":{"id":50326252,"uuid":"518503330","full_name":"acerv/runltp-ng-original","owner":"acerv","description":"Next-gen LTP runner","archived":true,"fork":false,"pushed_at":"2022-12-16T10:50:00.000Z","size":237,"stargazers_count":2,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-05T16:50:10.436Z","etag":null,"topics":["kernel","linux","linux-kernel","linux-test","linux-test-project","python","testing-framework"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/acerv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-27T15:00:01.000Z","updated_at":"2024-01-12T18:34:00.000Z","dependencies_parsed_at":"2023-01-29T12:45:28.032Z","dependency_job_id":null,"html_url":"https://github.com/acerv/runltp-ng-original","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acerv%2Frunltp-ng-original","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acerv%2Frunltp-ng-original/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acerv%2Frunltp-ng-original/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acerv%2Frunltp-ng-original/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/acerv","download_url":"https://codeload.github.com/acerv/runltp-ng-original/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248736452,"owners_count":21153598,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kernel","linux","linux-kernel","linux-test","linux-test-project","python","testing-framework"],"created_at":"2024-11-07T21:31:19.117Z","updated_at":"2025-04-13T15:33:56.191Z","avatar_url":"https://github.com/acerv.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Runltp-ng\n=========\n\nLTP Next-Gen runner is a new version of the `runltp` script used by the\n[Linux Test Project](https://github.com/linux-test-project/ltp).\n\n    Host information\n\n        System: Linux\n        Node: susy\n        Kernel Release: 5.14.21-150400.24.33-default\n        Kernel Version: #1 SMP PREEMPT_DYNAMIC Fri Nov 4 13:55:06 UTC 2022 (76cfe60)\n        Machine Architecture: x86_64\n        Processor: x86_64\n\n        Temporary directory: /tmp/runltp.acer/tmpcwtket0m\n\n    Connecting to SUT: host\n    Downloading suite: math\n    Starting suite: math\n    abs01: pass | tained  (0.005s)\n    atof01: pass | tained  (0.005s)\n    float_bessel: pass | tained  (0.702s)\n    float_exp_log: pass | tained  (0.703s)\n    float_iperb: pass | tained  (0.288s)\n    float_power: pass | tained  (0.540s)\n    float_trigo: pass | tained  (0.643s)\n    fptest01: pass | tained  (0.020s)\n    fptest02: pass | tained  (0.005s)\n    nextafter01: pass | tained  (0.004s)\n\n    Suite Name: math\n    Total Run: 10\n    Elapsed Time: 2.9 seconds\n    Passed Tests: 22\n    Failed Tests: 0\n    Skipped Tests: 0\n    Broken Tests: 0\n    Warnings: 0\n    Kernel Version: Linux 5.14.21-150400.24.33-default #1 SMP PREEMPT_DYNAMIC Fri Nov 4 13:55:06 UTC 2022 (76cfe60)\n    CPU: x86_64\n    Machine Architecture: x86_64\n    RAM: 15569564 kB\n    Swap memory: 2095424 kB\n    Distro: opensuse-leap\n    Distro Version: 15.4\n\n\n    Disconnecting from SUT: host\n\n\nQuickstart\n==========\n\nSome basic commands are the following:\n\n    # run syscalls and dio testing suites on host\n    ./runltp-ng --run-suite syscalls dio\n\n    # run syscalls and dio testing suites in qemu VM\n    ./runltp-ng --sut qemu:image=folder/image.qcow2 \\\n        --run-suite syscalls dio\n\n    # run syscalls and dio testing suites via SSH\n    # NOTE: paramiko and scp packages must be installed in the system\n    ./runltp-ng --sut=ssh:host myhost.com:user=root:key_file=myhost_id_rsa \\\n        --run-suite syscalls dio\n\nIt's possible to run a single command before running testing suites using\n`--run-cmd` option as following:\n\n    runltp-ng --run-cmd /mnt/testcases/kernel/systemcalls/bpf/bpf_prog02 \\\n        --sut qemu:image=folder/image.qcow2:virtfs=/home/user/ltp \\\n        --ltp-dir /mnt \\\n        --run-suite syscalls dio\n\nIt can be used also to run a single command without running testing suites:\n\n    runltp-ng --run-cmd /mnt/testcases/kernel/systemcalls/bpf/bpf_prog02 \\\n        --sut qemu:image=folder/image.qcow2\n\nEvery session has a temporary directory which can be found in\n`/\u003cTMPDIR\u003e/runltp-of\u003cusername\u003e`. Inside this folder there's a symlink\ncalled `latest`, pointing to the latest session's temporary directory, and the\napplication will rotate over 5 sessions.\n\nFor more information, checkout the following video at the SUSE Labs Conference\n2022:\n\n[![Watch the video](https://img.youtube.com/vi/JMeJBt3S7B0/hqdefault.jpg)](https://www.youtube.com/watch?v=JMeJBt3S7B0)\n\nSetting up console for Qemu\n===========================\n\nTo enable console on a tty device for a VM do:\n\n* open `/etc/default/grub`\n* add `console=$tty_name, console=tty0` to `GRUB_CMDLINE_LINUX`\n* run `grub-mkconfig -o /boot/grub/grub.cfg`\n\nWhere `$tty_name` should be `ttyS0`, unless virtio serial type is used (i.e.\nif you set the `serial=virtio` backend option, then use `hvc0`)\n\nImplementing SUT\n================\n\nSometimes we need to cover complex testing scenarios, where the SUT uses\nparticular protocols and infrastructures, in order to communicate with our\nhost machine and to execute tests binaries.\n\nFor this reason, `runltp-ng` provides a plugin system to recognize custom SUT\nclass implementations inside the `ltp` package folder. Please check `host.py`\nor `ssh.py` implementations for more details.\n\nOnce a new SUT class is implemented and placed inside the `ltp` package folder,\n`runltp-ng -s help` command can be used to see if application correctly\nrecognise it.\n\nDevelopment\n===========\n\nThe application is validated using `pytest` and `pylint`.\nTo run unittests:\n\n    pytest\n\nTo run linting checks:\n\n    pylint --rcfile=pylint.ini ./ltp\n\nHistory\n=======\n\nThe LTP `runltp` code is hard to read, maintain and some of its parts\nare legacy features which are not supported anymore. But if we focus closer on\nthe results, `runltp` has done its job for a while since 2001. Nowadays, with\nnew automation systems, easily accesible virtualization and bigger computing\npower, `runltp` became more and more obsolete, since its main goal was to test\nLinux Kernel on target and specific distro(s). Let's take a look at the issues\nwe have:\n\n- it's hard to maintain and it's based on a mixture of bash/C, both hard\n  to read and not maintained anymore\n- it contains many features which are not used and they can be deprecated\n- report files are custom format logs or HTML files which are both hard to parse\n  inside i.e. an automation system\n- if a test causes system crash, which is common for kernel tests, the tool\n  crashes and we loose most or even all results we obtained before its\n  execution. This means we need to run it inside a virtualized system to be\n  sure that if system crashes, we won't loose control of the machine. And, in\n  any case, we will loose testing report\n\nThe last point is really important, since in a world where cloud and embedded\nsystems are having a big market, we need to provide a usable and a stable way to\ntest Linux Kernel. Something that `runltp` is not able to achieve nowadays.\n\nThe new runltp-ng features\n--------------------------\n\nCyril Hrubis started the first Perl prototype of `runltp-ng`\n(https://github.com/metan-ucw/runltp-ng/), a next generation tests runner that\nallows to run tests on a host, as well as inside a Qemu instance or over a SSH.\nThe tool provided results in a machine parsable format which were easy to\nconsume by automation systems.\nHowever as the community didn't like the choice of Perl programming language we\ndecided to switch from Perl to Python to take advantage of the Python community\nsize, easier maintenance and packages (https://github.com/acerv/runltp-ng).\n\nIn particular, we tried to focus on missing features and got rid of the ones\nwhich were not strictly needed. We ended up with a simple and light tool having\nthe following features:\n\n- test suites can run inside a virtualized system using Qemu or they can be\n  executed via SSH protocol\n- runner became more robust so it can gracefully handle kernel crashes and\n  tained statuses of the kernel. At the moment, only Qemu supports this feature\n- report file type is JSON by default, so it will be easier to parse with\n  external tools and automation systems\n- the user interface has been simplified, so we have two modes: quiet and\n  verbose mode. The quiet mode is the default one and it shows only tests names\n  and their results on a list. Verbose mode is similar to the current `runltp`\n  stdout\n\nWhat's next?\n------------\n\nNowadays `runltp-ng` is a simple and lightweight implementation that is based on\nPython 3.6+ and it doesn't have any dependency from external packages.\nIts skeleton is easy to understand and features can be added easily.\n\nA missing feature that is currently under development is the possibility to\nexecute tests via LTX (experimental).\n\nLTX is a small service that runs on target and it permits to communicate via\nmsgpack (https://msgpack.org/) in order to execute binaries on host in the\nfastest way as possible. Its development is currently maintained by Richard\nPalethorpe and we plan to make it the default LTP runner in the next future.\nWhen LTP metadata file will be completed, LTX will also permit to execute tests\nin parallel.\n\nBy taking in consideration previous topics, we can still provide an usable and\nsimple tool that can replace current `runltp` script inside the LTP upstream.\nIts usage is simple, pretty stable and we are starting to move forward into\na modern approach to schedule and run tests in the Linux Testing Project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facerv%2Frunltp-ng-original","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Facerv%2Frunltp-ng-original","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facerv%2Frunltp-ng-original/lists"}