{"id":20415428,"url":"https://github.com/riot-os/release-specs","last_synced_at":"2025-08-23T01:09:47.742Z","repository":{"id":36241031,"uuid":"40545382","full_name":"RIOT-OS/Release-Specs","owner":"RIOT-OS","description":"Specification for RIOT releases and corresponding test configurations","archived":false,"fork":false,"pushed_at":"2025-04-07T12:58:41.000Z","size":913,"stargazers_count":4,"open_issues_count":22,"forks_count":21,"subscribers_count":28,"default_branch":"master","last_synced_at":"2025-06-16T06:07:17.867Z","etag":null,"topics":["release","riot","riot-os","testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RIOT-OS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-08-11T14:08:57.000Z","updated_at":"2025-04-29T11:58:55.000Z","dependencies_parsed_at":"2024-04-18T14:47:16.338Z","dependency_job_id":"71d2d7c6-3ca6-41b9-ad76-79cfd54592b6","html_url":"https://github.com/RIOT-OS/Release-Specs","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/RIOT-OS/Release-Specs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RIOT-OS%2FRelease-Specs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RIOT-OS%2FRelease-Specs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RIOT-OS%2FRelease-Specs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RIOT-OS%2FRelease-Specs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RIOT-OS","download_url":"https://codeload.github.com/RIOT-OS/Release-Specs/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RIOT-OS%2FRelease-Specs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271727675,"owners_count":24810562,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-22T02:00:08.480Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["release","riot","riot-os","testing"],"created_at":"2024-11-15T06:16:00.551Z","updated_at":"2025-08-23T01:09:47.718Z","avatar_url":"https://github.com/RIOT-OS.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"RIOT Release Specs\n==================\n\nThis repository contains specifications and corresponding tests that have to be\nfulfilled before a new RIOT release can be tagged.\n\n\nProcedures\n----------\n\nEach RIOT release is accompanied by a set of release specifications, that have\nto be passed before each RIOT release. The current master of this repository\ncontains the specs, that are applicable for the next release. So if we are in\nfeature freeze before a release, the release can only be tagged once all specs\nthat are in master of this repo are passed.\n\nFor traceability of past releases, a branch and a tag with the RIOT release name\nis created on the day of the RIOT release. This way it is always clear, which\nset of specs has been passed for specific release.\n\n\nSpecifications\n--------------\n\n### Structure\n\nEach specification resides in it's own folder. Inside the folder you can find\na markdown file, that describes the specification and it's tasks in detail.\nOptionally there are can be a number of test scripts and test applications for\neach task.\n\nA specification folder might look like the following:\n```\n/04-short-description\n  | - spec-04.md\n  | - test-04.01.py     -\u003e test script for spec 04, task 01\n  | - test-04.02.py\n  | - test-04.03/       -\u003e folder containing a specific RIOT test application\n  |     | - main.c\n  |     | - Makefile\n  | - test-04.03.sh     -\u003e some script making use of the test-04.03 application\n  | - ....\n/05-some-other-spec\n  | - spec-05.md\n  ...\n/README.md\n```\n\n### Specs document\n\nThe release specifications are descriptions of test-cases and certain other\ncriteria that have to be tested/applied before each RIOT release. Each\nspecification contains a description of it's goal as well as a number of tasks\nthat have to be carried out (as much automated as possible).\n\nEach tasks is to be written in the style of a test-case. This means, that the\nfollowing information must be given for each task:\n- pre-requisites (e.g. RIOT application used, hardware used, network topology, )\n- descriptions of how to exactly carry out the task (step-by-step if possible)\n- a precise list of passing/failing criteria, so that there is no doubt if a\n  task was successful or not\n\n#### Missing a test or wondering why it got replaced?\n\nThe table in [REPLACED.md] documents for which release tests got deprecated and when\nthey are to be replaced and by what.\n\n### Test scripts and applications\n\nEach release specification should be accompanied by supporting scripts (and RIOT\napplications if needed), that enable to execute it's tasks to the highest degree\nof automation possible. The long-term goal is the complete automated execution\nof all tasks, though there might be some cases where this is not achievable...\n\n[REPLACED.md]: ./REPLACED.md\n\nTracking\n--------\n\n**TODO: verify if feasible of find other means to track the state**\n\nThe release spec repository also contains a tracking document (`results.md`),\nwhich contains for each release candidate a full list of tasks that have to be\nrun, including the results, the exact RIOT commit the task was run with and the\ndate when  this was done.\n\nThis document serves (i) as a tracking document during the feature freeze phase\nof a new release and (ii) as a back-reference, so we can trace which tasks have\nbeen run for old releases.\n\nThe document `results.md` should have the following format:\n```\n...\n\n215.12-RC2          -\u003e same name as the tag in RIOT\n----------\n\n#### 01-ci          -\u003e name of spec\ntask-01.01   2015-09-04   69aca17db6341f0f5acf683092af532b6eab6c53   success\ntask-01.02   2015-09-04   69aca17db6341f0f5acf683092af532b6eab6c53   fail\n#### 02-tests\ntask-02.01   2015-09-03   69aca17db6341f0f5acf683092af532b6eab6c53   success\ntask-02.02   2015-09-04   69aca17db6341f0f5acf683092af532b6eab6c53   success\n...\n\n2015.12-RC1\n-----------\n\ntask-01.01   2015-09-04   667ad24c557dd02b18a298ce50fc49b4a3c46269   fail\ntask-01.02   2015-09-04   667ad24c557dd02b18a298ce50fc49b4a3c46269   fail\n#### 02-tests\ntask-02.01   2015-09-03   2eb21d8f9694146deca8c69cbc4a82acd62d395f   success\ntask-02.02   2015-09-04   2eb21d8f9694146deca8c69cbc4a82acd62d395f   success\n#### ...\n...\n\n```\n\n`pytest` runner\n---------------\n\nMany of the specs have python script to run the tests automatically. By default\nexperiments will be launched on [IoT-LAB saclay site] for tests that require\nnon-`native` boards. The saclay site was chosen since it has most of the\nnon-`native` boards used in the release-specs. Tests running on `native` will\nalways be run locally on the executing machine.  Non-`native` can also be ran\nlocally as long as the required `BOARD`s are available (see\n[local requirements](#local-requirements)).\n\n### Requirements\n\nTo use [pytest] you need to install the [`riotctrl`][riotctrl] and\n[`iotlabcli`][iotlabcli] python packages:\n\n```sh\npip install -r requirements.txt\n```\n\nFurthermore the `PYTHONPATH` needs to include the `pythonlibs` of RIOT:\n\n```sh\nexport PYTHONPATH=${RIOTBASE}/dist/pythonlibs:${PYTHONPATH}\n```\n\nThe environment variable `RIOTBASE` must be set to *absolute path* of the\nversion of RIOT under test. E.g.\n\n```\nexport RIOTBASE=$(readlink -f ../RIOT)\n```\n\nSome tests on the `native` platform need a certain number of TAP interfaces in a\nbridge or otherwise will be skipped. The most number of TAP interfaces to date\nis required for 3.5 \"ICMPv6 stress test on native (neighbor cache\nstress)\" (11 TAP interfaces) so to not skip that, all of them should be bridged.\n\n```sh\nsudo ${RIOTBASE}/dist/tools/tapsetup/tapsetup -c 11\n```\n\n#### Iot-LAB Requirements\n\nThe site on which to run the experiments can be changed by setting the\n`IOTLAB_SITE` environment variable.\n\nMake sure you can access the testbed frontend via SSH without providing a\npassword, either by generating a dedicated key pair without password\n\n```sh\nssh-keygen\n```\n\nand adding that to your SSH config\n\n```\nHost *.iot-lab.info\n    IdentityFile \u003cgenerated private key\u003e\n```\n\nor by configuring a dedicated `ssh-agent` (you might already have one provided\nby your OS, check with `env | grep SSH_AUTH_SOCK`)\n\n```sh\neval $(ssh-agent)\nssh-add\n```\n\n#### The Things Network Requirements\n\nTo be able to run the automatic tests in spec [11-lorawan] a valid TTN account\nis required, and an application needs to be created with a device configure to\ndo `OTAA` and another configured to do `ABP`.\n\n- [Create a TTN account]\n- [Add an application]. Take note of:\n  - Application ID as `TTN_APP_ID`\n- [Register your device], one for `ABP` and a second for `OTAA`. Take note of:\n  - ABP device ID as `TTN_DEV_ID_ABP`\n  - OTAA device ID as `TTN_DEV_ID`\n- Personalize a first device for OTAA (this is the default configuration).\n  Take note of:\n  - Device EUI as `DEVEUI`\n  - Application EUI as `APPEUI`\n  - App Key as `APPKEY`\n- [Personalize device for ABP] and enable  **Reset Frame Counter** to avoid\n  having to reset the frame counters before every test run. Take note of:\n  - Device Address as `DEVADDR`\n  - Network Session Key as `NWKSKEY`\n  - App Session Key as `APPSKEY`\n- [Add an access key] to be able to use the MQTT API, the key must at least have\n  access to `messages`. Take note of:\n  - Access Key as `TTN_DL_KEY`\n\nWhen configuring a device:\n   - **LoRaWAN version**: choose **MAC V1.0.3**\n   - **Frequency plan**: Choose Europe 863-870 MHz (SF9 for RX2 - recommended)\n\nThe listed variables need to be set in the environment.\n\n[11-lorawan]: 11-lorawan/test_spec11.py\n[Create a TTN account]: https://www.thethingsnetwork.org/docs/devices/node/quick-start/#create-an-account\n[Add an application]: https://www.thethingsindustries.com/docs/integrations/adding-applications/\n[Register your device]: https://www.thethingsindustries.com/docs/devices/adding-devices/\n[Personalize device for ABP]: https://www.thethingsindustries.com/docs/devices/adding-devices/#abp-devices\n[Add an access key]: https://www.thethingsindustries.com/docs/integrations/mqtt/#creating-an-api-key\n\n#### Local Requirements\n\nTo be able to run tests locally the following requirements must be fulfilled:\n\n- udev rules mapping every `BOARD` to a specific `PORT`, e.g.: `/dev/tty${BOARD}`.\n- `makefile` included in `RIOT_MAKEFILES_GLOBAL_PRE` that sets `DEBUG_ADAPTER_ID`\n  and `PORT` for each `BOARD`.\n\nSee [multiple boards udev] for a detailed walkthrough.\n\n##### Limitations\n\n- Currently only tests that run on different `BOARD`s will work out of the box,\n  it would currently require setting an additional variable to identify each\n  `BOARD`, see [multiple boards udev].\n\nFuture work could integrate a yml file that holds the configuration, [rjl] might\nbe a good reference.\n\n### Usage\n\n```\nusage: pytest [--boards] [--hide-output] [--local] [--non-RC] [--self-test]\n              [--log-file-fmt=[LOG_FILE_FMT]]\n\noptional arguments:\n  --boards              String list of boards to use for the test, can be\n                        IOTLAB_NODE or RIOT BOARDs.\n  --hide-output         Do not log output from nodes\n  --local               Use local boards, default=False (will use IoT-LAB unless\n                        all boards are native)\n  --non-RC              Runs test even if RIOT version under test is not an RC\n  --self-test           Tests the testutils rather than running the release\n                        tests\n  --log-file-fmt=[LOG_FILE_FMT]\n                        Format for the log file name. The available variables\n                        are: `module`: The module (=specXX) of the test,\n                        `function`: The function (=taskXX) of the test, `node`:\n                        Name of the node (on IoT-LAB the URL of the node,\n                        locally board name + port), `time`: UNIX timestamp at\n                        creation time. If the provided argument is an empty\n                        string the format will be\n                        '{module}-{function}-{node}-{time}.log' and stored in\n                        the current work directory\n```\n\nRunning `tox` will do most of that for you\n\n```sh\ntox\n```\n\nWant to see what's going on? Run\n\n```sh\ntox -- --capture=tee-sys\n```\n\nTo run only local tests, run\n\n```sh\ntox -- --local\n```\n\nTo run only tests that require root permissions, run\n\n```sh\nsudo RIOTBASE=${RIOTBASE} tox -- -m sudo_only\n```\n\nTo run only a specific task you can use the `-k` option of `pytest`. It uses a\nsimple logical syntax for pattern matching so e.g.\n\n```sh\ntox -- -k \"spec03 and (task01 or task05)\"\n```\n\nwill run task 1 and 5 of spec 3. The `-k` option can be used multiple times.\nThe expressions will be AND'd e.g.\n\n```sh\ntox -- -k spec03 -k \"task01 or task05\"\n```\n\nis identical to the first example.\n\n##### Using an env file to keep persistent environment variables\n\nMost tests require a set of user specific environment variable (path to\nRIOTBASE, LoRaWAN keys, etc). In order to keep these variables persistent, a\n.devdata.env file can be place in the root directory. For that, please make\na copy of \".devdata.env.sample\" and modify accordingly:\n\n```sh\ncp .devdata.env.sample .devdata.env\n```\n\n[pytest]: https://pytest.org\n[riotctrl]: https://pypi.org/project/riotctrl/\n[IoT-LAB saclay site]: https://www.iot-lab.info/deployment/saclay/\n[multiple boards udev]: https://doc.riot-os.org/flashing.html#multiple-boards-udev\n[rjl]: https://github.com/haukepetersen/rjl\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Friot-os%2Frelease-specs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Friot-os%2Frelease-specs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Friot-os%2Frelease-specs/lists"}