{"id":20415437,"url":"https://github.com/riot-os/murdock-scripts","last_synced_at":"2025-04-12T17:06:31.441Z","repository":{"id":12421456,"uuid":"71760995","full_name":"RIOT-OS/murdock-scripts","owner":"RIOT-OS","description":"RIOT specific Murdock scripts","archived":false,"fork":false,"pushed_at":"2025-04-07T11:49:09.000Z","size":252,"stargazers_count":2,"open_issues_count":3,"forks_count":8,"subscribers_count":28,"default_branch":"master","last_synced_at":"2025-04-07T12:38:53.001Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RIOT-OS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-10-24T07:01:54.000Z","updated_at":"2025-04-07T11:49:13.000Z","dependencies_parsed_at":"2024-01-02T14:36:08.658Z","dependency_job_id":"5a61064f-51a1-4661-bdb6-4e17308e97b6","html_url":"https://github.com/RIOT-OS/murdock-scripts","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RIOT-OS%2Fmurdock-scripts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RIOT-OS%2Fmurdock-scripts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RIOT-OS%2Fmurdock-scripts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RIOT-OS%2Fmurdock-scripts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RIOT-OS","download_url":"https://codeload.github.com/RIOT-OS/murdock-scripts/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248602279,"owners_count":21131614,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T06:16:07.849Z","updated_at":"2025-04-12T17:06:31.406Z","avatar_url":"https://github.com/RIOT-OS.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\nThis repository contains RIOT specific Murdock build scripts.\nIt is also the place to collect documentation about RIOT's CI system.\n\nSee https://github.com/kaspar030/murdock for general information about Murdock.\n\n# Overview\n\nRIOT's CI is composed of the following tools:\n\n- Murdock as bridge between github and running scripts (https://github.com/kaspar030/murdock)\n- dwq for distributing jobs to worker slaves (https://github.com/kaspar030/dwq)\n- disque, the backend of dwq (https://github.com/antirez/disque)\n- Docker build containers containing RIOT's build environment and a dwq slave (https://github.com/kaspar030/riotdocker branch dwq)\n- The scripts in this repository bridging Murdock build jobs and using dwq to build them on worker slaves\n- Some HTML files to nicely present Murdock's state (https://github.com/RIOT-OS/murdock-html)\n- A Web server proxying HTTPS to Murdock\n- SSH for authenticating worker nodes\n\n# Setup of Murdock master node\n\n## disque\n\nUbuntu has disque packaged. Other distros might need compilation by hand.\nMake sure disque is running and listening on 127.0.0.1.\n\n## python modules\n\n    # pip3 install dwq\n    # pip3 install agithub\n    # pip3 install pytoml\n\n## Murdock\n\nSee https://github.com/kaspar030/murdock/blob/master/README.md for generic\nMurdock instructions. This guide assumes that there's a murdock user, its home\ndirectory is \"/srv/murdock\".\n\nAdditionally, clone this repository (murdock-scripts) to\n\"/srv/murdock/murdock-scripts\".\n\nCreate \"/etc/murdock.toml\" with the following content (adapt URLs, paths and\ncredentials):\n\n    data_dir = \"/srv/http/ci.riot-labs.de-devel/devel\"\n    scripts_dir = \"/srv/murdock/murdock-scripts\"\n    http_root = \"https://ci.riot-labs.de/devel\"\n    context = \"Murdock2\"\n    github_username = \"kaspar030\"\n    github_password = \"xxx\"\n    repos = [\"RIOT-OS/RIOT\"]\n    fail_labels = [ \"NEEDS SQUASHING\", \"Waiting For Other PR\" ]\n    set_status = true\n\n(For testing, better keep \"set_status\" set to false\"...)\n\nNow start \"/srv/murdock/murdock/murdock.py /etc/murdock.toml\" using your\nservice supervisor of choice.\n\n## setup dwq authentication\n\nCreate a user that'll be used as SSH authentication user.\nThis user can be restricted to do port forwarding to localhost:7711.\nIts .ssh/authorized_keys will contain ssh keys for every allowed worker.\n\n## Setup worker nodes\n\nSee https://github.com/kaspar030/riotdocker/blob/dwq/README.md.\n\n## Setup frontend HTTP proxy\n\nnginx example (in addition to SSL setup):\n\n- location / points to the content of https://github.com/RIOT-OS/murdock-html\n\n\n```\n    location /github {\n\n        proxy_set_header        Host $host;\n        proxy_set_header        X-Real-IP $remote_addr;\n        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;\n        proxy_set_header        X-Forwarded-Proto $scheme;\n\n    # Fix the “It appears that your reverse proxy set up is broken\" error.\n        proxy_pass          http://localhost:3000/github;\n        proxy_read_timeout  90;\n    }\n\n\n    location /api {\n\n        proxy_set_header        Host $host;\n        proxy_set_header        X-Real-IP $remote_addr;\n        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;\n        proxy_set_header        X-Forwarded-Proto $scheme;\n\n    # Fix the “It appears that your reverse proxy set up is broken\" error.\n        proxy_pass          http://localhost:3000/api;\n        proxy_read_timeout  90;\n    }\n\n    location /status {\n        proxy_pass http://localhost:3000/status;\n        proxy_http_version 1.1;\n        proxy_set_header Upgrade $http_upgrade;\n        proxy_set_header Connection \"upgrade\";\n    }\n```\n\n\n# Cluster management\n\n## Perequisites\n\nInstall autossh using package manager, and \"dwq\" from pip3.\n\n\n## Setup\n\n1. set up ssh authentication to ci.riot-os.org.\n\nE.g., add this to `~/.ssh/config`:\n\n```\nHost murdock\nHostName ci.riot-os.org\nUser murdock-slave\nPort 22\nIdentityFile ~/.ssh/id_rsa_murdock-slave\nIdentitiesOnly yes\nLocalForward 7711 127.0.0.1:7711\nLocalForward 6379 127.0.0.1:6379\nServerAliveInterval 60\nServerAliveCountMax 2\n```\n\nMake sure `~/.ssh/id_rsa_murdock-slave` can log in to `murdock-slave@ci.riot-os.org`.\n\n2. keep an ssh connection open that forwards the ports 7711 and 6379.\n\nE.g., use this alias and \"autossh\":\n\n    $ alias dwq_connect='autossh -M0 -N -C -f murdock'\n\nThen start up autossh with `dwq_connect` (automate this or repeat for each session).\n\n\n## dwqm (dwq management utility)\n\nTry \"dwqm --help\".\n\nUseful things:\n\n- list all queues in the disque instance:\n\n    $ dwqm queue --list\n\nThis is a raw queue listing and includes queues used internally by dwq. Those\nare named \"control::*\" and \"status::*\".\n\n- list all connected workers:\n\n    $ dwqm control --list\n\n- set worker(s) to \"paused\", will not run any jubs until resumed or restarted:\n\n    $ dwqm control --pause  worker1 [worker2] ...\n\n- resume worker(s):\n\n    $ dwqm control --resume  worker1 [worker2] ...\n\n- shutdown worker(s) (with our current murdock scripts, this will shutdown the\n  worker, pull the newest build container, then __restart__ the worker):\n\n    $ dwqm control --shutdown  worker1 [worker2] ...\n\n## dwqc (dwq client, runs jobs on queue)\n\nIn our setup, every build worker listens on the \"default\" queue. Those workers\nare executing inside of the build container.\n\nEvery test worker listens on a queue named after the board it is connected to,\ne.g., \"samr21-xpro\", \"nrf52dk\" or \"esp32-wroom-32\".\n\n__every__ worker also listens on a queue named after it's hostname\n\nFor example, in our setup, \"riotbuild\" listens on the queues \"default\" and\n\"riotbuild\", \"pi-36f90aef\" listend on \"pi-36f90aef\" and \"nrf52dk\".\n\n`dwqc` needs a git repo and commit either as parameters or via environment.\nEither manually set \"DWQ_REPO\" and \"DWQ_COMMIT\", or use an alias:\n\n    $ alias dwqset='export DWQ_REPO=https://github.com/RIOT-OS/RIOT DWQ_COMMIT=$(git rev-parse HEAD)'\n    $ cd src/riot\n    $ dwqset    # following dwqc jobs will now be executed in the specified checkout\n\n\nRun a single job on the queue named \"default\":\n\n    $ dwqc \"echo hello world!\"\n\nRun a single job on a specific queue:\n\n    $ dwqc -q riotbuild \"ccache -s\"\n\nRun multiple jobs on a single queue:\n\n    $ for i in $(seq 10); do echo \"echo $i\"; done | dwqc -q queue_name\n\nCreate command from stdin plus base command:\n\n    $ echo \"first second third\" | dwqc -s \"echo \\${1}\" # will create job \"echo first\"\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Friot-os%2Fmurdock-scripts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Friot-os%2Fmurdock-scripts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Friot-os%2Fmurdock-scripts/lists"}