{"id":18299779,"url":"https://github.com/containersolutions/tinkerbell-rpi4-workflow","last_synced_at":"2025-04-05T13:35:18.929Z","repository":{"id":87118106,"uuid":"257294730","full_name":"ContainerSolutions/tinkerbell-rpi4-workflow","owner":"ContainerSolutions","description":"Instructions and configuration files to create tinkerbell workflow for raspberry pi 4 ","archived":false,"fork":false,"pushed_at":"2020-07-15T07:10:41.000Z","size":153052,"stargazers_count":51,"open_issues_count":1,"forks_count":5,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-21T05:11:44.516Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ContainerSolutions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-20T13:46:57.000Z","updated_at":"2024-10-18T00:59:52.000Z","dependencies_parsed_at":"2023-06-19T18:10:46.082Z","dependency_job_id":null,"html_url":"https://github.com/ContainerSolutions/tinkerbell-rpi4-workflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2Ftinkerbell-rpi4-workflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2Ftinkerbell-rpi4-workflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2Ftinkerbell-rpi4-workflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2Ftinkerbell-rpi4-workflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ContainerSolutions","download_url":"https://codeload.github.com/ContainerSolutions/tinkerbell-rpi4-workflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247342641,"owners_count":20923633,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T15:10:22.171Z","updated_at":"2025-04-05T13:35:18.904Z","avatar_url":"https://github.com/ContainerSolutions.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Configuring Kubernetes cluster using tinkerbell\n\nThis repository contains an example tinkerbell (https://tinkerbell.org/) workflow for provisioning k8s control plane and\nworker nodes on top of the raspberries pi version 4 (RPi4). \n\n## Introduction:\n\nTinkerbell is engine for provisioning bare metal servers, it helps users to build fully bootable and\noperational machine from scratch. It can be used for any type of the machine from traditional \nx86_64 servers to arm based single-board computers. This repository contains step by step instruction, how \nleverage tinkerbell provisioning engine to build fully operational k8s cluster. It is assumed some familiarity \nwith tinkerbell concepts in the below instruction. \n\n## Preparation:\n### Setting local environment\nIn order to install tinkerbell locally we need to setup a virtual machine or a dedicated bare metal server. \nIn the presented example the following setup was created: \n\n![Alt text](img/tinkerbell-lab.png \"lab design\")\n\nThe lab consist of:\n1) KVM+QEMU hypervisor running on top of fedora 31\n1) Centos 7 virtual machine hosting tinkerbell provisioner\n1) Raspberries Pies version 4 workers\n\n#### Configure hypervisor network\nIn order to ensure flawless communication between RPi4 - host- vm, it is required to first setup a bridge \nin the host. In the fedora31 it can be achieved as follows: \n```bash\nsudo nmcli con add ifname packet type bridge con-name packet\nsudo ip addr add \u003cip addr\u003e dev packet\nsudo ip link  set up dev packet\nsudo nmcli con add type bridge-slave ifname \u003chv lan interface\u003e master packet\nsudo sysctl -w .net.ipv4.conf.packet.bc_forwarding=1\nsudo sysctl -w net.ipv4.ip_forward=1\nsudo sysctl -w net.bridge.bridge-nf-call-iptables=0\n```\n**Note:** Adjust the above configuration to your host. The listed kernel parameters are used to allow forwarding of dhcp\nrequest to virtual machine.\n\n#### Create virtual machine\nEnsure virtual machine is connected to the previously created bridge and it does not block traffic for http, https, dhcp, bootpc, tftp, nfs, mountd, rpc-bind.\\\nOn centos 7 as root:  \n```bash\nfirewall-cmd --permanent --add-service nfs3\nfirewall-cmd --permanent --add-service mountd\nfirewall-cmd --permanent --add-service rpc-bind\nfirewall-cmd --permanent --add-service tftp\nfirewall-cmd --permanent --add-service dhcp\nfirewall-cmd --permanent --add-port 68/udp\nfirewall-cmd --permanent --add-service http\nfirewall-cmd --permanent --add-service https\n```\n\nOptionally if you are planning on making provisioner a gateway for rpi4, configure NAT:\n```bash\nsysctl -w net.ipv4.ip_forward=1\nfirewall-cmd --permanent --direct --passthrough ipv4 -t nat -I POSTROUTING -o eth0 -j MASQUERADE -s \u003crpi4 subnet\u003e\nfirewall-cmd --reload\n```\n\n#### Prepare raspberries pies\nTake them out of the box.\n\n## Installation and configuration of the provisioner and RPi4\n### Install tinkerbell\n\nConnect to the virtual machine and follow steps from https://tinkerbell.org/setup/packet-with-terraform/provisioner/ .\nAt the end of installation, `envrc` file should be created, note its location.\\\nReplace `vmlinuz-aarch64` and `initramfs-aarch64` with the ones from:\n```bash\nwget -O /var/tinkerbell/nginx/misc/osie/current/vmlinuz-aarch64  https://storage.googleapis.com/rpi4-uefi-tinkerbell/vmlinuz-aarch64\nwget -O /var/tinkerbell/nginx/misc/osie/current/initramfs-aarch64 https://storage.googleapis.com/rpi4-uefi-tinkerbell/initramfs-aarch64\n```\n\nIt is a custom build kernel and initramfs for rpi4 to execute a workflow. The difference from the standard osie, is that workflow\nis directly executed in the initramfs stage. The kernel was build to execute workflow contained in this repository, it is possible\nit won't work with other workflows.\n   \n### Raspberry Pi\n\nInstall rpi4 uefi (https://rpi4-uefi.dev/) onto sd card,  use the prepared image:\n\n```bash\nwget -O rpi4-uefi.img https://storage.googleapis.com/rpi4-uefi-tinkerbell/rpi4-uefi.img\ndd if=worker/rpi4-uefi.img of=/dev/mmcblk0 #replace it according to your sd card device\n```\n\nWhen booting rpi4 for the first time you have to change boot order to boot from pxe.\n\n **Note:** If you decide to use upstream rpi4-uefi image, make sure that it is using `device tree`. The option in \n question can be found in the uefi menu under: `Device Manager` → `Raspberry Pi Configuration` → `Advanced Configuration`\n \n ## Configure workflows\n Workflow is a set of task which are executed in order and are used to configure the bare metal machine. \n Each of the task is executed in th separate docker container. All task are supervised by a tink-worker, which\n gathers logs, communicates with external services and handles errors.\n \n ### Prepare workflow\n Since workflows are executed in-memory on raspberry pi, it is required to ensure tink-worker and fluent-bit support that architecture, at the time \n of writing tink-worker was not available, but we prepared one for you, you can pull it from here:\n ```bash\ndocker pull ottovsky/tink-worker:armv7-latest\ndocker pull fluent/fluent-bit:arm32v7-1.3.8\n```\nOnce pulled, tag it with your ${TINKERBELL_HOST_IP} registry and push it:\n````bash\ndocker tag ottovsky/tink-worker:armv7-latest ${TINKERBELL_HOST_IP}/tink-worker:armv7\ndocker tag fluent/fluent-bit:arm32v7-1.3.8  ${TINKERBELL_HOST_IP}/fluent-bit:1.3-arm\ndocker push ${TINKERBELL_HOST_IP}/tink-worker:armv7\n${TINKERBELL_HOST_IP}/fluent-bit:1.3-arm\n````\n\nNext follow instructions from workflow directory.\n\n### Create workflow\n1) Source `envrc`\n1) Register raspberries pies hardware in tinkerbell\n    ```bash\n    cat templates/hw.json | envsubst UUID=\"$(uuidgen)\" IP=\"\u003cdesired ip\u003e\" MASK=\"255.255.255.0\" GW=\"\u003cdesired gw ip\u003e\" MAC=\"\u003cmac of RPi\u003e\" HOSTNAME=\"\u003cdesire hostname\u003e\" \u003e device.json\n    docker cp device.json deploy_tink-cli_1:/root\n    docker exec -ti deploy_tink-cli_1 tink hardware push --file /root/device.json\n    ```\n   Example output after all RPi were registered:\n   ````bash\n    docker exec -ti deploy_tink-cli_1 tink hardware all\n    {\"id\": \"5156f2b7-b0bc-403d-a3f7-5a2db8a40918\", \"arch\": \"aarch64\", \"hostname\": \"master-1\", \"allow_pxe\": true, \"ip_addresses\": [{\"address\": \"192.168.2.35\", \"netmask\": \"255.255.255.0\", \"address\n    _family\": 4}], \"network_ports\": [{\"data\": {\"mac\": \"dc:a6:32:7a:28:91\"}, \"name\": \"eth0\", \"type\": \"data\"}], \"allow_workflow\": true}\n    {\"id\": \"e247beba-0a29-4f76-91f5-c74e89b8b74d\", \"arch\": \"aarch64\", \"hostname\": \"worker-1\", \"allow_pxe\": true, \"ip_addresses\": [{\"address\": \"192.168.2.36\", \"netmask\": \"255.255.255.0\", \"address\n    _family\": 4}], \"network_ports\": [{\"data\": {\"mac\": \"dc:a6:32:7a:2a:65\"}, \"name\": \"eth0\", \"type\": \"data\"}], \"allow_workflow\": true}\n    {\"id\": \"9f26de7a-3149-4833-a453-8a73e95a1d53\", \"arch\": \"aarch64\", \"hostname\": \"worker-2\", \"allow_pxe\": true, \"ip_addresses\": [{\"address\": \"192.168.2.37\", \"netmask\": \"255.255.255.0\", \"address\n    _family\": 4}], \"network_ports\": [{\"data\": {\"mac\": \"dc:a6:32:7a:29:e1\"}, \"name\": \"eth0\", \"type\": \"data\"}], \"allow_workflow\": true}\n   ````\n   \n   **Note:** `\"facility_code\": \"onprem initrd=initramfs-aarch64 #\"` is a workaround to disable `console` and to ensure proper boot of kernel. \n   We are leveraging the absence of sanity check in the `boots` component: https://github.com/tinkerbell/boots/blob/master/installers/osie/main.go#L90 .\n\n1) Create templates\n    ```bash\n    export SSID=\u003cwifi ssid\u003e\n    export PSK=\u003cwifi psk\u003e\n    export COUNTRY=\u003cwifi country\u003e\n    export SECRET=\u003ck8s secret\u003e\n    export K3S=\"https://\u003cmaster ip\u003e:6443 \n    cat templates/workflow-master.tmpl | envsubst \u003e k8s-master.tmpl\n    cat templates/workflow-worker.tmpl | envsubst \u003e k8s-worker.tmpl\n    docker cp k8s-master.tmpl deploy_tink-cli_1:/root\n    docker cp k8s-worker.tmpl deploy_tink-cli_1:/root\n    docker exec -ti deploy_tink-cli_1 tink template create -n k8s-master -p /root/k8s-master.tmpl\n    docker exec -ti deploy_tink-cli_1 tink template create -n k8s-worker -p /root/k8s-worker.tmpl\n    ```\n    Note down the returned ids of the templates. \\\n    Example output of created templates:\n    ```bash\n    docker exec -ti deploy_tink-cli_1 tink template list\n    +--------------------------------------+---------------+-------------------------------+-------------------------------+\n    | TEMPLATE ID                          | TEMPLATE NAME | CREATED AT                    | UPDATED AT                    |\n    +--------------------------------------+---------------+-------------------------------+-------------------------------+\n    | 4c4deca8-7e80-4518-9729-4874a166a729 | k8s-master    | 2020-05-07 08:07:31 +0000 UTC | 2020-05-07 08:07:31 +0000 UTC |\n    | b6fa353b-d20e-434d-9652-4ebe5231257b | k8s-worker    | 2020-05-07 08:05:57 +0000 UTC | 2020-05-07 08:05:57 +0000 UTC |\n    +--------------------------------------+---------------+-------------------------------+-------------------------------+\n    ```\n\n1) Create workflows, in orderd to this, it is required to assign template id with target id.\n    ```bash\n    docker exec -ti deploy_tink-cli_1 tink workflow create -t \u003ctemplate id\u003e -r '{\"device_1\": \"\u003cip of the taget device\u003e\"}'\n    ```\n    In this step, you decide what role should be assigned to RPi. for example if you would like one RPi to be k8s master, \n    you have to assign targeted hardware IP to the k8s-master template.\n    Note down the workflow ids.\n\n ## Execute workflow\n \n Insert empty sd card to the raspberry pi and power it up. Wait till the workflow is executed:\n ```bash\ndocker exec -ti deploy_tink-cli_1 tink workflow events \u003cworkflow id\u003e \n+--------------------------------------+------------------+--------------+----------------+---------------------------------+--------------------+\n| WORKER ID                            | TASK NAME        | ACTION NAME  | EXECUTION TIME | MESSAGE                         |      ACTION STATUS |\n+--------------------------------------+------------------+--------------+----------------+---------------------------------+--------------------+\n| 5156f2b7-b0bc-403d-a3f7-5a2db8a40918 | k8s-installation | disk-wipe    |              0 | Started execution               | ACTION_IN_PROGRESS |\n| 5156f2b7-b0bc-403d-a3f7-5a2db8a40918 | k8s-installation | disk-wipe    |             11 | Finished Execution Successfully |     ACTION_SUCCESS |\n| 5156f2b7-b0bc-403d-a3f7-5a2db8a40918 | k8s-installation | os-install   |              0 | Started execution               | ACTION_IN_PROGRESS |\n| 5156f2b7-b0bc-403d-a3f7-5a2db8a40918 | k8s-installation | os-install   |            501 | Finished Execution Successfully |     ACTION_SUCCESS |\n| 5156f2b7-b0bc-403d-a3f7-5a2db8a40918 | k8s-installation | os-configure |              0 | Started execution               | ACTION_IN_PROGRESS |\n| 5156f2b7-b0bc-403d-a3f7-5a2db8a40918 | k8s-installation | os-configure |              3 | Finished Execution Successfully |     ACTION_SUCCESS |\n| 5156f2b7-b0bc-403d-a3f7-5a2db8a40918 | k8s-installation | install-k8s  |              0 | Started execution               | ACTION_IN_PROGRESS |\n| 5156f2b7-b0bc-403d-a3f7-5a2db8a40918 | k8s-installation | install-k8s  |             19 | Finished Execution Successfully |     ACTION_SUCCESS |\n| 5156f2b7-b0bc-403d-a3f7-5a2db8a40918 | k8s-installation | reboot       |              0 | Started execution               | ACTION_IN_PROGRESS |\n| 5156f2b7-b0bc-403d-a3f7-5a2db8a40918 | k8s-installation | reboot       |             14 | Finished Execution Successfully |     ACTION_SUCCESS |\n+--------------------------------------+------------------+--------------+----------------+---------------------------------+--------------------+\n\n```\n\nOnce the workflow finished, the RPi should be an operational k8s master/worker node.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontainersolutions%2Ftinkerbell-rpi4-workflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcontainersolutions%2Ftinkerbell-rpi4-workflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontainersolutions%2Ftinkerbell-rpi4-workflow/lists"}