{"id":13510448,"url":"https://github.com/dasJ/emergency-kexec","last_synced_at":"2025-03-30T16:33:40.645Z","repository":{"id":83663965,"uuid":"173478315","full_name":"dasJ/emergency-kexec","owner":"dasJ","description":"Kexec into an in-memory emergency system","archived":false,"fork":false,"pushed_at":"2022-03-01T10:01:37.000Z","size":19,"stargazers_count":30,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-28T16:44:59.033Z","etag":null,"topics":["emergency","kexec","linux","nix","nixos","recovery","wcgw"],"latest_commit_sha":null,"homepage":null,"language":"Nix","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dasJ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-03-02T17:33:05.000Z","updated_at":"2024-09-13T12:15:35.000Z","dependencies_parsed_at":"2023-05-04T00:41:57.729Z","dependency_job_id":null,"html_url":"https://github.com/dasJ/emergency-kexec","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dasJ%2Femergency-kexec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dasJ%2Femergency-kexec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dasJ%2Femergency-kexec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dasJ%2Femergency-kexec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dasJ","download_url":"https://codeload.github.com/dasJ/emergency-kexec/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222566739,"owners_count":17004237,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["emergency","kexec","linux","nix","nixos","recovery","wcgw"],"created_at":"2024-08-01T02:01:39.418Z","updated_at":"2024-11-01T11:30:19.163Z","avatar_url":"https://github.com/dasJ.png","language":"Nix","funding_links":[],"categories":["Nix","nix"],"sub_categories":[],"readme":"# emergency-kexec\n\nOkay, your system is completely broken, and you need to umount `/` or something like that.\nWhat do you do?\n\n## Motivation\n\nOne of our servers had a broken root filesystem (btrfs, don't judge me).\nOnline recovery was not possible, so the filesystem needed to be unmounted which is not possible for the root fs.\nAdditionally, as errors were detected, the kernel decided to mount it read only and didn't let me remount it as `rw`.\nIPMI? Yes, I had the password in my password store but not the username.\nSo the only logical solution was to kexec into an emergency system.\nThis code is what I used.\nIt recovers all IP addresses as well as SSH host and user keys from the old system and kexecs into a new one - entirely in-memory.\n\n## What it does\n\nThe `emergency` script (found in the repository root) will SSH over and execute the following things:\n\n1. Build the recovery image (a `.tar.xz` with a small nix store and a `kexec` script) from the files in this repository locally on the machine you're executing this code on\n\t1. The system configuration is found in `configuration.nix`\n\t2. Some `kexec`-related features are imported from `kexec.nix`\n\t3. The scripts will be included to be used in the `kexec` script (see below)\n2. Try to `mkdir` `/nix` and `/tmp`. If the don't already exist and your root fs is read-only, you have a problem this project can't fix\n3. Mount a fresh `tmpfs` on `/tmp` because there might not be one already\n4. `scp` the emergency image over and extract it\n5. Mount the nix store from the emergency image over `/nix` using `overlayfs`\n6. Run the kexec script\n\nThe `kexec` script (found in `kexec.nix`) will do the following:\n\n1. Prepare a second initrd\n2. Put your SSH host keys into the initrd\n3. Put all of your SSH user keys into the initrd\n4. Fetch all your IP addresses and routes and put them into the initrd\n5. Pack the second initrd and append it to the default NixOS initrd from the emergency image\n6. `kexec` into the kernel from the emergency image while using the new initrd\n7. In case you didn't already notice: **This will crash your currently running system, so maybe it's a good idea to gracefully shut down remaining daemons if that's still possible**\n\nThe script that is packed into the initrd of the new system will do the following:\n\n1. Place the SSH host key\n2. Place the SSH user keys\n3. Place a script for the IP addresses which will be executed using `networking.localCommands` so the interfaces are available\n\nIf you set the environment variable `EMERGENCY_DUMP_NETWORK` to `1`, all IPs, routes, and nameservers will be placed in the `emergency_ips`, `emergency_routes`, and `emergency_nameservers` files, respectively.\n\n## How to use\n\n```\n$ ./emergency root@somehost\n# or\n$ ./emergency somebody@somehost\n```\n\n## Disclaimer and license\n\nIf it doesn't work for you, I'm sorry.\nI can probably not help you, but if you're able to fix something, feel free to create a PR.\n\nThe code is based on [clever's](https://github.com/cleverca22) kexec nix-test (found [here](https://github.com/cleverca22/nix-tests/tree/master/kexec)).\n\nThe code is licensed under the [LGPL3](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FdasJ%2Femergency-kexec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FdasJ%2Femergency-kexec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FdasJ%2Femergency-kexec/lists"}