{"id":13558361,"url":"https://github.com/saschagrunert/kubernix","last_synced_at":"2025-05-14T22:07:46.100Z","repository":{"id":35145064,"uuid":"210179484","full_name":"saschagrunert/kubernix","owner":"saschagrunert","description":"Single dependency Kubernetes clusters for local testing, experimenting and development","archived":false,"fork":false,"pushed_at":"2023-01-25T11:05:38.000Z","size":4334,"stargazers_count":779,"open_issues_count":27,"forks_count":26,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-13T18:44:37.890Z","etag":null,"topics":["kubernetes","kubernetes-cluster","kubernetes-deployment","kubernetes-development","kubernetes-setup","nix","nix-shell","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saschagrunert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":["saschagrunert"]}},"created_at":"2019-09-22T16:33:58.000Z","updated_at":"2025-04-11T09:05:45.000Z","dependencies_parsed_at":"2023-02-14T06:50:13.660Z","dependency_job_id":null,"html_url":"https://github.com/saschagrunert/kubernix","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saschagrunert%2Fkubernix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saschagrunert%2Fkubernix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saschagrunert%2Fkubernix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saschagrunert%2Fkubernix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saschagrunert","download_url":"https://codeload.github.com/saschagrunert/kubernix/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254235696,"owners_count":22036963,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kubernetes","kubernetes-cluster","kubernetes-deployment","kubernetes-development","kubernetes-setup","nix","nix-shell","rust"],"created_at":"2024-08-01T12:04:54.644Z","updated_at":"2025-05-14T22:07:41.073Z","avatar_url":"https://github.com/saschagrunert.png","language":"Rust","funding_links":["https://github.com/sponsors/saschagrunert"],"categories":["Rust","Tools and Libraries","kubernetes","rust","Deployment Tools","\u003ca name=\"Rust\"\u003e\u003c/a\u003eRust"],"sub_categories":["Development Tools","Discovery"],"readme":"\u003cimg src=\".github/kubernix.png\" width=\"300px\"\u003e\u003c/img\u003e\n\n[![CircleCI](https://circleci.com/gh/saschagrunert/kubernix.svg?style=shield)](https://circleci.com/gh/saschagrunert/kubernix)\n[![Docs master](https://img.shields.io/badge/doc-master-orange.svg)](https://saschagrunert.github.io/kubernix/doc/kubernix/index.html)\n[![Docs release](https://docs.rs/kubernix/badge.svg)](https://docs.rs/kubernix)\n[![Coverage](https://codecov.io/gh/saschagrunert/kubernix/branch/master/graph/badge.svg)](https://codecov.io/gh/saschagrunert/kubernix)\n[![Dependencies](https://deps.rs/repo/github/saschagrunert/kubernix/status.svg)](https://deps.rs/repo/github/saschagrunert/kubernix)\n[![Crates.io](https://img.shields.io/crates/v/kubernix.svg)](https://crates.io/crates/kubernix)\n[![License MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/saschagrunert/kubernix/blob/master/LICENSE)\n\n## Kubernetes development cluster bootstrapping with Nix packages\n\nThis project aims to provide **single dependency** [Kubernetes][1] clusters\nfor local testing, experimenting and development purposes.\n\n[1]: https://kubernetes.io\n\nMoving pictures are worth more than thousand words, so here is a short demo:\n\n![demo](.github/kubernix.svg)\n\n### Nix?\n\nHave you ever heard about [Nix][2], the functional package manager?\n\nIn case you haven't, don’t worry – the important thing is that it provides all the third-party\ndependencies needed for this project, pinned to a dedicated version. This guarantees stable,\nreproducible installations.\n\n[2]: https://nixos.org/nix\n\nKuberNix itself is a Rusty helper program, which takes care of bootstrapping\nthe Kubernetes cluster, passing the right configuration parameters around and\nkeeping track of the running processes.\n\n### What is inside\n\nThe following technology stack is currently being used:\n\n| Application     | Version      |\n| --------------- | ------------ |\n| cfssl           | v1.5.0       |\n| cni-plugins     | v0.9.0       |\n| conmon          | v2.0.25      |\n| conntrack-tools | v1.4.6       |\n| cri-o-wrapper   | v1.20.0      |\n| cri-tools       | v1.20.0      |\n| etcd            | v3.3.25      |\n| iproute2        | v5.10.0      |\n| iptables        | v1.8.6       |\n| kmod            | v27          |\n| kubectl         | v1.19.5      |\n| kubernetes      | v1.19.5      |\n| nss-cacert      | v3.60        |\n| podman-wrapper  | v2.2.1       |\n| runc            | v1.0.0-rc92  |\n| socat           | v1.7.4.1     |\n| sysctl          | v1003.1.2008 |\n| util-linux      | v2.36.1      |\n\nSome other tools are not explicitly mentioned here, because they are no\nfirst-level dependencies.\n\n### Single Dependency\n\n#### With Nix\n\nAs already mentioned, there is only one single dependency needed to run this\nproject: **Nix**. To setup Nix, simply run:\n\n```shell\n$ curl https://nixos.org/nix/install | sh\n```\n\nPlease make sure to follow the instructions output by the script.\n\n#### With the Container Runtime of your Choice\n\nIt is also possible to run KuberNix in the container runtime of your choice. To\ndo this, simply grab the latest image from [`saschagrunert/kubernix`][40].\nPlease note that running KuberNix inside a container image requires to run\n`privileged` mode and `host` networking. For example, we can run KuberNix with\n[podman][41] like this:\n\n[40]: https://cloud.docker.com/u/saschagrunert/repository/docker/saschagrunert/kubernix\n[41]: https://github.com/containers/libpod\n\n```\n$ sudo podman run \\\n    --net=host \\\n    --privileged \\\n    -it docker.io/saschagrunert/kubernix:latest\n```\n\n### Getting Started\n\n#### Cluster Bootstrap\n\nTo bootstrap your first cluster, download one of the latest [release binaries][18] or\nbuild the application via:\n\n[18]: https://github.com/saschagrunert/kubernix/releases/latest\n\n```shell\n$ make build-release\n```\n\nThe binary should now be available in the `target/release/kubernix` directory of\nthe project. Alternatively, install the application via `cargo install kubernix`.\n\nAfter the successful binary retrieval, start KuberNix by running it as `root`:\n\n```\n$ sudo kubernix\n```\n\nKuberNix will now take care that the Nix environment gets correctly setup,\ndownloads the needed binaries and starts the cluster. Per default it will create\na directory called `kubernix-run` in the current path which contains all necessary\ndata for the cluster.\n\n#### Shell Environment\n\nIf everything went fine, you should be dropped into a new shell session,\nlike this:\n\n```\n[INFO ] Everything is up and running\n[INFO ] Spawning interactive shell\n[INFO ] Please be aware that the cluster stops if you exit the shell\n\u003e\n```\n\nNow you can access your cluster via tools like `kubectl`:\n\n```\n\u003e kubectl get pods --all-namespaces\nNAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE\nkube-system   coredns-85d84dd694-xz997   1/1     Running   0          102s\n```\n\nAll configuration files have been written to the target directory, which is now\nthe current one:\n\n```\n\u003e ls -1\napiserver/\ncontrollermanager/\ncoredns/\ncrio/\nencryptionconfig/\netcd/\nkubeconfig/\nkubelet/\nkubernix.env\nkubernix.toml\nnix/\npki/\npolicy.json\nproxy/\nscheduler/\n```\n\nFor example, the log files for the different running components are now\navailable within their corresponding directory:\n\n```\n\u003e ls -1 **.log\napiserver/kube-apiserver.log\ncontrollermanager/kube-controller-manager.log\ncrio/crio.log\netcd/etcd.log\nkubelet/kubelet.log\nproxy/kube-proxy.log\nscheduler/kube-scheduler.log\n```\n\nIf you want to spawn an additional shell session, simply run `kubernix shell` in\nthe same directory as where the initial bootstrap happened.\n\n```\n$ sudo kubernix shell\n[INFO  kubernix] Spawning new kubernix shell in 'kubernix-run'\n\u003e kubectl run --generator=run-pod/v1 --image=alpine -it alpine sh\nIf you don't see a command prompt, try pressing enter.\n/ #\n```\n\nThis means that you can spawn as many shells as you want to.\n\n#### Cleanup\n\nThe whole cluster gets automatically destroyed if you exit the shell session\nfrom the initial process:\n\n```\n\u003e exit\n[INFO ] Cleaning up\n```\n\nPlease note that the directory where all the data is stored is not being\nremoved after the exit of KuberNix. This means that you’re still able to\naccess the log and configuration files for further processing. If you start\nthe cluster again, then the cluster files will be reused. This is especially\nhandy if you want to test configuration changes.\n\n#### Restart\n\nIf you start KuberNix again in the same run directory, then it will re-use the\nconfiguration during the cluster bootstrapping process. This means that you\ncan modify all data inside the run root for testing and debugging purposes. The\nstartup of the individual components will be initiated by YAML files called\n`run.yml`, which are available inside the directories of the corresponding\ncomponents. For example, etc gets started via:\n\n```\n\u003e cat kubernix-run/etcd/run.yml\n```\n\n```yml\n---\ncommand: /nix/store/qlbsv0hvi0j5qj3631dzl9srl75finlk-etcd-3.3.13-bin/bin/etcd\nargs:\n  - \"--advertise-client-urls=https://127.0.0.1:2379\"\n  - \"--client-cert-auth\"\n  - \"--data-dir=/…/kubernix-run/etcd/run\"\n  - \"--initial-advertise-peer-urls=https://127.0.0.1:2380\"\n  - \"--initial-cluster-state=new\"\n  - \"--initial-cluster-token=etcd-cluster\"\n  - \"--initial-cluster=etcd=https://127.0.0.1:2380\"\n  - \"--listen-client-urls=https://127.0.0.1:2379\"\n  - \"--listen-peer-urls=https://127.0.0.1:2380\"\n  - \"--name=etcd\"\n  - \"--peer-client-cert-auth\"\n  - \"--cert-file=/…/kubernix-run/pki/kubernetes.pem\"\n  - \"--key-file=/…/kubernix-run/pki/kubernetes-key.pem\"\n  - \"--peer-cert-file=/…/kubernix-run/pki/kubernetes.pem\"\n  - \"--peer-key-file=/…/kubernix-run/pki/kubernetes-key.pem\"\n  - \"--peer-trusted-ca-file=/…/kubernix-run/pki/ca.pem\"\n  - \"--trusted-ca-file=/…/kubernix-run/pki/ca.pem\"\n```\n\n### Configuration\n\nKuberNix has some configuration possibilities, which are currently:\n\n| CLI argument              | Description                                                                         | Default        | Environment Variable         |\n| ------------------------- | ----------------------------------------------------------------------------------- | -------------- | ---------------------------- |\n| `-r, --root`              | Path where all the runtime data is stored                                           | `kubernix-run` | `KUBERNIX_ROOT`              |\n| `-l, --log-level`         | Logging verbosity                                                                   | `info`         | `KUBERNIX_LOG_LEVEL`         |\n| `-c, --cidr`              | CIDR used for the cluster network                                                   | `10.10.0.0/16` | `KUBERNIX_CIDR`              |\n| `-s, --shell`             | The shell executable to be used                                                     | `$SHELL`/`sh`  | `KUBERNIX_SHELL`             |\n| `-e, --no-shell`          | Do not spawn an interactive shell after bootstrap                                   | `false`        | `KUBERNIX_NO_SHELL`          |\n| `-n, --nodes`             | The number of nodes to be registered                                                | `1`            | `KUBERNIX_NODES`             |\n| `-u, --container-runtime` | The container runtime to be used for the nodes, irrelevant if `nodes` equals to `1` | `podman`       | `KUBERNIX_CONTAINER_RUNTIME` |\n| `-o, --overlay`           | Nix package overlay to be used                                                      |                | `KUBERNIX_OVERLAY`           |\n| `-p, --packages`          | Additional Nix dependencies to be added to the environment                          |                | `KUBERNIX_PACKAGES`          |\n\nPlease ensure that the CIDR is not overlapping with existing local networks and\nthat your setup has access to the internet. The CIDR will be automatically split\nup over the necessary cluster components.\n\n#### Multinode Support\n\nIt is possible to spawn multiple worker nodes, too. To do this, simply adjust\nthe `-n, --nodes` command line argument as well as your preferred container\nruntime via `-u, --container-runtime`. The default runtime is [podman][41],\nbut every other Docker drop-in replacement should work out of the box.\n\n#### Overlays\n\nOverlays provide a method to extend and change Nix derivations. This means, that\nwe’re able to change dependencies during the cluster bootstrapping process. For\nexample, we can exchange the used CRI-O version to use a local checkout by\nwriting this simple `overlay.nix`:\n\n```nix\nself: super: {\n  cri-o = super.cri-o.overrideAttrs(old: {\n    src = ../path/to/go/src/github.com/cri-o/cri-o;\n  });\n}\n```\n\nNow we can run KuberNix with the `--overlay, -o` command line argument:\n\n```\n$ sudo kubernix --overlay overlay.nix\n[INFO  kubernix] Nix environment not found, bootstrapping one\n[INFO  kubernix] Using custom overlay 'overlay.nix'\nthese derivations will be built:\n  /nix/store/9jb43i2mqjc94mbx30d9nrx529w6lngw-cri-o-1.15.2.drv\n  building '/nix/store/9jb43i2mqjc94mbx30d9nrx529w6lngw-cri-o-1.15.2.drv'...\n```\n\nUsing this technique makes it easy for daily development of Kubernetes\ncomponents, by simply changing it to local paths or trying out new versions.\n\n#### Additional Packages\n\nIt is also possible to add additional packages to the KuberNix environment by\nspecifying them via the `--packages, -p` command line parameter. This way you\ncan easily utilize additional tools in a reproducible way. For example, when to\ncomes to using always the same [Helm][20] version, you could simply run:\n\n```\n$ sudo kubernix -p kubernetes-helm\n[INFO ] Nix environment not found, bootstrapping one\n[INFO ] Bootstrapping cluster inside nix environment\n…\n\u003e helm init\n\u003e helm version\nClient: \u0026version.Version{SemVer:\"v2.14.3\", GitCommit:\"\", GitTreeState:\"clean\"}\nServer: \u0026version.Version{SemVer:\"v2.14.3\", GitCommit:\"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085\", GitTreeState:\"clean\"}\n```\n\nAll available packages are listed on the [official Nix index][21].\n\n[20]: https://helm.sh\n[21]: https://nixos.org/nixos/packages.html?channel=nixpkgs-unstable\n\n## Contributing\n\nYou want to contribute to this project? Wow, thanks! So please just fork it and\nsend me a pull request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaschagrunert%2Fkubernix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaschagrunert%2Fkubernix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaschagrunert%2Fkubernix/lists"}