{"id":16334766,"url":"https://github.com/berbiche/infrastructure","last_synced_at":"2025-03-22T23:32:11.703Z","repository":{"id":41555867,"uuid":"336473345","full_name":"berbiche/infrastructure","owner":"berbiche","description":"Monorepo for my new^2 homelab","archived":false,"fork":false,"pushed_at":"2024-04-13T09:07:37.000Z","size":1274,"stargazers_count":9,"open_issues_count":8,"forks_count":1,"subscribers_count":2,"default_branch":"cluster-v2","last_synced_at":"2024-04-13T21:54:33.629Z","etag":null,"topics":["deploy-rs","kubernetes-cluster","nixos","okd-4","terraform"],"latest_commit_sha":null,"homepage":"","language":"YAML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/berbiche.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-02-06T06:37:33.000Z","updated_at":"2024-04-15T01:48:46.782Z","dependencies_parsed_at":"2024-01-08T04:53:52.130Z","dependency_job_id":"54acd626-06ef-4b79-8b65-71a7eb2e3a33","html_url":"https://github.com/berbiche/infrastructure","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/berbiche%2Finfrastructure","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/berbiche%2Finfrastructure/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/berbiche%2Finfrastructure/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/berbiche%2Finfrastructure/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/berbiche","download_url":"https://codeload.github.com/berbiche/infrastructure/tar.gz/refs/heads/cluster-v2","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245036112,"owners_count":20550662,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deploy-rs","kubernetes-cluster","nixos","okd-4","terraform"],"created_at":"2024-10-10T23:39:12.672Z","updated_at":"2025-03-22T23:32:11.279Z","avatar_url":"https://github.com/berbiche.png","language":"YAML","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n# My new infrastructure repository\n\nIn this repository resides the configuration for my new homelab\nand some of my VPS hosts.\n\nTechnologies used:\n\n-   [NixOS](https://nixos.org) for the declarative OS and service configuration\n-   [deploy-rs](https://github.com/serokel/deploy-rs) to deploy my NixOS configuration\n-   [Terraform](https://terraform.io) for the automated DNS and OVH configuration\n-   [Backblaze](https://www.backblaze.com) for the cheap object storage\n-   [Ansible](https://github.com/ansible/ansible) for imperative configuration management with tasks, roles, etc.\n-   [OKD](https://okd.io) is an OpenShift (Kubernetes) distribution without license requirements\n-   [Ovirt](https://ovirt.org) and Ovirt Node is used on my single server to host my Kubernetes deployment\n\nTODO: \u003chttps://plantuml.com/nwdiag\u003e\n\n\n## Table of contents\n\n1.  [Table of contents](#table-of-contents)\n2.  [Initial configuration](#initial-configuration)\n    1.  [Configuring the VPS](#configuring-the-vps)\n    2.  [Configuring Terraform](#configuring-terraform)\n3.  [SOPS Cheatsheet](#sops-cheatsheet)\n4.  [Terraform](#terraform)\n5.  [Backblaze](#backblaze)\n6.  [Kubernetes](#kubernetes)\n    1.  [Deploying oVirt Node](#orga6e25f0)\n    2.  [Installing and configuring GlusterFS](#orgea838f4)\n    3.  [Deploying the HostedEngine VM](#orgef118a7)\n    4.  [Configuring a new user for OCP](#org54f82c0)\n    5.  [Deploying OKD](#org33bda34)\n    6.  [Stuff to look at](#orgcee9d46)\n    7.  [Applications](#applications)\n7.  [NixOS](#nixos)\n    1.  [Deployment](#deployment)\n    2.  [Modules](#modules)\n\n\n## Initial configuration\n\n\n### Configuring the VPS\n\n-   Create a VPS at `$HOSTING_PROVIDER`\n-   `nixos-infect` the VPS\n\n\n### Configuring Terraform\n\nEnter the shell with `nix run .#terraform-fhs` to get access to all the required\nvariables and to be able to use the `terraform-provider-b2`.\n\n-   Get an API token from OVH and update [secrets/ovh.yaml](./secrets/ovh.yaml)\n-   Get an API token from Cloudflare and update [secrets/cloudflare.yaml](./secrets/cloudflare.yaml)\n-   Get an API token from Backblaze and update [secrets/backblaze.yaml](./secrets/backblaze.yaml)\n-   Create a Backblaze bucket and application key for that bucket for the\n    Terraform state and update [secrets/terraform-backend.yaml](./secrets/terraform-backend.yaml)\n\nAll required secrets keys are public in the appropriate SOPS file in\n`secrets/` (but not their values).\n\n-   `cd terraform \u0026\u0026 terraform init`\n\n\n## SOPS Cheatsheet\n\n    $ sops -i secrets/cloudflare.yaml\n    edit stuff in $EDITOR\n    :wq\n    \n    File is encrypted inline\n\n    $ sops exec-env secrets/some-file bash\n    bash-4.4$\n\n\n## Terraform\n\nThe Terraform state is managed outside of the repository in a B2 bucket.\n\nTerraform needs to be run from the FHS provided in the flake default\npackage because the Backblaze B2 provider extracts a binary embedded in\nits binary and the paths needs to be =patchelf=d.\n\n`nix-shell` will spawn you in the FHS with the required packages for the\nTerraform B2 plugin to work.\n\n\n## Backblaze\n\nDocumentation about capabilities:\n\u003chttps://www.backblaze.com/b2/docs/application_keys.html\u003e\n\nRetention settings for the dovecot email bucket: 30 days\n\n\n## Kubernetes\n\n\n\n### Deploying oVirt Node\n\nSince I have a single server with 56 cores (vCPU) and 200GB of RAM,\nI deploy my kubernetes cluster using OpenShift\u0026rsquo;s free OKD distribution\non a single oVirt Node host.\n\nIn order to deploy OKD, the oVirt node has to be setup alongside a storage\ndevice.\nAfterwards, HostedEngine can be deployed in oVirt.\n\n\n### Installing and configuring GlusterFS\n\n1.  Make sure to generate an ssh private key (with no password) for the root user.\n    1.  Then, as the root user on the ovirt node, ssh on the ovirt node (`ssh root@localhost`)\n        This will add the ovirt node to the ssh `known_hosts` file.\n2.  Create a raid device\n3.  Setup glusterfs through Cockpit\u0026rsquo;s UI to use the raid device\n    1.  Add a brick for the kubernetes cluster with at least 500GB of space.\n\n\n### Deploying the HostedEngine VM\n\nHostedEngine needs to be deployed and should use the configured glusterfs storage.\n\n1.  Deploy a hyperconverged HostedEngine VM through Cockpit\u0026rsquo;s UI\n\n\n### Configuring a new user for OCP\n\n1.  Add a new user for the OCP deployment\n    1.  Go to the Keycloak admin interface\n    2.  Create a user named `kubernetes@ovirt`\n    3.  Set this user\u0026rsquo;s password and save it\n    4.  Update the password secret in the \\`install-config.yaml\\` configuration\n    5.  Connect as the `kubernetes@ovirt` user on the Ovirt Portal\n    6.  Add the following permissions to the `kubernetes@ovirt` user under `Administration \u003e Users`:\n        -   `ClusterAdmin`\n        -   `DiskCreator`\n        -   `DiskOperator`\n        -   `TemplateCreator`\n        -   `TemplateOwner`\n        -   `UserTemplateBasedVm`\n\n\n### Deploying OKD\n\nThe steps to deploy OKD are the following:\n\n1.  Configure a DHCP server to allocate IP addresses for the nodes\n2.  Configure DNS entries\n3.  Install the `openshift-install` CLI\n4.  Generate the install configuration and manifests\n5.  Patch the generated manifests\n6.  Create the cluster with the patched manifests\n\n1.  Configuring OKD dns entries\n\n    OKD requires the following DNS entries during the bootstrap phase and after.\n    \n    -   `*.apps.{cluster}.baseDomain`: points to the Haproxy LoadBalancer IP\n    -   `api-internal.{cluster}.baseDomain`: points to a virtual IP for the API server\n    -   `api.{cluster}.baseDomain`: ditto\n\n2.  Installing the `openshift` deployment CLI\n\n    In order to deploy OKD, the `openshift-install` cli will need to be fetched from the official repository and unpacked.\n    \n    The cli is also available as a Nix derivation in the `Flake.nix`.\n    It is automatically available when using direnv.\n\n3.  Generating the install configuration and manifests\n\n    1.  Generate the base configuration with `openshift-install create configs --dir .`\n    2.  Fetch the latest Tigera manifests from [here](https://projectcalico.docs.tigera.io/getting-started/openshift/installation) and add them to a folder named `calico`.\n        \n        The script below provides an automated way of creating the `kustomization.yaml` for the Calico/Tigera manifests.\n        \n            $ mkdir -p calico\n            # Copy the block of code and run it through this\n            $ wl-paste | awk 'gsub(/manifests/, \"calico\", $4)' \u003e script.sh\n            $ cat script.sh\n            $ bash script.sh\n            $ resources=\"$(find calico -type f -printf '%f\\0' | sort -z | xargs -r0 printf '- ./%s\\n')\"\n            $ cat \u003c\u003cEOF \u003ecalico/kustomization.yaml\n            apiVersion: kustomize.config.k8s.io/v1beta1\n            kind: Kustomization\n            \n            commonAnnotations:\n              qt.rs/installer-dir: manifests\n            \n            resources:\n            $resources\n            EOF\n    3.  Generate the openshift manifests with `openshift-install create manifests --dir .`\n        This may consume the `openshift-install.yaml` file.\n    4.  Generate a `kustomization.yaml` file for the manifests in `manifests` and `openshift`\n\n4.  Creating the cluster\n\n    1.  Generate the final resources\n        \n            $ mkdir -p bootstrap/install-dir\n            $ kustomize build --enable-alpha-plugins bootstrap | ./slice.py -o bootstrap/install-dir\n        \n        Make sure the file `manifests/cluster-config.yaml` exists.\n    \n    2.  Begin the installation\n        \n        Make sure to delete the file `install-config.yaml` in the installation directory\n        or to move it out of the `install-dir` folder.\n        \n        The hidden file `.openshift_install_state.json` ****MUST**** exist\n        otherwise the installer will not use ANY generated manifests.\n        \n        The installation directory should look like this:\n        \n            install-dir\n            ├── .openshift_install_state.json\n            ├── manifests\n            │   ├── 00-namespace-tigera-operator.yaml\n            │   ├── 01-cr-apiserver.yaml\n            │   ├── 01-crd-apiserver.yaml\n            │   ├── 01-crd-imageset.yaml\n            │   ├── 01-crd-installation.yaml\n            │   ├── 01-crd-tigerastatus.yaml\n            │   ├── 01-cr-installation.yaml\n            │   ├── 02-configmap-calico-resources.yaml\n            │   ├── 02-rolebinding-tigera-operator.yaml\n            │   ├── 02-role-tigera-operator.yaml\n            │   ├── 02-serviceaccount-tigera-operator.yaml\n            │   ├── 02-tigera-operator.yaml\n            │   ├── 04-openshift-machine-config-operator.yaml\n            │   ├── cluster-config.yaml\n            │   ├── cluster-dns-02-config.yml\n            │   ├── cluster-infrastructure-02-config.yml\n            │   ├── cluster-ingress-02-config.yml\n            │   ├── cluster-network-01-crd.yml\n            │   ├── cluster-network-02-config.yml\n            │   ├── cluster-proxy-01-config.yaml\n            │   ├── cluster-scheduler-02-config.yml\n            │   ├── configmap-root-ca.yaml\n            │   ├── crd.projectcalico.org_bgpconfigurations.yaml\n            │   ├── crd.projectcalico.org_bgppeers.yaml\n            │   ├── crd.projectcalico.org_blockaffinities.yaml\n            │   ├── crd.projectcalico.org_caliconodestatuses.yaml\n            │   ├── crd.projectcalico.org_clusterinformations.yaml\n            │   ├── crd.projectcalico.org_felixconfigurations.yaml\n            │   ├── crd.projectcalico.org_globalnetworkpolicies.yaml\n            │   ├── crd.projectcalico.org_globalnetworksets.yaml\n            │   ├── crd.projectcalico.org_hostendpoints.yaml\n            │   ├── crd.projectcalico.org_ipamblocks.yaml\n            │   ├── crd.projectcalico.org_ipamconfigs.yaml\n            │   ├── crd.projectcalico.org_ipamhandles.yaml\n            │   ├── crd.projectcalico.org_ippools.yaml\n            │   ├── crd.projectcalico.org_ipreservations.yaml\n            │   ├── crd.projectcalico.org_kubecontrollersconfigurations.yaml\n            │   ├── crd.projectcalico.org_networkpolicies.yaml\n            │   ├── crd.projectcalico.org_networksets.yaml\n            │   ├── cvo-overrides.yaml\n            │   ├── kube-cloud-config.yaml\n            │   ├── openshift-kubevirt-infra-namespace.yaml\n            │   ├── secret-machine-config-server-tls.yaml\n            │   └── secret-pull-secret.yaml\n            └── openshift\n                ├── 99_openshift-cluster-api_master-machines-0.yaml\n                ├── 99_openshift-cluster-api_master-machines-1.yaml\n                ├── 99_openshift-cluster-api_master-machines-2.yaml\n                ├── 99_openshift-cluster-api_worker-machineset-0.yaml\n                ├── 99_openshift-machineconfig_99-master-ssh.yaml\n                ├── 99_openshift-machineconfig_99-worker-ssh.yaml\n                ├── 99_role-cloud-creds-secret-reader.yaml\n                ├── openshift-install-manifests.yaml\n                ├── secret-kubeadmin.yaml\n                ├── secret-master-user-data.yaml\n                ├── secret-ovirt-credentials.yaml\n                └── secret-worker-user-data.yaml\n        \n            $ openshift-install create cluster --dir install-dir --log-level=debug\n            DEBUG .....\n            INFO Consuming Install Config from target directory\n\n\n### Stuff to look at\n\n-   [Customizing HAProxy error code response pages](https://docs.openshift.com/container-platform/4.9/networking/ingress-operator.html#nw-customize-ingress-error-pages_configuring-ingress)\n-   [Enabling HTTP Strict Transport Security per-route](https://docs.openshift.com/container-platform/4.9/networking/routes/route-configuration.html#nw-enabling-hsts-per-route_route-configuration)\n-   [Creating a route through an Ingress object](https://docs.openshift.com/container-platform/4.9/networking/routes/route-configuration.html#nw-ingress-creating-a-route-via-an-ingress_route-configuration)\n-   [Installing a specific version of an Operator](https://docs.openshift.com/container-platform/4.9/operators/admin/olm-adding-operators-to-cluster.html#olm-installing-specific-version-cli_olm-adding-operators-to-a-cluster)\n\n\n### Applications\n\nThis requires the [kustomize-sops plugin](https://github.com/viaduct-ai/kustomize-sops).\nThis plugin is automatically exposed in the flake shell.\n\nTo encrypt a secret: `sops -i -e k8s/something/overlays/prod/secrets/some-secret` for instance.\n\n1.  Technologies\n\n    -   Kustomize to scaffold, modify and apply patches on top of external resources\n    -   Cert-manager to manage certificates\n    -   metallb to manage ip allocation in the cluster\n    -   calico as the CNI\n    -   ExternalDNS to expose DNS records to Cloudflare\n    -   OKD provides the monitoring stack and a CSI driver with the oVirt deployment\n\n2.  Kustomize\n\n        kustomize build --enable-alpha-plugins something/overlays/prod\n        kustomize build --enable-alpha-plugins something/overlays/prod | kubectl apply -f -\n\n3.  Deployment\n\n    1.  Deploy metallb\n    2.  Deploy external-dns\n    3.  Deploy cert-manager\n    4.  Deploy Traefik (has a dependency on Cert Manager and MetalLB)\n    5.  Deploy CSI\n    6.  Deploy the remaining resources\n\n4.  ExternalDNS\n\n    ExternalDNS automatically inserts CNAME entries pointing to `k8s.qt.rs` for each ingress defined\n    and annotated with `external-dns.alpha.kubernetes.io/target: k8s.qt.rs`.\n    \n    While I could use a generic `*` CNAME entry that points to `k8s.qt.rs`, I prefer having\n    unresolvable domains.\n    Also, when my ISP will properly support IPv6, I will add `AAAA` records and `CNAME` records for the IPv4 scenario.\n    \n    Domains that only allow access by administrators (myself) are gated behind an OAuth middleware in Traefik.\n\n5.  CSI\n\n    \n    1.  Setup\n    \n        Follow the democratic-csi documentation here: \u003chttps://github.com/democratic-csi/democratic-csi\u003e\n        \n        TL;DR:\n        \n        1.  Get an API key for a user with enough privileges (root for instance)\n        2.  Configure iSCSI in TrueNAS interface\n        3.  Set iSCSI authentication method to CHAP or Mutual-CHAP\n        4.  Set the username/password combination used in the previous step in the `node-stage-secret.yaml` file\n        5.  Set the API key from the first step in the `driver-config-iscsi.yaml` file\n        6.  Deploy\n            \n                $ kubectl kustomize --enable-alpha-plugins ./overlays/prod | kubectl create -f- --save-config\n\n\n## NixOS\n\n\n### Deployment\n\nUsing deploy-rs, `deploy .#mouse --auto-rollback=false` for instance.\n\n\n### Modules\n\nI host different services on my NixOS VMs.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fberbiche%2Finfrastructure","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fberbiche%2Finfrastructure","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fberbiche%2Finfrastructure/lists"}