Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/jwillikers/home-lab-helm

Kubernetes YAML and Podman systemd configuration files for running my services at home
https://github.com/jwillikers/home-lab-helm

bitcoin-core caddy certbot eclipse-mosquitto esphome gitea homelab homelab-setup icinga jellyfin kubernetes miniflux minio nextcloud nginx omada-controller podman systemd vaultwarden

Last synced: 6 days ago
JSON representation

Kubernetes YAML and Podman systemd configuration files for running my services at home

Awesome Lists containing this project

README 

        
= Home Lab Helm

Jordan Williams 

:experimental:

:icons: font

:keywords: container helm k8s kubernetes linux podman systemd

ifdef::env-github[]

:tip-caption: :bulb:

:note-caption: :information_source:

:important-caption: :heavy_exclamation_mark:

:caution-caption: :fire:

:warning-caption: :warning:

endif::[]



Kubernetes YAML and https://podman.io/[Podman] systemd configuration files for running my services at home.



Everything is meant to be run rootless via Podman.

The configuration is intended for systems using https://selinuxproject.org/page/Main_Page[SELinux].

https://fedoraproject.org/iot/[Fedora IoT] is the current OS used for deployment.



https://caddyserver.com/[Caddy] is used on per-machine basis as a reverse-proxy to manage connections between the containers and everything external to the host.

Caddy configuration files for these services are stored in the https://codeberg.org/jwillikers/caddy-config[Caddy Config] repository.

Anything that allows using a choice of database uses https://www.postgresql.org/[PostgreSQL].



== Organization



Why use Kubernetes YAML?

Good question.

It's standard.

Kubernetes YAML isn't the equivalent of Docker Compose, but it accomplishes many of the same goals while being a common standard specification.

It's a bit trickier to learn, but it also supports _pods_, which are an incredibly powerful construct when orchestrating containers.

Plus, you can more easily scale up from Podman to full Kubernetes implementations.



As mentioned, this repository is meant to be run via Podman.

The reason for this is approach is to reduce the overhead and complexity required for a full Kubernetes deployment.

A minimalist approach like this makes a lot of sense for a self-hosted home lab which likely doesn't benefit as much from the redundancy and scalability provided by Kubernetes.

That said, I still kind of want those features and very well might migrate to a real Kubernetes deployment in the future.



To address scalability, this repository is organized in a modular fashion by service.

Since Podman is used instead of a true Kubernetes implementation, the distribution of services across servers must be handled manually.

Individual services can be deployed by symlinking the desired systemd files for the service to the appropriate directory to activate them.

This configuration typically requires a dedicated reverse-proxy on each device which proxies connections from the host to the internal Podman networks on which the services are running.

Configurations for the reverse-proxies are provided in a similar where individual configurations for each service can be activated individually.



What about templates?

Yeah, that comes back to minimalism and simplicity.

I haven't figured out how I want to go about centralizing the configuration so that host names and what not are managed across all of the various configuration files.

That's something that might just be addressed by moving things to a full Kubernetes configuration.

For now, things are simply just hard-coded, which makes the configuration really easy to read at least.



I use OVH to manage public DNS entries for my domain.

I don't make any services accessible to the public internet, which requires me to use the DNS-01 ACME challenge to automatically provision TLS certificates.

Plugins exist which automate this process for OVH.



To simplify configuration and make it easy to move services to different devices, the DNS entries for each service use CNAME records which point to the host device.

Two different domains are typically published for each service, one at `jwillikers.io` and another at `lan.jwillikers.io`.

The former is accessible via Tailscale from both inside and outside the LAN.

The latter is only available on the LAN.

TLS certificates are acquired for both domains, allowing the use of TLS to access services over Tailscale and the local network.



[TIP]

====

Don't mess with self-signed certificates.footnote:[_Everything_ will break.]

====



Tailscale is used to secure connections between individual devices while also allowing external connections to services.

This requires no port-forwarding at the router level and is free up to a lot of devices.



[TIP]

====

Disable key-expiry to reduce confusing troubleshooting sessions.

You can manually replace keys instead without surprises.

====



Storage is managed via persistent volume claims and host mounts.

The primary difference in how they are used is based primarily on whether or not the data for the volume should be backed up.

Persistent volume claims are used for ephemeral data that is _not_ backed up, which includes things like caches and less important persistent data.

Host mounts are used for mounting in configuration files provided by Git repositories, which are things under the `~/Projects` directory, and for important data that should be backed up, which resides under the `~/container-volumes` directory.



Both persistent volume claims and mounted volumes in the `~/container-volumes` directory follow a consistent naming scheme that mirrors the name of the Podman containers.

These volumes are named according to the pod and container to which they belong.

The naming scheme is `pod-container-name`, where dashes separate the different components.

As an example, the data directory for the vaultwarden container in the vaultwarden pod is named `vaultwarden-vaultwarden-data`.

The data directory for the PostgreSQL container in the pod is named `vaultwarden-postgresql-data`.

Pods with a single container may omit the container name for brevity.



For my backups, which use Btrbk, the `~/container-volumes` directory is a Btrfs subvolume.

This allows for fast, consistent snapshots which are easily backed up to other devices using incremental Btrfs send and receive commands.

Hourly backups are _extremely_ helpful.

In order to reduce storage consumption, only the volumes I find critical for restoration are mounted from the `~/container-volumes` directory.

For example, I don't mind losing the InfluxDB database, so that uses a persistent volume claim instead.

So, be sure to adjust the volumes according to your own needs.

My Btrbk backup configuration can be found in the https://codeberg.org/jwillikers/btrbk-config[btrbk Config] directory if you're interested.

Comprehensive backup strategies should probably incorporate database dumps and copies of the data stored in S3-compatible object storage like MinIO.



== Create a Dedicated User



These services are meant to be run under a user account dedicated to running rootless Podman containers.

This should be a system account and should use the same user id and group id across systems to simplify permissions.



. On rpm-ostree systems, the _systemd-journal_ group does not exist in the `/etc/group`, so it must be added.

+

[,sh]

----

echo $(getent group systemd-journal) | sudo tee -a /etc/group

----



. Create a `core` system group.

+

[,sh]

----

sudo groupadd --gid 818 --system core

----



. Create a primary user account named `core` for running containers with Podman.

+

--

[,sh]

----

sudo useradd \

  --add-subids-for-system \

  --btrfs-subvolume-home \

  --comment "Primary account for running containers" \

  --create-home \

  --gid core \

  --groups systemd-journal \

  --shell /usr/sbin/nologin \

  --system \

  --uid 818 \

  core

----



[TIP]

====

The `--root` flag can be used to create this user in the given chroot environment.

====

--



. Verify that the new `core` user has entries in `/etc/subuid` and `/etc/subgid`.

If for some reason, there are no subuid and subgid ranges for the user, follow these steps.

I don't know why this happens, but it does sometimes.

+

[NOTE]

====

These commands use the fish shell because I can never remember how to math in Bash.footnote:[Or anything else in Bash for that matter.]

====



.. Calculate the first value for the next subuid allotment.

+

--

If `/etc/subuid` is empty, use 100,000 as the initial value.



[,sh]

----

set new_subuid 100000

----



Otherwise, use the following function to calculate the next available subuid range.



[,sh]

----

set new_subuid (math (tail -1 /etc/subuid | awk -F ":" '{print $2}') + 65536)

----

--



.. Calculate the first value for the next subuid allotment.

+

--

If `/etc/subgid` is empty, use 100,000 as the initial value.



[,sh]

----

set new_subgid 100000

----



Otherwise, use the following function to calculate the next available subgid range.



[,sh]

----

set new_subgid (math (tail -1 /etc/subgid | awk -F ":" '{print $2}') + 65536)

----

--



.. Configure the `core` user with the calculated subuid and subgid ranges.

+

[,sh]

----

sudo usermod \

  --add-subuids $new_subuid-(math $new_subuid + 65535) \

  --add-subgids $new_subgid-(math $new_subgid + 65535) \

  core

----



. Automatically start the core user's session.

+

[,sh]

----

sudo loginctl enable-linger core

----



. Open a shell as the `core` user with the following command.

I prefer the fish shell, so I use that here, but substitute Bash, ZSH, etc. per your preference.

+

[,sh]

----

sudo -H -u core fish -c 'cd; fish'

----



. Configure the `XDG_RUNTIME_DIR` environment variable for the user in order for sockets to be found correctly.

+

[,sh]

----

set -Ux XDG_RUNTIME_DIR /run/user/(id -u)

----



. To get automatic updates, enable Podman's automatic update timer for the user.

+

[,sh]

----

systemctl --user enable --now podman-auto-update.timer

----



== Usage



. Open a shell as the system user account for running the containers.

+

[,sh]

----

sudo -H -u core fish -c 'cd; fish'

----



. Create the `~/Projects` directory.

+

[,sh]

----

mkdir ~/Projects

----



. Clone this repository to the `~/Projects` directory.

The configurations rely on this repository being at this location.

Sorry.

+

[,sh]

----

git -C ~/Projects clone https://codeberg.org/jwillikers/home-lab-helm.git

----



. Now follow the instructions in the _README.adoc_ files for the desired services.



== Services



* <>

* <>

* <>

* <>

* <>

* <>

* <>

* <>

* <>

* <>

* <>

* <>

* <>

* <>

* <>

* <>

* <>



== References



.Documentation

* https://docs.podman.io/en/latest/[Podman Documentation]

* https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html[podman-systemd.unit]

* https://www.freedesktop.org/software/systemd/man/latest/[systemd Manual Pages]

* https://github.com/containers/podman/blob/main/libpod/define/annotations.go[Podman Pod Annotations]

* https://github.com/containers/podman/blob/main/pkg/util/kube.go[Podman Volume Annotations]



== License



This project is licensed under the https://creativecommons.org/licenses/by-sa/4.0/legalcode[Creative Commons Attribution-ShareAlike 4.0 International License].



© 2023-2024 Jordan Williams



== Authors



mailto:{email}[{author}]