Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/baiganio/k3s-pi5

Quick instructions on how to setup k3s on Raspberry Pi 5 and tunnel domain name with Cloudflare
https://github.com/baiganio/k3s-pi5

instructions k3s raspberry-pi

Last synced: 2 months ago
JSON representation

Quick instructions on how to setup k3s on Raspberry Pi 5 and tunnel domain name with Cloudflare

Awesome Lists containing this project

README 

          
# 👉 [https://baiganio.github.io/k3s-pi5](https://baiganio.github.io/k3s-pi5/)



### Step 3: Grant Non-Root Access to kubectl (Optional)



To avoid typing `sudo` every time:



```bash

# Create kubeconfig at user level

mkdir -p ~/.kube

sudo k3s kubectl config view --raw | sudo tee ~/.kube/config > /dev/null

sudo chown $(id -u):$(id -g) ~/.kube/config

sudo chmod 600 ~/.kube/config



# Now you can use kubectl without sudo

kubectl get nodes

```



### Step 4: Verify Docker/containerd



K3s uses containerd (not Docker) by default. Check it's working:



```bash

# List running containers

sudo crictl ps



# This shows all containers managed by k3s itself

# (e.g., coredns, metrics-server, local-path-provisioner)

```



**Why containerd instead of Docker:** Containerd is lighter and more suitable for resource-constrained environments.



---



## Cloudflare Tunnel Configuration



### What's Happening Here



You'll configure Cloudflare Tunnel (formerly Argo Tunnel) to:

1. Route external traffic from your Cloudflare domain → your Pi's k3s Nginx Ingress

2. Create a secure tunnel without exposing your home IP

3. Let you access apps like `app.yourdomain.com`



### Step 1: Install Cloudflare Tunnel Agent



```bash

# Download the latest cloudflared binary for ARM64

wget -q https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-arm64



# Make it executable

chmod +x cloudflared-linux-arm64



# Move to system path

sudo mv cloudflared-linux-arm64 /usr/local/bin/cloudflared



# Verify installation

cloudflared --version

```



### Step 2: Authenticate with Cloudflare



```bash

# This opens your browser to Cloudflare login

cloudflared tunnel login



# You'll:

# 1. Log in to your Cloudflare account

# 2. Select your domain

# 3. A cert file is saved locally



# Check it was saved

cat ~/.cloudflared/cert.pem

# Should show: -----BEGIN CERTIFICATE-----

```



### Step 3: Create a Tunnel



```bash

# Create a new tunnel (replace "my-pi" with a meaningful name)

cloudflared tunnel create my-pi



# Output:

# Tunnel credentials written to ~/.cloudflared/[tunnel-id].json

# Tunnel my-pi created with ID [tunnel-id]



# Save the tunnel ID for next steps

TUNNEL_ID=$(cloudflared tunnel list | grep my-pi | awk '{print $1}')

echo $TUNNEL_ID

```



### Step 4: Configure Tunnel Routing



Create a config file at `~/.cloudflared/config.yml`:



```bash

nano ~/.cloudflared/config.yml

```



Add this content (adjust domains as needed):



```yaml

tunnel: my-pi  # Match the tunnel name from Step 3

credentials-file: /home/pi/.cloudflared/[TUNNEL_ID].json

# Replace [TUNNEL_ID] with actual tunnel ID from previous step



ingress:

  # Route your domain to k3s Nginx Ingress (running on port 80)

  - hostname: yourdomain.com

    service: http://localhost/

  

  - hostname: "*.yourdomain.com"

    service: http://localhost/

  

  # Catch-all for any other traffic

  - service: http_status:404

```



**What this does:**

- `hostname: yourdomain.com` → Routes root domain to your k3s Ingress

- `"*.yourdomain.com"` → Routes all subdomains (app.yourdomain.com, api.yourdomain.com, etc.)

- `service: http_status:404` → Returns 404 for unmapped hostnames



### Step 5: Test Tunnel Connection



```bash

# Start tunnel manually to test

cloudflared tunnel run my-pi



# You should see:

# ... tunnel registered with Cloudflare

# ... routing rules active

# ... connected to edge



# Press Ctrl+C to stop (you'll create a systemd service next)

```



### Step 6: Create Systemd Service for Tunnel



```bash

# Create systemd unit file

sudo nano /etc/systemd/system/cloudflared.service

```



Add:



```ini

[Unit]

Description=Cloudflare Tunnel

After=network.target

Wants=network-online.target



[Service]

Type=simple

User=pi

WorkingDirectory=/home/pi/.cloudflared

ExecStart=/usr/local/bin/cloudflared tunnel run my-pi

Restart=on-failure

RestartSec=5s



[Install]

WantedBy=multi-user.target

```



```bash

# Enable and start the service

sudo systemctl daemon-reload

sudo systemctl enable cloudflared

sudo systemctl start cloudflared



# Check status

sudo systemctl status cloudflared



# View logs

sudo systemctl journalctl -u cloudflared -f

```



---



## Persistent Storage Setup



### What's Happening



For databases (PostgreSQL, MySQL) to survive pod restarts, we need persistent storage. K3s provides a built-in storage provisioner, but we'll also set up proper volumes.



### Step 1: Check Default Storage Class



K3s ships with a "local-path-provisioner" storage class:



```bash

# List storage classes

kubectl get storageclass



# Expected output:

# NAME                   PROVISIONER             RECLAIM POLICY

# local-path (default)   rancher.io/local-path   Delete

```



This stores data at `/var/lib/rancher/k3s/storage/` on the Pi.



### Step 2: Create a Data Directory for Databases



For better organization, create a dedicated directory:



```bash

# Create directory for persistent data

sudo mkdir -p /mnt/k3s-data

sudo mkdir -p /mnt/k3s-data/databases

sudo mkdir -p /mnt/k3s-data/applications



# Set permissions (allow k3s user to access)

sudo chown -R 1000:1000 /mnt/k3s-data

sudo chmod -R 755 /mnt/k3s-data

```



**Why:**

- Separates Kubernetes data from OS system files

- Easier to back up or expand storage later

- If needed, you can mount an external USB drive here



### Step 3: Create Persistent Volume (Manual Example)



For testing, we'll create a PersistentVolume manually:



```bash

# Create a manifest for persistent storage

nano pv-example.yaml

```



Add:



```yaml

apiVersion: v1

kind: PersistentVolume

metadata:

  name: database-pv

spec:

  capacity:

    storage: 5Gi

  accessModes:

    - ReadWriteOnce

  storageClassName: local-path

  hostPath:

    path: "/mnt/k3s-data/databases"

    type: Directory

---

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: database-pvc

  namespace: default

spec:

  accessModes:

    - ReadWriteOnce

  storageClassName: local-path

  resources:

    requests:

      storage: 5Gi

```



Apply it:



```bash

kubectl apply -f pv-example.yaml



# Verify

kubectl get pv

kubectl get pvc

```



**What this does:**

- **PersistentVolume (PV):** Allocates 5GB from `/mnt/k3s-data/databases`

- **PersistentVolumeClaim (PVC):** Applications claim storage through this



---



## Nginx Ingress Controller



### What's an Ingress?



An Ingress is a Kubernetes resource that:

- Routes external traffic to internal services

- Handles hostname-based routing (example.com → Service A, api.example.com → Service B)

- Provides load balancing and SSL termination



### Step 1: Check If Traefik is Already Installed



K3s comes with Traefik (an ingress controller) by default:



```bash

# Check Traefik pods

kubectl get pods -n kube-system | grep traefik



# Expected: traefik pod should be running

```



Traefik is simpler than Nginx for k3s. We'll use it, but if you prefer Nginx, see the Alternative section below.



### Step 2: Verify Traefik Service



```bash

# Check Traefik service

kubectl get svc -n kube-system traefik



# Expected output shows:

# TYPE: LoadBalancer

# EXTERNAL-IP: 192.168.x.x (your Pi's IP)

# PORT(S): 80:xxxxx/TCP, 443:xxxxx/TCP

```



This means Traefik is already listening on ports 80 and 443.



### Step 3: Test with a Simple Ingress



Create a test ingress:



```bash

nano test-ingress.yaml

```



Add:



```yaml

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: test-ingress

  annotations:

    # Traefik annotation (different from standard Nginx)

    traefik.ingress.kubernetes.io/router.entrypoints: web

spec:

  rules:

    - host: test.yourdomain.com

      http:

        paths:

          - path: /

            pathType: Prefix

            backend:

              service:

                name: nginx-test

                port:

                  number: 80

```



```bash

kubectl apply -f test-ingress.yaml

```



---



## K3s Dashboard



### What's the Dashboard?



A web UI to visualize:

- Pods, Services, Deployments

- Resource usage (CPU, memory)

- Logs from containers

- Deploy new applications



### Step 1: Install K3s Dashboard



```bash

# K3s doesn't include dashboard by default, install it:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml



# Wait for dashboard pod to start (takes ~30 seconds)

kubectl wait --for=condition=ready pod \

  -l k8s-app=kubernetes-dashboard \

  -n kubernetes-dashboard \

  --timeout=300s

```



### Step 2: Create Admin User and Get Access Token



```bash

# Create admin user

kubectl apply -f - <   nginx.yourdomain.com     192.168.x.x

# default     dotnet-hello              dotnet.yourdomain.com    192.168.x.x

```



### Step 3: Test DNS Resolution



```bash

# From your Pi, test if Cloudflare tunnel is routing correctly

curl -H "Host: nginx.yourdomain.com" http://localhost/



# Expected: HTML content from Nginx welcome page (or your dashboard)

```



### Step 4: Monitor Cloudflare Tunnel



```bash

# Check tunnel status

cloudflared tunnel list



# View recent connections

cloudflared tunnel info my-pi

```



### Step 5: Monitor Resource Usage



```bash

# Check node resource usage

kubectl top nodes



# Check pod resource usage

kubectl top pods -A



# Example output:

# NAME                    CPU(cores)   MEMORY(Mi)

# nginx-welcome-...       5m           32Mi

# dotnet-hello-...        10m          64Mi

# postgres-...            20m          128Mi

```



This shows how much CPU and memory each pod is using.



### Step 6: View Pod Logs



```bash

# View logs from specific pod

kubectl logs -l app=nginx



# Follow logs in real-time

kubectl logs -l app=nginx -f



# View logs from .NET app

kubectl logs -l app=dotnet-hello

```



---



## Troubleshooting



### Issue 1: K3s Pod Won't Start ("ImagePullBackOff")



**Symptom:** `kubectl get pods` shows pod stuck in "ImagePullBackOff"



**Cause:** Image doesn't exist for ARM64 architecture



**Solution:**

```bash

# Check what architecture the image supports

docker inspect image-name | grep -i "Architecture"



# Use only ARM64 compatible images. Safe choices:

# - alpine (all versions)

# - debian (all versions)

# - ubuntu:22.04

# - nginx:latest

# - postgres:15-alpine



# Do NOT use:

# - Images tagged "amd64" (Intel only)

# - Images tagged "windows" or "nanoserver"

```



### Issue 2: Pod Running But App Not Responding



**Symptom:** Pod is "Running" but website shows 503 or timeout



**Cause:** Service networking issue or app crashed inside container



**Solution:**

```bash

# 1. Check if pod is actually running or crashed

kubectl describe pod 



# 2. Check logs for errors

kubectl logs 



# 3. Test connectivity inside cluster

kubectl exec -it  -- /bin/sh

# Then inside container: curl localhost:80



# 4. Check service

kubectl get svc 

# Verify CLUSTER-IP is assigned and not 

```



### Issue 3: Persistent Storage Not Working



**Symptom:** Database loses data after pod restart



**Cause:** PVC not bound or wrong path



**Solution:**

```bash

# Check PVC status

kubectl get pvc



# Should show:

# NAME           STATUS   VOLUME   ...

# database-pvc   Bound    pv-1     ...



# If status is "Pending":

kubectl describe pvc database-pvc



# Check PV exists

kubectl get pv

```



### Issue 4: Cloudflare Tunnel Not Routing Traffic



**Symptom:** Website shows "Error 522" or "Origin unreachable"



**Cause:** Tunnel not connected or config incorrect



**Solution:**

```bash

# Check tunnel is connected

sudo systemctl status cloudflared



# Check tunnel logs

sudo journalctl -u cloudflared -f



# Verify config has correct hostnames and ports

cat ~/.cloudflared/config.yml



# Test locally without tunnel:

curl -H "Host: nginx.yourdomain.com" http://localhost/

```



### Issue 5: High Memory Usage / Pod Eviction



**Symptom:** Pods getting killed or "MemoryLimitExceeded"



**Cause:** Pods using more memory than limit



**Solution:**

```bash

# Check current resource usage

kubectl top pods -A



# Increase limits in YAML:

# resources:

#   limits:

#     memory: "256Mi"   # Increase from original



# Or reduce replicas:

kubectl scale deployment nginx-welcome --replicas=1

```



---



## Next Steps After Testing



1. **Add More Apps:** Create Deployments for your Node.js apps using similar patterns

2. **Enable HTTPS:** Add SSL certificate with cert-manager

3. **Backups:** Set up backup strategy for persistent storage

4. **Monitoring:** Install Prometheus and Grafana for metrics

5. **CI/CD:** Set up automated deployments with GitHub Actions



---



## Quick Reference Commands



```bash

# Pod management

kubectl get pods                           # List pods

kubectl describe pod                 # Detailed info

kubectl logs                          # View logs

kubectl exec -it  -- /bin/sh          # Shell access

kubectl delete pod                   # Remove pod



# Deployment management

kubectl get deployments                    # List deployments

kubectl scale deployment  --replicas=3  # Scale up/down

kubectl set image deployment/ app=image:tag  # Update image



# Service and Ingress

kubectl get svc                            # List services

kubectl get ingress                        # List ingress rules



# Resource monitoring

kubectl top nodes                          # Node CPU/memory

kubectl top pods                           # Pod CPU/memory



# Debugging

kubectl get events                         # Recent cluster events

kubectl describe node                      # Node details

```



---



## Important Security Notes



⚠️ **For Production:**

- Change all default passwords (postgres, dashboard token)

- Enable RBAC properly (current setup is admin-only)

- Use sealed secrets for credentials (not plain ConfigMap)

- Enable network policies to restrict traffic between pods

- Regularly update k3s: `sudo systemctl stop k3s && sudo systemctl start k3s`

- Back up your persistent data regularly



---



**Setup Complete!** Your k3s cluster on Raspberry Pi 5 is now ready for testing. Start with the Nginx deployment, verify it works through Cloudflare, then proceed to .NET and database applications.