{"id":43161544,"url":"https://github.com/mitchross/k3s-argocd-starter","last_synced_at":"2026-02-01T01:21:10.809Z","repository":{"id":273298483,"uuid":"882835722","full_name":"mitchross/k3s-argocd-starter","owner":"mitchross","description":"starter kit for k3s, argocd","archived":false,"fork":false,"pushed_at":"2026-01-27T09:50:48.000Z","size":1573,"stargazers_count":60,"open_issues_count":10,"forks_count":28,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-01-27T22:29:01.107Z","etag":null,"topics":["argocd","cloudflare","k8s","k8s-at-home","kuberentes"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mitchross.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-03T21:55:30.000Z","updated_at":"2026-01-22T22:40:47.000Z","dependencies_parsed_at":"2026-01-27T12:03:45.589Z","dependency_job_id":null,"html_url":"https://github.com/mitchross/k3s-argocd-starter","commit_stats":null,"previous_names":["mitchross/k3s-argocd-starter"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mitchross/k3s-argocd-starter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitchross%2Fk3s-argocd-starter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitchross%2Fk3s-argocd-starter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitchross%2Fk3s-argocd-starter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitchross%2Fk3s-argocd-starter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mitchross","download_url":"https://codeload.github.com/mitchross/k3s-argocd-starter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitchross%2Fk3s-argocd-starter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28963628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T00:42:38.011Z","status":"ssl_error","status_checked_at":"2026-02-01T00:42:35.920Z","response_time":128,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["argocd","cloudflare","k8s","k8s-at-home","kuberentes"],"created_at":"2026-02-01T01:21:06.792Z","updated_at":"2026-02-01T01:21:10.800Z","avatar_url":"https://github.com/mitchross.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"🚀 Kubernetes Starter Kit\n=========================\n\nTutorial Video\n\n[![Tutorial Walkthrough Video](https://img.youtube.com/vi/AY5mC5rDUcw/0.jpg)](https://youtu.be/AY5mC5rDUcw)\n\n\n\n\u003e Modern GitOps deployment structure using Argo CD on Kubernetes\n\nThis starter kit provides a production-ready foundation for deploying applications and infrastructure components using GitOps principles. Compatible with both Raspberry Pi and x86 systems.\n\n## 📋 Table of Contents\n\n- [Prerequisites](#-prerequisites)\n- [Architecture](#-architecture)\n- [Quick Start](#-quick-start)\n  - [System Setup](#1-system-setup)\n  - [K3s Installation](#2-k3s-installation)\n  - [Networking Setup](#3-networking-setup-cilium)\n  - [GitOps Setup](#4-gitops-setup-argo-cd-part-1-of-2)\n- [Security Setup](#-security-setup)\n  - [Cloudflare Integration](#cloudflare-integration)\n- [Verification](#-verification)\n- [Applications](#-included-applications)\n- [Contributing](#-contributing)\n- [License](#-license)\n- [Troubleshooting](#-troubleshooting)\n\n## 📋 Prerequisites\n\n- Kubernetes cluster (tested with K3s v1.32.0+k3s1)\n- Linux host (ARM or x86) with:\n  - Storage support (OpenEBS works with ZFS or standard directories)\n  - NFS and CIFS support (optional)\n  - Open-iSCSI\n- Cloudflare account (for DNS and Tunnel)\n- Local DNS setup (one of the following):\n  - Local DNS server ([AdGuard Home setup guide](docs/adguard-home-setup.md))\n  - Router with custom DNS capabilities (e.g., Firewalla)\n  - Ability to modify hosts file on all devices\n\n## 🏗️ Architecture\n\n```mermaid\ngraph TD\n    subgraph \"Argo CD Projects\"\n        IP[Infrastructure Project] --\u003e IAS[Infrastructure ApplicationSet]\n        AP[Applications Project] --\u003e AAS[Applications ApplicationSet]\n        MP[Monitoring Project] --\u003e MAS[Monitoring ApplicationSet]\n    end\n    \n    subgraph \"Infrastructure Components\"\n        IAS --\u003e N[Networking]\n        IAS --\u003e S[Storage]\n        IAS --\u003e C[Controllers]\n        \n        N --\u003e Cilium\n        N --\u003e Cloudflared\n        N --\u003e Gateway\n        \n        S --\u003e OpenEBS\n        \n        C --\u003e CertManager\n    end\n    \n    subgraph \"Monitoring Stack\"\n        MAS --\u003e Prometheus\n        MAS --\u003e Grafana\n        MAS --\u003e AlertManager\n        MAS --\u003e NodeExporter\n        MAS --\u003e Loki\n        MAS --\u003e Promtail\n    end\n    \n    subgraph \"User Applications\"\n        AAS --\u003e P[Privacy Apps]\n        AAS --\u003e Web[Web Apps]\n        AAS --\u003e Other[Other Apps]\n        \n        P --\u003e ProxiTok\n        P --\u003e SearXNG\n        P --\u003e LibReddit\n        \n        Web --\u003e Nginx\n        Web --\u003e Dashboard\n        \n        Other --\u003e HelloWorld\n    end\n\n    style IP fill:#f9f,stroke:#333,stroke-width:2px\n    style AP fill:#f9f,stroke:#333,stroke-width:2px\n    style MP fill:#f9f,stroke:#333,stroke-width:2px\n    style IAS fill:#bbf,stroke:#333,stroke-width:2px\n    style AAS fill:#bbf,stroke:#333,stroke-width:2px\n    style MAS fill:#bbf,stroke:#333,stroke-width:2px\n```\n\n### Key Features\n- **GitOps Structure**: Two-level Argo CD ApplicationSets for infrastructure/apps\n- **Security Boundaries**: Separate projects with RBAC enforcement\n- **Sync Waves**: Infrastructure deploys first (negative sync waves)\n- **Self-Healing**: Automated sync with pruning and failure recovery\n\n## 🚀 Quick Start\n\n### 1. System Setup\n```bash\n# Essential packages (ZFS/NFS/iSCSI)\nsudo apt update \u0026\u0026 sudo apt install -y \\\n  zfsutils-linux \\\n  nfs-kernel-server \\\n  cifs-utils \\\n  open-iscsi  # Optional but recommended\n\n# Critical kernel modules for Cilium\nsudo modprobe iptable_raw xt_socket\necho -e \"xt_socket\\niptable_raw\" | sudo tee /etc/modules-load.d/cilium.conf\n```\n\n### 2. K3s Installation\n```bash\n# Customize these values!\nexport SETUP_NODEIP=192.168.101.176  # Your node IP\nexport SETUP_CLUSTERTOKEN=randomtokensecret12343  # Strong token\n\ncurl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=\"v1.33.3+k3s1\" \\\n  INSTALL_K3S_EXEC=\"--node-ip $SETUP_NODEIP \\\n  --disable=flannel,local-storage,metrics-server,servicelb,traefik \\\n  --flannel-backend='none' \\\n  --disable-network-policy \\\n  --disable-cloud-controller \\\n  --disable-kube-proxy\" \\\n  K3S_TOKEN=$SETUP_CLUSTERTOKEN \\\n  K3S_KUBECONFIG_MODE=644 sh -s -\n\n# Configure kubectl access\nmkdir -p $HOME/.kube \u0026\u0026 sudo cp -i /etc/rancher/k3s/k3s.yaml $HOME/.kube/config\nsudo chown $(id -u):$(id -g) $HOME/.kube/config \u0026\u0026 chmod 600 $HOME/.kube/config\n```\n\n### Adding Additional Nodes\n```bash\n# On the MASTER node - Get the token if you forgot it\nsudo cat /var/lib/rancher/k3s/server/node-token\n\n# On each WORKER node:\nexport MASTER_IP=192.168.101.202  # IP of your master node\nexport NODE_IP=192.168.101.203    # IP of THIS worker node\nexport K3S_TOKEN=your-node-token # From master's node-token file\n\ncurl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=\"v1.33.3+k3s1\" \\\n  K3S_URL=\"https://$MASTER_IP:6443\" \\\n  K3S_TOKEN=$K3S_TOKEN \\\n  INSTALL_K3S_EXEC=\"--node-ip $NODE_IP\" sh -\n\n# On the MASTER node - Verify the new node joined\nkubectl get nodes -o wide\n```\n\n### Setting Up Lens (Optional but Recommended)\n\n1. Install Lens from https://k8slens.dev/\n2. Get the kubeconfig:\n   - Copy from `/etc/rancher/k3s/k3s.yaml`, or\n   - Run: `kubectl config view --raw \u003e kubeconfig.yaml`\n3. When adding to Lens:\n   - Replace the server URL with your K3s node IP\n   - Example: `server: https://192.168.10.202:6443`\n4. Save and connect\n\n### 3. Networking Setup (Cilium)\n```bash\n# Install Cilium CLI\n## CHECK ARCH FIRST\nCILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)\nCLI_ARCH=amd64 \u0026\u0026 [ \"$(uname -m)\" = \"aarch64\" ] \u0026\u0026 CLI_ARCH=arm64\ncurl -L --fail --remote-name-all \\\n  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}\nsha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum\nsudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin\nrm cilium-linux-${CLI_ARCH}.tar.gz*\n\n# Helm install Cilium\n# use  helm install if first time, or helm upgrade if trying to update/upgrade/redo something \nhelm repo add cilium https://helm.cilium.io \u0026\u0026 helm repo update\nhelm install cilium cilium/cilium -n kube-system \\\n  -f infrastructure/networking/cilium/values.yaml \\\n  --version 1.18.0 \\\n  --set operator.replicas=1\n\n# Validate installation\ncilium status \u0026\u0026 cilium connectivity test\n\n# Critical L2 Configuration Note:\n# Before applying the CiliumL2AnnouncementPolicy, you MUST identify your correct network interface:\n\n# 1. List all network interfaces:\nip a\n\n# 2. Look for your main interface with an IP address matching your network\n# Common interface names:\n# - Ubuntu/Debian: enp1s0, ens18, eth0\n# - macOS: en0\n# - RPi: eth0\n# The interface should show your node's IP address, for example:\n#   enp1s0: \u003cBROADCAST,MULTICAST,UP,LOWER_UP\u003e ... inet 192.168.1.100/24\n\n# 3. Make note of your interface name for the CiliumL2AnnouncementPolicy\n# You'll need this when applying the infrastructure components via Argo CD\n\n# DO NOT apply the policy here - it will be applied through Argo CD\n# The policy file is located at: infrastructure/networking/cilium/l2policy.yaml\n```\n\n### 4. GitOps Setup (Argo CD - Part 1/2)\n```bash\n# Install Helm\ncurl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash\n\n# Gateway API CRDs\nkubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml\nkubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/experimental-install.yaml\n\n# Argo CD Bootstrap\nkubectl create namespace argocd\nkubectl kustomize --enable-helm infrastructure/controllers/argocd | kubectl apply -f -\nkubectl apply -f infrastructure/controllers/argocd/projects.yaml\n\n# Wait for Argo CD\nkubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=argocd-server -n argocd --timeout=300s\n\n# Get initial password (change immediately!)\nARGO_PASS=$(kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath=\"{.data.password}\" | base64 -d)\necho \"Initial Argo CD password: $ARGO_PASS\"\n\n#Generate a New Password:\nUse a bcrypt hash generator tool (such as https://www.browserling.com/tools/bcrypt) to create a new bcrypt hash for the password.\nUpdate the argocd-secret secret with the new bcrypt hash.\nkubectl -n argocd patch secret argocd-secret -p '{\"stringData\": { \"admin.password\": \"$2a$10$rgDBwhzr0ygDfH6scxkdddddx3cd612Cutw1Xu1X3a.kVrRq\", \"admin.passwordMtime\": \"'$(date +%FT%T%Z)'\" }}'\n```\n\n### 5. Monitoring Setup (kube-prometheus-stack with Custom Dashboards)\n\nThe monitoring stack uses kube-prometheus-stack Helm chart deployed via Argo CD, providing comprehensive Kubernetes and application monitoring with custom dashboard support.\n\n**Components Included:**\n- **Prometheus**: Metrics collection and storage with increased memory (1Gi) for cluster monitoring\n- **Grafana**: Visualization with custom dashboard auto-discovery via sidecar\n- **AlertManager**: Alert handling and routing\n- **Node Exporter**: Node-level metrics collection\n- **kube-state-metrics**: Kubernetes object state metrics\n\n**Custom Dashboard Management:**\n- Dashboard ConfigMaps are automatically discovered using `grafana_dashboard: \"1\"` labels\n- Stored in `monitoring/kube-prometheus-stack/dashboards/` directory\n- Includes pre-configured K3s cluster overview and community dashboards\n- Tagged with \"custom\" for easy identification in Grafana\n\n**Access URLs (after DNS/Gateway setup):**\n- **Grafana**: `https://grafana.yourdomain.xyz` (default: `admin` / `admin`)\n- **Prometheus**: `https://prometheus.yourdomain.xyz`\n- **AlertManager**: `https://alertmanager.yourdomain.xyz`\n\n**Storage (with Longhorn):**\n- **Prometheus**: `2Gi` with 7-day retention\n- **Grafana**: `1Gi` for dashboards and config\n- **AlertManager**: `512Mi` for alert state\n\n**For detailed dashboard management, see [`monitoring/kube-prometheus-stack/dashboards/README.md`](monitoring/kube-prometheus-stack/dashboards/README.md).**\n\n---\n\nTo add or remove monitoring components, edit `monitoring/monitoring-components-appset.yaml` and comment/uncomment the desired subfolders. Each component is managed as a separate ArgoCD Application in its own namespace.\n\n## 🔒 Security Setup\n\n### Cloudflare Integration\n\nYou'll need to create two secrets for Cloudflare integration:\n1. DNS API Token for cert-manager (DNS validation)\n2. Tunnel credentials for cloudflared (Tunnel connectivity)\n\n#### 1. DNS API Token 🔑\n```bash\n# REQUIRED BROWSER STEPS FIRST:\n# Navigate to Cloudflare Dashboard:\n# 1. Profile \u003e API Tokens\n# 2. Create Token\n# 3. Use \"Edit zone DNS\" template\n# 4. Configure permissions:\n#    - Zone - DNS - Edit\n#    - Zone - Zone - Read\n# 5. Set zone resources to your domain\n# 6. Copy the token and your Cloudflare account email\n\n# Set credentials - NEVER COMMIT THESE!\nexport CLOUDFLARE_API_TOKEN=\"your-api-token-here\"\nexport CLOUDFLARE_EMAIL=\"your-cloudflare-email\"\nexport DOMAIN=\"yourdomain.com\"\nexport TUNNEL_NAME=\"k3s-cluster\"  # Must match config.yaml\n```\n\n#### 2. Cloudflare Tunnel 🌐\n```bash\n# First-time setup only\n# ---------------------\n# Install cloudflared\n# Linux:\nwget -q https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb\nsudo dpkg -i cloudflared-linux-amd64.deb\n# macOS:\nbrew install cloudflare/cloudflare/cloudflared\n\n# Authenticate (opens browser)\ncloudflared tunnel login\n\n# Generate credentials (run from $HOME)\ncloudflared tunnel create $TUNNEL_NAME\ncloudflared tunnel token --cred-file tunnel-creds.json $TUNNEL_NAME\n\nexport DOMAIN=\"yourdomain.com\"\nexport TUNNEL_NAME=\"k3s-cluster\"  # This should match the name in your config.yaml\n\n# Create namespace for cloudflared\nkubectl create namespace cloudflared\n\n# Create Kubernetes secret\nkubectl create secret generic tunnel-credentials \\\n  --namespace=cloudflared \\\n  --from-file=credentials.json=tunnel-creds.json\n\n# SECURITY: Destroy local credentials ( Optional )\nrm -v tunnel-creds.json \u0026\u0026 echo \"Credentials file removed\"\n\n# Configure DNS\nTUNNEL_ID=$(cloudflared tunnel list | grep $TUNNEL_NAME | awk '{print $1}')\ncloudflared tunnel route dns $TUNNEL_ID \"*.$DOMAIN\"\n```\n\n### Certificate Management\n```bash\n# Create cert-manager secrets\nkubectl create namespace cert-manager\nkubectl create secret generic cloudflare-api-token -n cert-manager \\\n  --from-literal=api-token=$CLOUDFLARE_API_TOKEN \\\n  --from-literal=email=$CLOUDFLARE_EMAIL\n\n# Verify secrets\nkubectl get secret cloudflare-api-token -n cert-manager -o jsonpath='{.data.email}' | base64 -d\nkubectl get secret cloudflare-api-token -n cert-manager -o jsonpath='{.data.api-token}' | base64 -d\n```\n\n## 🛠️ Final Deployment\n```bash\n# Apply infrastructure components\n# Run from root of git repo\nkubectl apply -f infrastructure/controllers/argocd/projects.yaml -n argocd\nkubectl apply -f infrastructure/infrastructure-components-appset.yaml -n argocd\n\n# Wait for core services (5-30 mins for certs)\nkubectl wait --for=condition=Available deployment -l type=infrastructure --all-namespaces --timeout=1800s\n\n# Deploy monitoring stack\nkubectl apply -f monitoring/monitoring-components-appset.yaml -n argocd\n\n# Wait for monitoring components to initialize\necho \"Waiting for kube-prometheus-stack to become ready... (this may take a few minutes)\"\nkubectl wait --for=condition=Available deployment -l app.kubernetes.io/name=grafana -n kube-prometheus-stack --timeout=600s\nkubectl wait --for=condition=Available deployment -l app.kubernetes.io/name=kube-state-metrics -n kube-prometheus-stack --timeout=600s\nkubectl wait --for=condition=Ready statefulset -l app.kubernetes.io/name=prometheus -n kube-prometheus-stack --timeout=600s\n\n# Deploy applications\nkubectl apply -f my-apps/myapplications-appset.yaml\n```\n\n## 🔍 Verification\n```bash\n# Cluster status\nkubectl get pods -A --sort-by=.metadata.creationTimestamp\n\n# Argo CD status\nkubectl get applications -n argocd -o wide\n\n# Monitoring stack status\nkubectl get pods -n kube-prometheus-stack\n\n# Certificate checks\nkubectl get certificates -A\nkubectl describe clusterissuer cloudflare-cluster-issuer\n\n# Network validation\ncilium status --verbose\ncilium connectivity test --all-flows\n```\n\n**Access Endpoints:**\n- Argo CD: `https://argocd.$DOMAIN`\n- Grafana: `https://grafana.$DOMAIN`\n- Prometheus: `https://prometheus.$DOMAIN`\n- AlertManager: `https://alertmanager.$DOMAIN`\n- ProxiTok: `https://proxitok.$DOMAIN`\n- SearXNG: `https://search.$DOMAIN`\n- LibReddit: `https://reddit.$DOMAIN`\n\n## 📦 Included Applications\n\n| Category       | Components                          |\n|----------------|-------------------------------------|\n| **Monitoring** | Prometheus, Grafana, Loki, Promtail |\n| **Privacy**    | ProxiTok, SearXNG, LibReddit        |\n| **Infra**      | Cilium, Gateway API, Cloudflared    |\n| **Storage**    | OpenEBS                             |\n| **Security**   | cert-manager, Argo CD Projects      |\n\n## 🤝 Contributing\nContributions welcome! Please:\n1. Maintain existing comment structure\n2. Keep all warnings/security notes\n3. Open issue before major changes\n\n## 📝 License\nMIT License - Full text in [LICENSE](LICENSE)\n\n## 🔧 Troubleshooting\n\n**Common Issues:**\n```bash\n# Certificates not issuing\nkubectl logs -n cert-manager -l app=cert-manager\n\n# Tunnel connection failures\ncloudflared tunnel info $TUNNEL_NAME\nkubectl logs -n cloudflared -l app=cloudflared\n\n# Cilium connectivity issues\ncilium status --verbose\ncilium hubble ui\n\n# L2 Announcement Problems\nip -o link show | awk -F': ' '{print $2}'  # Verify node interfaces\nkubectl describe CiliumL2AnnouncementPolicy -n kube-system\n```\n\n**Monitoring Stack Issues:**\n```bash\n# Check pod status in the kube-prometheus-stack namespace\nkubectl get pods -n kube-prometheus-stack\n\n# If pods are stuck, check the Argo CD UI for sync errors.\n# Look at the 'kube-prometheus-stack' application.\n\n# Describe a pod to see its events and find out why it's not starting\nkubectl describe pod \u003cpod-name\u003e -n kube-prometheus-stack\n\n# Check logs for specific monitoring components\nkubectl logs -l app.kubernetes.io/name=grafana -n kube-prometheus-stack\nkubectl logs -l app.kubernetes.io/name=prometheus -n kube-prometheus-stack\n\n# Check Grafana sidecar for dashboard loading issues\nkubectl logs -l app.kubernetes.io/name=grafana -c grafana-sc-dashboard -n kube-prometheus-stack\n\n# Verify custom dashboard ConfigMaps are labeled correctly\nkubectl get configmaps -n kube-prometheus-stack -l grafana_dashboard=1\n```\n\n**Multi-Attach Volume Errors (ReadWriteOnce Issues):**\n```bash\n# PROBLEM: Multiple pods trying to mount the same ReadWriteOnce (RWO) volume\n# SYMPTOMS: Pods stuck in Init:0/2 or Pending state with multi-attach errors\n# COMMON CAUSE: ArgoCD rolling updates with Replace=true causing resource conflicts\n\n# Check for stuck pods and volume attachment issues\nkubectl get pods -A | grep -E \"(Init|Pending|ContainerCreating)\"\nkubectl get volumeattachments\nkubectl get pvc -A\n\n# Identify the problematic pod and PVC\nkubectl describe pod \u003cstuck-pod-name\u003e -n \u003cnamespace\u003e\n\n# IMMEDIATE FIX: Force delete the stuck pod (temporary solution)\nkubectl delete pod \u003cstuck-pod-name\u003e -n \u003cnamespace\u003e --force --grace-period=0\n\n# PERMANENT SOLUTION: Fix deployment strategies for RWO volumes\n# 1. Update ApplicationSet sync options (remove Replace=true)\n# 2. Set deployment strategy to 'Recreate' for apps using RWO volumes\n# 3. Add RespectIgnoreDifferences=true to prevent unnecessary syncs\n\n# Verify fixes are applied:\n# Check ApplicationSet sync options\nkubectl get applicationset -n argocd -o yaml | grep -A 10 syncOptions\n\n# Check deployment strategies for RWO volume users\nkubectl get deployment grafana -n kube-prometheus-stack -o jsonpath='{.spec.strategy.type}'\nkubectl get deployment proxitok-web -n proxitok -o jsonpath='{.spec.strategy.type}'\nkubectl get deployment homepage-dashboard -n homepage-dashboard -o jsonpath='{.spec.strategy.type}'\nkubectl get deployment redis -n searxng -o jsonpath='{.spec.strategy.type}'\n\n# All should return 'Recreate' for apps using persistent volumes\n```\n\n**Key Prevention Strategies:**\n- **Use `Recreate` deployment strategy** for any app with ReadWriteOnce volumes\n- **Remove `Replace=true`** from ArgoCD ApplicationSet sync options\n- **Add `RespectIgnoreDifferences=true`** to prevent unnecessary rolling updates\n- **Use `ApplyOutOfSyncOnly=true`** to only update resources that are actually out of sync\n\n**Specific Changes Made to Fix Multi-Attach Errors:**\n\n1. **ApplicationSet Sync Options Updated:**\n   ```yaml\n   # REMOVED from all ApplicationSets:\n   # - Replace=true  # This was causing resource deletion/recreation\n   \n   # ADDED to all ApplicationSets:\n   syncOptions:\n     - RespectIgnoreDifferences=true  # Prevents unnecessary syncs\n     - ApplyOutOfSyncOnly=true       # Only sync out-of-sync resources\n   ```\n\n2. **Deployment Strategy Changes for RWO Volume Apps:**\n   ```yaml\n   # monitoring/kube-prometheus-stack/values.yaml\n   grafana:\n     deploymentStrategy:\n       type: Recreate  # Added to prevent multi-attach during updates\n   \n   # my-apps/homepage-dashboard/deployment.yaml\n   spec:\n     strategy:\n       type: Recreate  # Added for RWO volume safety\n   \n   # my-apps/proxitok/deployment.yaml  \n   spec:\n     strategy:\n       type: Recreate  # Added for cache PVC\n   \n   # my-apps/searxng/redis.yaml\n   spec:\n     strategy:\n       type: Recreate  # Added for Redis data persistence\n   ```\n\n\n\n**Why These Changes Work:**\n- **`Recreate` vs `RollingUpdate`**: With ReadWriteOnce volumes, `RollingUpdate` tries to start new pods before old ones terminate, causing volume conflicts. `Recreate` ensures complete pod termination first.\n- **Removing `Replace=true`**: This ArgoCD option deletes and recreates all resources during sync, triggering unnecessary rolling updates and volume conflicts.\n- **`RespectIgnoreDifferences=true`**: Prevents ArgoCD from syncing minor differences that don't affect functionality, reducing unnecessary pod restarts.\n- **Sync Wave Ordering**: Monitoring components use sync wave \"1\" to deploy after infrastructure (wave \"-2\" and \"0\"), ensuring proper resource availability.\n\n**Critical L2 Note:**\nIf LoadBalancer IPs aren't advertising properly:\n1. Verify physical interface name matches in CiliumL2AnnouncementPolicy\n2. Check interface exists on all nodes: `ip link show dev enp1s0`\n3. Ensure Cilium pods are running: `kubectl get pods -n kube-system -l k8s-app=cilium`\n\n**Longhorn Volume Mount Issues:**\n```bash\n# PROBLEM: Volumes fail to mount with \"device busy\" or multipath conflicts\n# COMMON CAUSE: Linux multipath daemon interfering with Longhorn device management\n\n# Check if multipathd is running (often enabled by default on Ubuntu/Debian)\nsystemctl status multipathd\n\n# SOLUTION: Disable multipath daemon on all nodes\nsudo systemctl disable --now multipathd\n\n# Verify it's stopped\nsystemctl is-active multipathd  # Should return \"inactive\"\n\n# After disabling multipathd, restart kubelet to clear any cached device state\nsudo systemctl restart k3s  # For K3s\n# OR\nsudo systemctl restart kubelet  # For standard Kubernetes\n\n# Check Longhorn volume status after restart\nkubectl get volumes -n longhorn-system\nkubectl get pods -n longhorn-system\n\n# Reference: https://longhorn.io/kb/troubleshooting-volume-with-multipath/\n```\n\nAll original comments, warnings, and TODOs preserved. Formatting optimized for readability while maintaining technical accuracy.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmitchross%2Fk3s-argocd-starter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmitchross%2Fk3s-argocd-starter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmitchross%2Fk3s-argocd-starter/lists"}