{"id":43982241,"url":"https://github.com/mutugading/goapps-infra","last_synced_at":"2026-02-07T09:05:55.986Z","repository":{"id":335772882,"uuid":"1145841193","full_name":"mutugading/goapps-infra","owner":"mutugading","description":null,"archived":false,"fork":false,"pushed_at":"2026-02-01T10:10:15.000Z","size":131,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-01T13:31:09.936Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mutugading.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-30T09:30:18.000Z","updated_at":"2026-02-01T10:10:19.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mutugading/goapps-infra","commit_stats":null,"previous_names":["mutugading/goapps-infra"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/mutugading/goapps-infra","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mutugading%2Fgoapps-infra","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mutugading%2Fgoapps-infra/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mutugading%2Fgoapps-infra/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mutugading%2Fgoapps-infra/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mutugading","download_url":"https://codeload.github.com/mutugading/goapps-infra/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mutugading%2Fgoapps-infra/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29190842,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-07T07:37:03.739Z","status":"ssl_error","status_checked_at":"2026-02-07T07:37:03.029Z","response_time":63,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-07T09:05:55.306Z","updated_at":"2026-02-07T09:05:55.976Z","avatar_url":"https://github.com/mutugading.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GoApps Infrastructure (goapps-infra)\n\nInfrastructure as Code (IaC) for **GoApps Microservices Platform** - managing Kubernetes deployments, monitoring, backups, and GitOps.\n\n---\n\n## 📋 Table of Contents\n\n1. [Overview](#overview)\n2. [Platform Architecture](#platform-architecture)\n3. [Technology Stack](#technology-stack)\n4. [Repository Structure](#repository-structure)\n5. [Quick Start](#quick-start)\n6. [Environment Configuration](#environment-configuration)\n7. [Infrastructure Components](#infrastructure-components)\n8. [Service Deployment](#service-deployment)\n9. [Monitoring \u0026 Observability](#monitoring--observability)\n10. [Backup \u0026 Disaster Recovery](#backup--disaster-recovery)\n11. [Security \u0026 Secrets Management](#security--secrets-management)\n12. [CI/CD Pipeline](#cicd-pipeline)\n13. [Troubleshooting](#troubleshooting)\n14. [Related Documentation](#related-documentation)\n\n---\n\n## Overview\n\nThis repository contains all infrastructure configurations for the GoApps platform, including:\n\n- **Kubernetes Manifests**: Deployments, Services, ConfigMaps, Secrets\n- **Kustomize Overlays**: Environment-specific configurations (staging/production)\n- **Helm Values**: Configuration for Prometheus, Grafana, Loki, ArgoCD\n- **GitOps Workflows**: ArgoCD Applications for auto-sync from Git\n- **Automation Scripts**: Bootstrap, monitoring, and maintenance scripts\n\n### GoApps Ecosystem Repositories\n\n```\ngoapps/\n├── goapps-infra/          # 🔧 Infrastructure (this repo)\n├── goapps-backend/        # 🖥️  Backend microservices (Go + gRPC)\n├── goapps-frontend/       # 🌐 Frontend application (Next.js)\n└── goapps-shared-proto/   # 📝 Protocol Buffer definitions\n```\n\n| Repository | Description | Tech Stack |\n|------------|-------------|------------|\n| `goapps-infra` | Infrastructure as Code | Kubernetes, Kustomize, Helm, ArgoCD |\n| `goapps-backend` | Microservices APIs | Go, gRPC, PostgreSQL, Redis |\n| `goapps-frontend` | Web Application | Next.js 15, React, TypeScript |\n| `goapps-shared-proto` | API Contracts | Protocol Buffers, Buf |\n\n---\n\n## Platform Architecture\n\n### High-Level Architecture\n\n```\n┌───────────────────────────────────────────────────────────────────────────────┐\n│                              goapps Platform                                  │\n├───────────────────────────────────────────────────────────────────────────────┤\n│                                                                               │\n│  ┌─────────────────────────────────────────────────────────────────────────┐  │\n│  │                         EXTERNAL ACCESS                                 │  │\n│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐    │  │\n│  │  │   HTTPS     │  │   HTTPS     │  │   HTTPS     │  │   gRPC      │    │  │\n│  │  │   :443      │  │   :443      │  │   :30090    │  │   :50051    │    │  │\n│  │  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘    │  │\n│  └─────────┼────────────────┼────────────────┼────────────────┼───────────┘  │\n│            │                │                │                │              │\n│  ┌─────────▼────────────────▼────────────────▼────────────────▼───────────┐  │\n│  │                    INGRESS LAYER (NGINX)                               │  │\n│  │                 TLS Termination + Path-Based Routing                   │  │\n│  └─────────────────────────────────────────────────────────────────────────┘  │\n│                                      │                                        │\n│            ┌─────────────────────────┼─────────────────────────┐              │\n│            │                         │                         │              │\n│            ▼                         ▼                         ▼              │\n│  ┌─────────────────┐      ┌─────────────────┐      ┌─────────────────────┐   │\n│  │   APPLICATIONS  │      │   MONITORING    │      │      STORAGE        │   │\n│  │                 │      │                 │      │                     │   │\n│  │ ┌─────────────┐ │      │ ┌─────────────┐ │      │ ┌─────────────────┐ │   │\n│  │ │frontend-svc │ │      │ │  Grafana    │ │      │ │   PostgreSQL    │ │   │\n│  │ │ (Next.js)   │ │      │ │  /grafana   │ │      │ │   (Primary DB)  │ │   │\n│  │ └─────────────┘ │      │ └─────────────┘ │      │ └─────────────────┘ │   │\n│  │ ┌─────────────┐ │      │ ┌─────────────┐ │      │ ┌─────────────────┐ │   │\n│  │ │finance-svc  │ │      │ │ Prometheus  │ │      │ │    PgBouncer    │ │   │\n│  │ │ (Go/gRPC)   │ │      │ │ /prometheus │ │      │ │   (Pool Conn)   │ │   │\n│  │ └─────────────┘ │      │ └─────────────┘ │      │ └─────────────────┘ │   │\n│  │ ┌─────────────┐ │      │ ┌─────────────┐ │      │ ┌─────────────────┐ │   │\n│  │ │ iam-svc     │ │      │ │    Loki     │ │      │ │      Redis      │ │   │\n│  │ │ (future)    │ │      │ │   (Logs)    │ │      │ │    (Caching)    │ │   │\n│  │ └─────────────┘ │      │ └─────────────┘ │      │ └─────────────────┘ │   │\n│  └─────────────────┘      │ ┌─────────────┐ │      │ ┌─────────────────┐ │   │\n│                           │ │ Alertmanager│ │      │ │    RabbitMQ     │ │   │\n│  ┌─────────────────┐      │ │   (Email)   │ │      │ │   (Messaging)   │ │   │\n│  │     GITOPS      │      │ └─────────────┘ │      │ └─────────────────┘ │   │\n│  │                 │      └─────────────────┘      │ ┌─────────────────┐ │   │\n│  │ ┌─────────────┐ │                               │ │     MinIO       │ │   │\n│  │ │   ArgoCD    │ │      ┌─────────────────┐      │ │ (Object Store)  │ │   │\n│  │ │  /argocd    │ │      │   OBSERVABILITY │      │ └─────────────────┘ │   │\n│  │ └─────────────┘ │      │                 │      └─────────────────────┘   │\n│  └─────────────────┘      │ ┌─────────────┐ │                                │\n│                           │ │   Jaeger    │ │                                │\n│                           │ │  (Tracing)  │ │                                │\n│                           │ └─────────────┘ │                                │\n│                           └─────────────────┘                                │\n└───────────────────────────────────────────────────────────────────────────────┘\n```\n\n### Namespace Architecture\n\n```mermaid\ngraph TB\n    subgraph \"Kubernetes Cluster (K3s)\"\n        \n        subgraph \"ingress-nginx\"\n            NGINX[NGINX Ingress Controller]\n        end\n        \n        subgraph \"goapps-staging\"\n            FS_STG[finance-service]\n            FE_STG[frontend]\n        end\n        \n        subgraph \"goapps-production\"\n            FS_PRD[finance-service]\n            FE_PRD[frontend]\n        end\n        \n        subgraph \"database\"\n            PG[PostgreSQL 18]\n            PGB[PgBouncer]\n            REDIS[Redis]\n            RMQ[RabbitMQ]\n            EXP[Postgres Exporter]\n        end\n        \n        subgraph \"minio\"\n            MINIO[MinIO Server]\n        end\n        \n        subgraph \"monitoring\"\n            PROM[Prometheus]\n            GRAF[Grafana]\n            LOKI[Loki]\n            PROMTAIL[Promtail]\n            ALERT[Alertmanager]\n        end\n        \n        subgraph \"observability\"\n            JAEGER[Jaeger]\n        end\n        \n        subgraph \"argocd\"\n            ARGO[ArgoCD Server]\n        end\n    end\n    \n    NGINX --\u003e FS_STG\n    NGINX --\u003e FE_STG\n    NGINX --\u003e FS_PRD\n    NGINX --\u003e FE_PRD\n    NGINX --\u003e GRAF\n    NGINX --\u003e PROM\n    NGINX --\u003e ARGO\n    \n    FS_STG --\u003e PG\n    FS_STG --\u003e REDIS\n    FS_PRD --\u003e PG\n    FS_PRD --\u003e REDIS\n```\n\n### Data Flow Diagram\n\n```\n┌──────────────────────────────────────────────────────────────────────────────┐\n│                              REQUEST FLOW                                     │\n└──────────────────────────────────────────────────────────────────────────────┘\n\n  User Request\n       │\n       ▼\n  ┌─────────┐     ┌─────────────┐     ┌─────────────┐     ┌─────────────┐\n  │  HTTPS  │────▶│   NGINX     │────▶│  Frontend   │────▶│  Backend    │\n  │  :443   │     │  Ingress    │     │  (Next.js)  │     │  (gRPC)     │\n  └─────────┘     └─────────────┘     └─────────────┘     └─────────────┘\n                                                                │\n                        ┌───────────────────────────────────────┤\n                        │               │               │       │\n                        ▼               ▼               ▼       ▼\n                  ┌──────────┐   ┌──────────┐   ┌──────────┐ ┌──────────┐\n                  │PostgreSQL│   │  Redis   │   │ RabbitMQ │ │  Oracle  │\n                  │ (Main DB)│   │ (Cache)  │   │ (Queue)  │ │(External)│\n                  └──────────┘   └──────────┘   └──────────┘ └──────────┘\n                        │\n                        ▼\n                  ┌──────────┐\n                  │ PgBouncer│ (Connection Pooling)\n                  └──────────┘\n\n\n┌──────────────────────────────────────────────────────────────────────────────┐\n│                              BACKUP FLOW                                      │\n└──────────────────────────────────────────────────────────────────────────────┘\n\n  ┌──────────────┐      ┌─────────────┐      ┌─────────────┐\n  │  PostgreSQL  │─────▶│   MinIO     │─────▶│ Backblaze   │\n  │   pg_dump    │      │  (Local S3) │      │ B2 (Cloud)  │\n  └──────────────┘      └─────────────┘      └─────────────┘\n         │                     │\n         ▼                     ▼\n  ┌──────────────┐      ┌─────────────┐\n  │  VPS Disk    │      │  VPS Disk   │\n  │  /mnt/backup │      │  /mnt/backup│\n  └──────────────┘      └─────────────┘\n```\n\n---\n\n## Technology Stack\n\n### Kubernetes \u0026 Orchestration\n\n| Component | Version | Description |\n|-----------|---------|-------------|\n| K3s | v1.34.x | Lightweight Kubernetes distribution |\n| Kustomize | v5.3.0 | Native Kubernetes configuration management |\n| Helm | v3.x | Package manager for Kubernetes charts |\n| ArgoCD | v7.7.5 | GitOps continuous delivery |\n\n### Database \u0026 Storage\n\n| Component | Version | Description |\n|-----------|---------|-------------|\n| PostgreSQL | 18-alpine | Primary relational database |\n| PgBouncer | latest | Connection pooling |\n| Redis | 7-alpine | In-memory caching |\n| RabbitMQ | 3-management | Message queue |\n| MinIO | latest | S3-compatible object storage |\n\n### Monitoring \u0026 Observability\n\n| Component | Version | Description |\n|-----------|---------|-------------|\n| Prometheus | 2.x (via kube-prometheus-stack) | Metrics collection |\n| Grafana | 11.x | Visualization \u0026 dashboards |\n| Loki | 2.x | Log aggregation |\n| Promtail | 2.x | Log shipping agent |\n| Alertmanager | 0.x | Alert routing \u0026 notifications |\n| Jaeger | latest | Distributed tracing |\n\n### Networking \u0026 Security\n\n| Component | Description |\n|-----------|-------------|\n| NGINX Ingress Controller | L7 load balancer \u0026 TLS termination |\n| TLS/SSL | Wildcard certificate (*.mutugading.com) |\n| Basic Auth | Prometheus protection (production) |\n\n---\n\n## Repository Structure\n\n```\ngoapps-infra/\n│\n├── 📁 base/                          # Base Kustomize resources (shared)\n│   ├── argocd/                       # ArgoCD base configuration\n│   │   └── kustomization.yaml\n│   ├── backup/                       # Backup configurations\n│   │   ├── cronjobs/                 # PostgreSQL backup schedules\n│   │   │   ├── minio-backup.yaml     # MinIO to VPS backup\n│   │   │   └── postgres-backup.yaml  # 3x daily backups\n│   │   ├── minio/                    # MinIO deployment\n│   │   │   ├── deployment.yaml\n│   │   │   └── service.yaml\n│   │   └── kustomization.yaml\n│   ├── database/                     # Database layer\n│   │   ├── exporter/                 # Postgres exporter for metrics\n│   │   ├── oracle/                   # Oracle external service config\n│   │   ├── pgbouncer/                # Connection pooler\n│   │   ├── postgres/                 # PostgreSQL StatefulSet\n│   │   ├── rabbitmq/                 # Message queue\n│   │   ├── redis/                    # Cache layer\n│   │   └── kustomization.yaml\n│   ├── ingress/                      # Ingress base configs\n│   ├── kubernetes-dashboard/         # K8s Dashboard admin\n│   ├── monitoring/                   # Monitoring stack\n│   │   ├── alert-rules/              # Grafana alert definitions\n│   │   ├── dashboards/               # Grafana dashboard JSONs\n│   │   ├── datasources/              # Grafana datasource configs\n│   │   └── helm-values/              # Prometheus/Loki Helm values\n│   ├── namespaces/                   # Namespace definitions\n│   ├── observability/                # Jaeger tracing\n│   └── secrets/                      # Secret templates (NOT REAL SECRETS!)\n│\n├── 📁 overlays/                      # Environment-specific patches\n│   ├── staging/                      # Staging environment\n│   │   ├── backup/                   # Staging backup paths\n│   │   ├── backup-patch.yaml         # Override backup locations\n│   │   ├── ingress.yaml              # Staging ingress rules\n│   │   └── minio/                    # Staging MinIO config\n│   └── production/                   # Production environment\n│       ├── backup/                   # Production backup paths\n│       ├── backup-patch.yaml         # Override backup locations\n│       ├── ingress.yaml              # Production ingress rules\n│       └── minio/                    # Production MinIO config\n│\n├── 📁 services/                      # Application deployments\n│   ├── finance-service/              # Finance microservice\n│   │   ├── base/                     # Base deployment\n│   │   │   ├── deployment.yaml       # Container spec\n│   │   │   ├── hpa.yaml              # Auto-scaling\n│   │   │   ├── ingress.yaml          # Service ingress\n│   │   │   ├── kustomization.yaml\n│   │   │   └── service.yaml\n│   │   └── overlays/\n│   │       ├── staging/              # Staging overrides\n│   │       └── production/           # Production overrides\n│   └── frontend/                     # Frontend service\n│       ├── base/\n│       └── overlays/\n│\n├── 📁 argocd/                        # ArgoCD GitOps configs\n│   ├── apps/                         # ArgoCD Application manifests\n│   │   ├── shared/                   # Shared apps (database, monitoring)\n│   │   ├── staging/                  # Staging-only apps\n│   │   └── production/               # Production-only apps\n│   └── projects/                     # ArgoCD Projects\n│\n├── 📁 scripts/                       # Automation scripts\n│   ├── bootstrap.sh                  # Initial cluster setup\n│   ├── reset-k3s.sh                  # Clean uninstall K3s\n│   ├── install-monitoring.sh         # Install Prometheus/Grafana/Loki\n│   ├── install-argocd.sh             # Install ArgoCD\n│   ├── install-nginx-ingress.sh      # Install NGINX Ingress\n│   ├── install-runner.sh             # Install GitHub Actions runner\n│   ├── fix-staging.sh                # Staging troubleshooting\n│   └── fix-production.sh             # Production troubleshooting\n│\n├── 📁 docs/                          # Documentation\n│   ├── deployment-guide.md           # Step-by-step deployment\n│   ├── vps-reset-guide.md            # Complete VPS reset procedure\n│   └── runbooks/                     # Operational runbooks\n│\n├── 📁 .github/                       # GitHub Actions\n│   ├── workflows/\n│   │   ├── ci.yml                    # Validate manifests \u0026 lint\n│   │   ├── health-check.yml          # Scheduled health checks\n│   │   └── sync-argocd.yml           # ArgoCD sync on push\n│   ├── ISSUE_TEMPLATE/               # Issue templates\n│   │   ├── bug_report.md\n│   │   ├── feature_request.md\n│   │   ├── new_service.md\n│   │   ├── incident_report.md\n│   │   └── config.yml\n│   ├── PULL_REQUEST_TEMPLATE.md\n│   └── actions/\n│       └── argocd-sync/              # Reusable ArgoCD sync action\n│\n├── .gitignore                        # Git ignore rules\n├── .yamllint.yml                     # YAML linting config\n├── Makefile                          # Common make targets\n├── README.md                         # This file\n├── RULES.md                          # Development rules \u0026 conventions\n├── CONTRIBUTING.md                   # Contribution guidelines\n└── LICENSE                           # Proprietary license\n```\n\n---\n\n## Quick Start\n\n### Prerequisites\n\nOn VPS (Ubuntu 24.04 LTS):\n- SSH access with root/sudo\n- Disk partition for backup (`/dev/sdb1`)\n- SSL certificates (`ssl-bundle.crt`, `mutugading.com.key`)\n\nOn local machine:\n- Git installed\n- kubectl configured\n\n### 1. Clone Repository\n\n```bash\nssh deploy@\u003cvps-hostname\u003e\ncd ~\ngit clone https://github.com/mutugading/goapps-infra.git\ncd goapps-infra\nchmod +x scripts/*.sh\n```\n\n### 2. Bootstrap K3s Cluster\n\n```bash\n# Staging VPS\n./scripts/bootstrap.sh\n\n# Production VPS\nENVIRONMENT=production ./scripts/bootstrap.sh\n```\n\nThis script will:\n- Install K3s (without Traefik - using NGINX Ingress)\n- Install Helm\n- Create namespaces: `database`, `monitoring`, `minio`, `argocd`, `goapps-staging/production`\n- Install VPA (Vertical Pod Autoscaler)\n\n### 3. Create Secrets\n\n\u003e ⚠️ **IMPORTANT**: Secrets must NOT be committed to Git!\n\n```bash\n# PostgreSQL\nkubectl create secret generic postgres-secret -n database \\\n  --from-literal=POSTGRES_USER=goapps_admin \\\n  --from-literal=POSTGRES_PASSWORD='\u003cSTRONG_PASSWORD\u003e' \\\n  --from-literal=POSTGRES_DB=goapps\n\n# MinIO\nkubectl create secret generic minio-secret -n minio \\\n  --from-literal=MINIO_ROOT_USER=admin \\\n  --from-literal=MINIO_ROOT_PASSWORD='\u003cSTRONG_PASSWORD\u003e'\n\n# Copy MinIO secret to database namespace\nkubectl get secret minio-secret -n minio -o yaml | \\\n  sed 's/namespace: minio/namespace: database/' | \\\n  kubectl apply -f -\n\n# TLS Certificate\nkubectl create secret tls goapps-tls -n monitoring \\\n  --cert=ssl-bundle.crt \\\n  --key=mutugading.com.key\n\n# Copy TLS to other namespaces\nfor ns in argocd ingress-nginx goapps-staging kubernetes-dashboard; do\n  kubectl create ns $ns 2\u003e/dev/null || true\n  kubectl get secret goapps-tls -n monitoring -o yaml | \\\n    sed \"s/namespace: monitoring/namespace: $ns/\" | \\\n    kubectl apply -f -\ndone\n\n# Grafana SMTP\nkubectl create secret generic grafana-smtp-secret -n monitoring \\\n  --from-literal=password='\u003cSMTP_PASSWORD\u003e'\n```\n\nSee [docs/vps-reset-guide.md](docs/vps-reset-guide.md) for complete secrets list.\n\n### 4. Install Monitoring Stack\n\n```bash\nexport GRAFANA_PASSWORD='your-secure-password'\n./scripts/install-monitoring.sh\n```\n\n### 5. Apply Base Infrastructure\n\n```bash\n# Apply all base configs\nkubectl apply -k base/database/\nkubectl apply -k base/backup/\nkubectl apply -k base/monitoring/alert-rules/\n```\n\n### 6. Install ArgoCD\n\n```bash\n./scripts/install-argocd.sh\n```\n\n### 7. Install NGINX Ingress\n\n```bash\n./scripts/install-nginx-ingress.sh\n```\n\n### 8. Apply Ingress \u0026 ArgoCD Apps\n\n```bash\n# Staging\nkubectl apply -f overlays/staging/ingress.yaml\nkubectl apply -f argocd/apps/staging/\nkubectl apply -f argocd/apps/shared/\nkubectl apply -f argocd/projects/\n\n# Production\nkubectl apply -f overlays/production/ingress.yaml\nkubectl apply -f argocd/apps/production/\nkubectl apply -f argocd/apps/shared/\nkubectl apply -f argocd/projects/\n```\n\n### 9. Verify Installation\n\n```bash\nmake status\n```\n\n---\n\n## Environment Configuration\n\n### Staging vs Production\n\n| Aspect | Staging | Production |\n|--------|---------|------------|\n| **VPS Specs** | 4 core, 8GB RAM | 8 core, 16GB RAM |\n| **Domain** | staging-goapps.mutugading.com | goapps.mutugading.com |\n| **Backup Mount** | `/staging-goapps-backup` | `/goapps-backup` |\n| **ArgoCD Sync** | Automatic | Manual approval |\n| **Prometheus Auth** | No auth | Basic Auth required |\n| **Namespace** | `goapps-staging` | `goapps-production` |\n\n### Access URLs\n\n#### Staging\n| Service | URL |\n|---------|-----|\n| Grafana | https://staging-goapps.mutugading.com/grafana |\n| Prometheus | https://staging-goapps.mutugading.com/prometheus |\n| ArgoCD | https://staging-goapps.mutugading.com/argocd |\n| MinIO Console | https://staging-goapps.mutugading.com:30090 |\n\n#### Production\n| Service | URL |\n|---------|-----|\n| Grafana | https://goapps.mutugading.com/grafana |\n| Prometheus | https://goapps.mutugading.com/prometheus |\n| ArgoCD | https://goapps.mutugading.com/argocd |\n| MinIO Console | https://goapps.mutugading.com:30090 |\n\n---\n\n## Infrastructure Components\n\n### Database Layer\n\n#### PostgreSQL 18\n\n- **Location**: `base/database/postgres/`\n- **Type**: StatefulSet with 20Gi PVC\n- **Access**: `postgres.database.svc.cluster.local:5432`\n\n```yaml\n# Schemas created automatically:\n- finance    # Finance service data\n- (future schemas added in init-schemas.sql)\n```\n\n#### PgBouncer\n\n- **Location**: `base/database/pgbouncer/`\n- **Mode**: Transaction pooling\n- **Pool Size**: 100 connections\n- **Access**: `pgbouncer.database.svc.cluster.local:5432`\n\n#### Redis\n\n- **Location**: `base/database/redis/`\n- **Purpose**: Session cache, rate limiting\n- **Access**: `redis.database.svc.cluster.local:6379`\n\n#### RabbitMQ\n\n- **Location**: `base/database/rabbitmq/`\n- **Purpose**: Async message queue\n- **Access**: `rabbitmq.database.svc.cluster.local:5672`\n- **Management UI**: Port 15672\n\n### Storage Layer\n\n#### MinIO (S3-Compatible)\n\n- **Location**: `base/backup/minio/`\n- **Purpose**: Local object storage for backups\n- **Access**: `minio.minio.svc.cluster.local:9000` (API), `:9001` (Console)\n- **TLS**: Enabled via NodePort 30090\n\n### Ingress Layer\n\n#### NGINX Ingress Controller\n\n- **Installation**: `scripts/install-nginx-ingress.sh`\n- **Purpose**: L7 load balancing, TLS termination, path-based routing\n- **Configuration**: `overlays/{staging,production}/ingress.yaml`\n\n---\n\n## Service Deployment\n\n### Service Deployment Pattern\n\nEach service follows the Kustomize base + overlays pattern:\n\n```\nservices/\u003cservice-name\u003e/\n├── base/\n│   ├── deployment.yaml      # Container spec\n│   ├── service.yaml         # K8s Service\n│   ├── hpa.yaml             # HorizontalPodAutoscaler\n│   ├── ingress.yaml         # Optional ingress rules\n│   └── kustomization.yaml   # Kustomize config\n└── overlays/\n    ├── staging/\n    │   ├── kustomization.yaml\n    │   └── patches/\n    └── production/\n        ├── kustomization.yaml\n        └── patches/\n```\n\n### Finance Service\n\n**Base Configuration** (`services/finance-service/base/`):\n\n```yaml\nPorts:\n  - containerPort: 50051  # gRPC\n  - containerPort: 8080   # HTTP\n  - containerPort: 8090   # Metrics (/metrics)\n\nResources:\n  Requests: 128Mi memory, 100m CPU\n  Limits: 512Mi memory, 500m CPU\n\nHPA:\n  Min: 1, Max: 5 replicas\n  Target CPU: 70%\n\nEnvironment Variables:\n  - DATABASE_HOST: postgres.database.svc.cluster.local\n  - REDIS_HOST: redis.database.svc.cluster.local\n  - JAEGER_ENDPOINT: jaeger-collector.monitoring.svc.cluster.local:4317\n```\n\n### Deploy New Service\n\nSee [RULES.md](./RULES.md) for complete guide on adding new services.\n\n```bash\n# 1. Create directory structure\nmkdir -p services/new-service/{base,overlays/{staging,production}}\n\n# 2. Copy template from finance-service\ncp -r services/finance-service/base/* services/new-service/base/\n\n# 3. Update manifests (image, ports, env)\n# Edit services/new-service/base/deployment.yaml\n\n# 4. Create ArgoCD Application\ncat \u003e argocd/apps/staging/new-service.yaml \u003c\u003c EOF\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: new-service-staging\n  namespace: argocd\nspec:\n  project: goapps\n  source:\n    repoURL: https://github.com/mutugading/goapps-infra.git\n    targetRevision: main\n    path: services/new-service/overlays/staging\n  destination:\n    server: https://kubernetes.default.svc\n    namespace: goapps-staging\n  syncPolicy:\n    automated:\n      prune: true\n      selfHeal: true\nEOF\n\n# 5. Commit and push\ngit add . \u0026\u0026 git commit -m \"feat: add new-service\" \u0026\u0026 git push\n```\n\n---\n\n## Monitoring \u0026 Observability\n\n### Prometheus\n\n- **Retention**: 30 days\n- **Storage**: 20Gi PVC\n- **Sub-path**: `/prometheus`\n- **Basic Auth**: Production only\n\n### Grafana\n\n- **Persistence**: 10Gi PVC\n- **Sub-path**: `/grafana`\n- **SMTP**: Email alerts via mgtalert@mutugading.com\n\n#### Available Dashboards\n\n| Dashboard | File | Description |\n|-----------|------|-------------|\n| Go Apps Overview | `grafana-dashboard-go-apps.json` | Service metrics, HPA status |\n| PostgreSQL | `grafana-dashboard-postgres.json` | Database performance |\n| Loki Logs | `grafana-dashboard-loki.json` | Centralized logging |\n\n### Alert Categories\n\n| Category | Alert Examples |\n|----------|----------------|\n| Node Health | High CPU/Memory/Disk usage |\n| Pod Status | CrashLoopBackOff, High restart count |\n| HPA Scaling | Reached max replicas |\n| PVC Storage | Volume nearly full |\n| PostgreSQL | Connection pool exhausted, Slow queries |\n| Backup | Backup job failed, Old backups |\n\n### Jaeger Tracing\n\n- **Namespace**: `observability`\n- **Collector**: `jaeger-collector.monitoring.svc.cluster.local:4317`\n- **Access**: Port-forward to localhost:16686\n\n---\n\n## Backup \u0026 Disaster Recovery\n\n### Backup Strategy\n\n```\n┌──────────────────────────────────────────────────────────────────────────┐\n│                           BACKUP STRATEGY                                │\n├──────────────┬──────────────────┬────────────┬──────────────┬───────────┤\n│    Target    │   Destination    │  Schedule  │  Retention   │   Type    │\n├──────────────┼──────────────────┼────────────┼──────────────┼───────────┤\n│ PostgreSQL   │ MinIO (Local)    │ 3x daily   │ 7 days       │ pg_dump   │\n│ PostgreSQL   │ Backblaze B2     │ 3x daily   │ 7 days       │ pg_dump   │\n│ PostgreSQL   │ VPS Disk         │ 3x daily   │ 7 days       │ pg_dump   │\n│ MinIO        │ VPS Disk only    │ Daily      │ 7 days       │ mc mirror │\n└──────────────┴──────────────────┴────────────┴──────────────┴───────────┘\n```\n\n### Backup Schedule\n\n| CronJob | Time (WIB) | Timezone |\n|---------|------------|----------|\n| `postgres-backup-morning` | 06:00 | Asia/Jakarta |\n| `postgres-backup-afternoon` | 14:00 | Asia/Jakarta |\n| `postgres-backup-night` | 22:00 | Asia/Jakarta |\n| `minio-backup` | 03:00 | Asia/Jakarta |\n\n### Backup Locations\n\n| Environment | VPS Path | MinIO Bucket |\n|-------------|----------|--------------|\n| Staging | `/staging-goapps-backup/postgres` | `postgres-backups` |\n| Production | `/goapps-backup/postgres` | `postgres-backups` |\n\n### Manual Backup\n\n```bash\n# Trigger backup manually\nmake backup-now\n\n# Or directly\nkubectl create job --from=cronjob/postgres-backup-morning \\\n  postgres-backup-manual-$(date +%Y%m%d%H%M%S) -n database\n```\n\n### Restore Procedure\n\n```bash\n# 1. List available backups\nls -la /mnt/goapps-backup/postgres/\n# or\nmc ls minio/postgres-backups/\n\n# 2. Copy backup to pod\nkubectl cp \u003cbackup-file\u003e.sql.gz database/postgres-0:/tmp/\n\n# 3. Restore\nkubectl exec -it postgres-0 -n database -- bash -c \"\n  gunzip -c /tmp/\u003cbackup-file\u003e.sql.gz | psql -U postgres -d goapps\n\"\n```\n\n---\n\n## Security \u0026 Secrets Management\n\n### Secrets Checklist\n\n| Secret Name | Namespace | Required Keys |\n|-------------|-----------|---------------|\n| `postgres-secret` | database | POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB |\n| `minio-secret` | minio, database | MINIO_ROOT_USER, MINIO_ROOT_PASSWORD |\n| `rabbitmq-secret` | database | RABBITMQ_USER, RABBITMQ_PASSWORD |\n| `oracle-credentials` | goapps-* | ORACLE_HOST, ORACLE_PORT, ORACLE_*_USER, ORACLE_*_PASSWORD |\n| `goapps-tls` | multiple | tls.crt, tls.key |\n| `grafana-admin-secret` | monitoring | admin-user, admin-password |\n| `grafana-smtp-secret` | monitoring | password |\n| `s3-cloud-credentials` | database | S3_ENDPOINT, S3_BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY |\n| `ghcr-secret` | goapps-* | Docker registry credentials |\n| `prometheus-basic-auth` | monitoring | auth (htpasswd format) |\n\n### Security Rules\n\n1. **Never commit secrets to Git** - Use `kubectl create secret` manually\n2. **Use separate passwords per environment** - Staging ≠ Production\n3. **Rotate credentials regularly** - Especially for production\n4. **Limit secret access** - Use RBAC appropriately\n5. **Monitor secret access** - Enable audit logging\n\n---\n\n## CI/CD Pipeline\n\n### GitHub Actions Workflows\n\n#### 1. CI Workflow (`ci.yml`)\n\nTriggers: Push to `main`/`develop`, PRs to `main`\n\nSteps:\n1. Validate base kustomizations\n2. Validate staging/production overlays\n3. Validate service manifests\n4. Lint YAML files (yamllint)\n5. Security scan (Trivy)\n\n#### 2. ArgoCD Sync (`sync-argocd.yml`)\n\nTriggers: Push to `main` (paths: base/**, overlays/**, services/**, argocd/**)\n\nSteps:\n1. Sync staging applications (automatic)\n2. Sync production applications (manual dispatch only)\n3. Wait for applications to be healthy\n\n#### 3. Health Check (`health-check.yml`)\n\nTriggers: Scheduled (cron), manual dispatch\n\nSteps:\n1. Check cluster connectivity\n2. Verify critical pods status\n3. Report health status\n\n### Self-Hosted Runners\n\nRunner labels:\n- `staging`: Runs on staging VPS\n- `production`: Runs on production VPS  \n- `goapps-runner`: Common label for all runners\n\nInstall runner:\n```bash\n./scripts/install-runner.sh\n```\n\n---\n\n## Troubleshooting\n\n### Common Issues \u0026 Solutions\n\n#### Pod CrashLoopBackOff\n\n```bash\n# Check pod events\nkubectl describe pod \u003cpod-name\u003e -n \u003cnamespace\u003e\n\n# Check previous logs\nkubectl logs \u003cpod-name\u003e -n \u003cnamespace\u003e --previous\n```\n\n#### Database Connection Issues\n\n```bash\n# Test PostgreSQL directly\nkubectl exec -it postgres-0 -n database -- \\\n  psql -U goapps_admin -d goapps -c \"SELECT 1\"\n\n# Test via PgBouncer\nkubectl run test-pg --rm -it --image=postgres:18-alpine -- \\\n  psql -h pgbouncer.database -U goapps_admin -d goapps\n```\n\n#### Ingress Not Working\n\n```bash\n# Check ingress status\nkubectl get ingress -A\nkubectl describe ingress \u003cname\u003e -n \u003cnamespace\u003e\n\n# Check NGINX controller logs\nkubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx\n```\n\n#### ArgoCD Sync Failed\n\n```bash\n# Get ArgoCD password\nkubectl -n argocd get secret argocd-initial-admin-secret \\\n  -o jsonpath=\"{.data.password}\" | base64 -d\n\n# Check application status\nkubectl get applications -n argocd\n\n# Force sync\nargocd app sync \u003capp-name\u003e --force\n```\n\n#### Storage Issues\n\n```bash\n# Check PVC status\nkubectl get pvc -A\n\n# Check PV status\nkubectl get pv\n\n# Describe PVC\nkubectl describe pvc \u003cpvc-name\u003e -n \u003cnamespace\u003e\n```\n\n### Useful Commands\n\n```bash\n# Cluster overview\nmake status\n\n# PostgreSQL logs\nmake logs-postgres\n\n# ArgoCD logs\nmake logs-argocd\n\n# Port forward Grafana\nmake port-forward-grafana\n\n# Port forward ArgoCD\nmake port-forward-argocd\n\n# Validate manifests\nmake lint\n\n# Manual backup\nmake backup-now\n```\n\n---\n\n## Related Documentation\n\n| Document | Path | Description |\n|----------|------|-------------|\n| Development Rules | [RULES.md](./RULES.md) | Conventions and guidelines |\n| Contributing Guide | [CONTRIBUTING.md](./CONTRIBUTING.md) | How to contribute |\n| Deployment Guide | [docs/deployment-guide.md](./docs/deployment-guide.md) | Step-by-step deployment |\n| VPS Reset Guide | [docs/vps-reset-guide.md](./docs/vps-reset-guide.md) | Complete VPS reset |\n| License | [LICENSE](./LICENSE) | Proprietary license |\n\n### Issue Templates\n\n| Template | Description |\n|----------|-------------|\n| [🐛 Bug Report](.github/ISSUE_TEMPLATE/bug_report.md) | Report bugs or infrastructure issues |\n| [✨ Feature Request](.github/ISSUE_TEMPLATE/feature_request.md) | Request new features or enhancements |\n| [🚀 New Service](.github/ISSUE_TEMPLATE/new_service.md) | Request deployment for new service |\n| [🚨 Incident Report](.github/ISSUE_TEMPLATE/incident_report.md) | Report production incidents |\n\n### Pull Request Template\n\nAll PRs use the standard template: [PULL_REQUEST_TEMPLATE.md](.github/PULL_REQUEST_TEMPLATE.md)\n\n---\n\n## Support \u0026 Contact\n\n- **Team**: GoApps DevOps\n- **Organization**: PT Mutu Gading Tekstil\n- **Repository Issues**: [GitHub Issues](https://github.com/mutugading/goapps-infra/issues)\n\n---\n\n## License\n\nThis project is proprietary software. See the [LICENSE](./LICENSE) file for details.\n\n**© 2024-2026 PT Mutu Gading Tekstil. All Rights Reserved.**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmutugading%2Fgoapps-infra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmutugading%2Fgoapps-infra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmutugading%2Fgoapps-infra/lists"}