{"id":42946281,"url":"https://github.com/cloudnative-pg/chaos-testing","last_synced_at":"2026-01-30T20:36:29.262Z","repository":{"id":316185022,"uuid":"1062317121","full_name":"cloudnative-pg/chaos-testing","owner":"cloudnative-pg","description":"Chaos testing project to enhance CloudnativePG's resilience and fault tolerance","archived":false,"fork":false,"pushed_at":"2025-12-11T14:11:14.000Z","size":324,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-01-30T17:49:47.635Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudnative-pg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":"GOVERNANCE.md","roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-23T05:24:29.000Z","updated_at":"2026-01-05T14:18:29.000Z","dependencies_parsed_at":"2025-09-23T07:23:18.109Z","dependency_job_id":"ff81a63a-459e-4156-804d-a59d42957601","html_url":"https://github.com/cloudnative-pg/chaos-testing","commit_stats":null,"previous_names":["cloudnative-pg/chaos-testing"],"tags_count":0,"template":false,"template_full_name":"cloudnative-pg/cnpg-template","purl":"pkg:github/cloudnative-pg/chaos-testing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudnative-pg%2Fchaos-testing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudnative-pg%2Fchaos-testing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudnative-pg%2Fchaos-testing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudnative-pg%2Fchaos-testing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudnative-pg","download_url":"https://codeload.github.com/cloudnative-pg/chaos-testing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudnative-pg%2Fchaos-testing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28918554,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T20:25:28.696Z","status":"ssl_error","status_checked_at":"2026-01-30T20:25:13.426Z","response_time":66,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-30T20:36:29.155Z","updated_at":"2026-01-30T20:36:29.235Z","avatar_url":"https://github.com/cloudnative-pg.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CloudNativePG Chaos Testing with Jepsen\n\n![CloudNativePG Logo](logo/cloudnativepg.png)\n\nProduction-ready Jepsen and Litmus chaos automation for CloudNativePG (CNPG) clusters.\n\n---\n\n## 🚀 Quick Start\n\n**Want to run chaos testing immediately?** Follow these streamlined steps:\n\n0. **Clone this repo** → Get the chaos experiments and scripts (section 0)\n1. **Setup cluster** → Bootstrap CNPG Playground (section 1)\n2. **Install CNPG** → Deploy operator + sample cluster (section 2)\n3. **Install Litmus** → Install operator, experiments, and RBAC (sections 3, 3.5, 3.6)\n4. **Smoke-test chaos** → Run the quick pod-delete check without monitoring (section 4)\n5. **Add monitoring** → Install Prometheus for probe validation (section 5; required before section 6 with probes enabled)\n6. **Run Jepsen** → Full consistency testing layered on chaos (section 6)\n\n**First time users:** Use section 4 as a smoke test without Prometheus, then return to section 5 to install monitoring before running the Jepsen workflow in section 6.\n\n---\n\n## ✅ Prerequisites\n\n- Linux/macOS shell with `bash`, `git`, `curl`, `jq`, and internet access.\n- Container + Kubernetes tooling: Docker **or** Podman, the [Kind CLI](https://kind.sigs.k8s.io/) tool, `kubectl`, `helm`, the [`kubectl cnpg` plugin](https://cloudnative-pg.io/documentation/current/kubectl-plugin/) binary, and the [`cmctl` utility](https://cert-manager.io/docs/reference/cmctl/) for cert-manager.\n- Install the CNPG plugin using kubectl krew (recommended):\n  ```bash\n  # Install or update to the latest version\n  kubectl krew update\n  kubectl krew install cnpg || kubectl krew upgrade cnpg\n  kubectl cnpg version\n  ```\n  \u003e **Alternative installation methods:**\n  \u003e\n  \u003e - For Debian/Ubuntu: Download `.deb` from [releases page](https://github.com/cloudnative-pg/cloudnative-pg/releases)\n  \u003e - For RHEL/Fedora: Download `.rpm` from [releases page](https://github.com/cloudnative-pg/cloudnative-pg/releases)\n  \u003e - See [official installation docs](https://cloudnative-pg.io/documentation/current/kubectl-plugin) for all methods\n- Optional but recommended: `kubectx`, `stern`, `kubectl-view-secret` (see the [CNPG Playground README](https://github.com/cloudnative-pg/cnpg-playground#prerequisites) for a complete list).\n- **Disk Space:** Minimum **30GB** free disk space recommended:\n  - Kind cluster nodes: ~5GB\n  - Container images: ~5GB (first run with image pull)\n  - Prometheus/MongoDB storage: ~10GB\n  - Jepsen results + logs: ~5GB\n  - Buffer for growth: ~5GB\n- Sufficient local resources for a multi-node Kind cluster (≈8 CPUs / 12 GB RAM) and permission to run port-forwards.\n\nOnce the tooling is present, everything else is managed via repository scripts and Helm charts.\n\n---\n\n## ⚡ Setup and Configuration\n\n\u003e Follow these sections in order; each references the authoritative upstream documentation to keep this README concise.\n\n### 0. Clone the Chaos Testing Repository\n\n**First, clone this repository to access the chaos experiments and scripts:**\n\n```bash\ngit clone https://github.com/cloudnative-pg/chaos-testing.git\ncd chaos-testing\n```\n\nAll subsequent commands reference files in this repository (experiments, scripts, monitoring configs).\n\n### 1. Bootstrap the CNPG Playground\n\nThe upstream documentation provides detailed instructions for prerequisites and networking. Follow the setup instructions here: \u003chttps://github.com/cloudnative-pg/cnpg-playground#usage\u003e.\n\nDeploy the `cnpg-playground` project in a parallel folder to `chaos-testing`:\n\n```bash\ncd ..\ngit clone https://github.com/cloudnative-pg/cnpg-playground.git\ncd cnpg-playground\n./scripts/setup.sh eu         # creates kind-k8s-eu cluster\n```\n\nFollow the instructions on the screen. In particular, make sure that you:\n\n1. export the `KUBECONFIG` variable, as described\n2. set the correct context for kubectl\n\nFor example:\n\n```\nexport KUBECONFIG=\u003cPATH_TO_CNPG_PLAYGROUND\u003e/k8s/kube-config.yaml\nkubectl config use-context kind-k8s-eu\n```\n\nIf unsure, type:\n\n```\n./scripts/info.sh             # displays contexts and access information\n```\n\n### 2. Install CloudNativePG and Create the PostgreSQL Cluster\n\nWith the Kind cluster running, install the operator using the **kubectl cnpg plugin** as recommended in the [CloudNativePG Installation \u0026 Upgrades guide](https://cloudnative-pg.io/documentation/current/installation_upgrade/). This approach ensures you get the latest stable operator version:\n\n**In the `cnpg-playground` folder:**\n\n```bash\n# Install the latest operator version using the kubectl cnpg plugin\nkubectl cnpg install generate --control-plane | \\\n  kubectl --context kind-k8s-eu apply -f - --server-side\n\n# Verify the controller rollout\nkubectl --context kind-k8s-eu rollout status deployment \\\n  -n cnpg-system cnpg-controller-manager\n```\n\n**In the `chaos-testing` folder:**\n\n```bash\ncd ../chaos-testing\n# Create the pg-eu PostgreSQL cluster for chaos testing\nkubectl apply -f clusters/pg-eu-cluster.yaml\n\n# Verify cluster is ready (this will watch until healthy)\nkubectl get cluster pg-eu -w  # Wait until status shows \"Cluster in healthy state\"\n# Press Ctrl+C when you see: pg-eu   3       3   ready   XX m\n```\n\n### 3. Install Litmus Chaos\n\nLitmus 3.x separates the operator (via `litmus-core`) from the ChaosCenter UI (via `litmus` chart). Install both, then add the experiment definitions and RBAC:\n\n```bash\n# Add Litmus Helm repository\nhelm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/\nhelm repo update\n\n# Install litmus-core (operator + CRDs)\nhelm upgrade --install litmus-core litmuschaos/litmus-core \\\n  --namespace litmus --create-namespace \\\n  --wait --timeout 10m\n\n# Verify CRDs are installed\nkubectl get crd chaosengines.litmuschaos.io chaosexperiments.litmuschaos.io chaosresults.litmuschaos.io\n\n# Verify operator is running\nkubectl -n litmus get deploy litmus\nkubectl -n litmus wait --for=condition=Available deployment/litmus --timeout=5m\n```\n\n### 3.5. Install ChaosExperiment Definitions\n\nThe ChaosEngine requires ChaosExperiment resources to exist before it can run. Install the `pod-delete` experiment:\n\n```bash\n# Install from Chaos Hub (has namespace: default hardcoded, so override it)\nkubectl apply --namespace=litmus -f https://hub.litmuschaos.io/api/chaos/master?file=faults/kubernetes/pod-delete/fault.yaml\n\n# Verify experiment is installed\nkubectl -n litmus get chaosexperiments\n# Should show: pod-delete\n```\n\n### 3.6. Configure RBAC for Chaos Experiments\n\nApply the RBAC configuration and verify the service account has correct permissions:\n\n```bash\n# Apply RBAC (ServiceAccount, ClusterRole, ClusterRoleBinding)\nkubectl apply -f litmus-rbac.yaml\n\n# Verify the ServiceAccount exists in litmus namespace\nkubectl -n litmus get serviceaccount litmus-admin\n\n# Verify the ClusterRoleBinding points to correct namespace\nkubectl get clusterrolebinding litmus-admin -o jsonpath='{.subjects[0].namespace}'\n# Should output: litmus (not default)\n\n# Test permissions (optional)\nkubectl auth can-i delete pods --as=system:serviceaccount:litmus:litmus-admin -n default\n# Should output: yes\n```\n\n\u003e **Important:** The `litmus-rbac.yaml` ClusterRoleBinding must reference `namespace: litmus` in the subjects section. If you see errors like `\"litmus-admin\" cannot get resource \"chaosengines\"`, verify the namespace matches where the ServiceAccount exists.\n\n### 4. (Optional) Test Chaos Without Monitoring\n\nBefore setting up the full monitoring stack, you can verify chaos mechanics work independently:\n\n```bash\n# Apply the probe-free chaos engine (no Prometheus dependency)\nkubectl apply -f experiments/cnpg-jepsen-chaos-noprobes.yaml\n\n# Watch the chaos runner pod start (refreshes every 2s)\n# Press Ctrl+C once you see the runner pod appear\nwatch -n2 'kubectl -n litmus get pods | grep cnpg-jepsen-chaos-noprobes-runner'\n\n# Monitor CNPG pod deletions in real-time\nbash scripts/monitor-cnpg-pods.sh pg-eu default litmus kind-k8s-eu\n\n# Wait for chaos runner pod to be created, then check logs\nkubectl -n litmus wait --for=condition=ready pod -l chaos-runner-name=cnpg-jepsen-chaos-noprobes --timeout=60s \u0026\u0026 \\\nrunner_pod=$(kubectl -n litmus get pods -l chaos-runner-name=cnpg-jepsen-chaos-noprobes -o jsonpath='{.items[0].metadata.name}') \u0026\u0026 \\\nkubectl -n litmus logs -f \"$runner_pod\"\n\n# After completion, check the result (engine name differs)\nkubectl -n litmus get chaosresult cnpg-jepsen-chaos-noprobes-pod-delete -o jsonpath='{.status.experimentStatus.verdict}'\n# Should output: Pass (if probes are disabled) or Error (if Prometheus probes enabled but Prometheus not installed)\n\n# Clean up for next test\nkubectl -n litmus delete chaosengine cnpg-jepsen-chaos-noprobes\n```\n\n**What to observe:**\n\n- The runner pod starts and creates an experiment pod (`pod-delete-xxxxx`)\n- CNPG primary pods are deleted every 60 seconds\n- CNPG automatically promotes a replica to primary after each deletion\n- Deleted pods are recreated by the StatefulSet controller\n- The experiment runs for 10 minutes (TOTAL_CHAOS_DURATION=600)\n\n\u003e **Note:** Keep using `experiments/cnpg-jepsen-chaos-noprobes.yaml` until Section 5 installs Prometheus/Grafana. Once monitoring is online, switch to `experiments/cnpg-jepsen-chaos.yaml` (probes enabled) for full observability.\n\n### 5. Configure monitoring (Prometheus + Grafana)\n\nThe **cnpg-playground** provides a built-in monitoring stack with Prometheus and Grafana. From the cnpg-playground directory:\n\n```bash\ncd ../cnpg-playground\n./monitoring/setup.sh eu\n```\n\nThis script installs:\n\n- **Prometheus Operator** (in `prometheus-operator` namespace)\n- **Grafana Operator** with the official CloudNativePG dashboard (in `grafana` namespace)\n- Auto-configured for the `kind-k8s-eu` cluster\n\nOnce installation completes, create the PodMonitor to expose CNPG metrics:\n\n```bash\n# Switch back to chaos-testing directory\ncd ../chaos-testing\n\n# Apply CNPG PodMonitor\nkubectl apply -f monitoring/podmonitor-pg-eu.yaml\n\n# Verify PodMonitor\nkubectl get podmonitor pg-eu -o wide\n\n# Verify Prometheus is scraping CNPG metrics\nkubectl -n prometheus-operator port-forward svc/prometheus-operated 9090:9090 \u0026\ncurl -s --data-urlencode 'query=sum(cnpg_collector_up{cluster=\"pg-eu\"})' \"http://localhost:9090/api/v1/query\"\n```\n\n**Access Grafana dashboard:**\n\n```bash\nkubectl -n grafana port-forward svc/grafana-service 3000:3000\n\n# Open http://localhost:3000 with:\n#   Username: admin\n#   Password: admin (you'll be prompted to change on first login)\n```\n\nThe official CloudNativePG dashboard is pre-configured and available at: **Home → Dashboards → grafana → CloudNativePG**\n\n\u003e **Note:** If you recreate the `pg-eu` cluster, reapply the PodMonitor so Prometheus resumes scraping: `kubectl apply -f monitoring/podmonitor-pg-eu.yaml`\n\n\u003e ✅ **Required before section 6 (when probes are enabled):** Complete this monitoring setup so the Prometheus probes defined in `experiments/cnpg-jepsen-chaos.yaml` can succeed.\n\n#### Dependency on cnpg-playground\n\nThis project relies on cnpg-playground's monitoring implementation. Be aware of the following dependencies:\n\n**What we depend on**:\n\n- Script: `/path/to/cnpg-playground/monitoring/setup.sh`\n- Namespace: `prometheus-operator`\n- Service: `prometheus-operated` (created by Prometheus Operator for CR named `prometheus`)\n- Port: `9090` (Prometheus default)\n\n**If cnpg-playground monitoring changes**, you may need to update:\n\n- Prometheus endpoint in `experiments/cnpg-jepsen-chaos.yaml` (lines 89, 132, 148)\n- Service check in `.github/workflows/chaos-test-full.yml` (line 57)\n- Service check in `scripts/run-jepsen-chaos-test.sh` (line 279)\n\n**Troubleshooting**: If probes fail with connection errors:\n\n```bash\n# Verify the Prometheus service exists\nkubectl -n prometheus-operator get svc\n\n# If service name changed, update all probe endpoints\n# in experiments/cnpg-jepsen-chaos.yaml\n```\n\n### 6. Run the Jepsen chaos test\n\n```bash\n./scripts/run-jepsen-chaos-test.sh pg-eu app 600\n```\n\nThis script deploys Jepsen (`jepsenpg` image), applies the Litmus ChaosEngine (primary pod delete), monitors logs, collects results, and cleans up transient resources **automatically** (no manual exit needed - the script handles everything).\n\n**Prerequisites before running the script:**\n\n- Section 5 completed (Prometheus/Grafana running) so probes succeed.\n- Chaos workflow validated (run `experiments/cnpg-jepsen-chaos.yaml` once manually if you need to confirm Litmus + CNPG wiring).\n- Docker registry access to pull `ardentperf/jepsenpg` image (or pre-pulled into cluster).\n- `kubectl` context pointing to the playground cluster with sufficient resources.\n- **Increase max open files limit** if needed (required for Jepsen on some systems):\n  ```bash\n  ulimit -n 65536\n  ```\n  \u003e This may need to be configured in your container runtime or Kind cluster configuration if running in a containerized environment.\n\n**Script knobs:**\n\n- `LITMUS_NAMESPACE` (default `litmus`) – set if you installed Litmus in a different namespace.\n- `PROMETHEUS_NAMESPACE` (default `prometheus-operator`) – used to auto-detect the Prometheus service backing Litmus probes.\n- `JEPSEN_IMAGE` is pinned to `ardentperf/jepsenpg@sha256:4a3644d9484de3144ad2ea300e1b66568b53d85a87bf12aa64b00661a82311ac` for reproducibility. Update this digest only after verifying upstream releases.\n\n### 7. Inspect test results\n\n- All test results are stored under `logs/jepsen-chaos-\u003ctimestamp\u003e/`.\n- Quick validation commands:\n\n  ```bash\n  # Check Litmus chaos verdict (note: use -n litmus, not -n default)\n  kubectl -n litmus get chaosresult cnpg-jepsen-chaos-pod-delete \\\n     -o jsonpath='{.status.experimentStatus.verdict}'\n\n  # View full chaos result details\n  kubectl -n litmus get chaosresult cnpg-jepsen-chaos-pod-delete -o yaml\n\n  # Check probe results (if Prometheus was installed)\n  kubectl -n litmus get chaosresult cnpg-jepsen-chaos-pod-delete \\\n     -o jsonpath='{.status.probeStatuses}' | jq\n  ```\n\n- Archive `history.edn` and `chaos-results/chaosresult.yaml` for analysis or reporting.\n\n---\n\n## 📦 Results \u0026 logs\n\n- Each run creates a folder under `logs/jepsen-chaos-\u003ctimestamp\u003e/`.\n- Key files:\n  - `results/history.edn` → Jepsen operation history.\n  - `results/chaos-results/chaosresult.yaml` → Litmus verdict + probe output.\n- Quick checks:\n\n  ```bash\n  # Chaos results (note: namespace is 'litmus' by default)\n  kubectl -n litmus get chaosresult cnpg-jepsen-chaos-pod-delete \\\n     -o jsonpath='{.status.experimentStatus.verdict}'\n  ```\n\n---\n\n## 🔗 References \u0026 more docs\n\n- CNPG Playground: https://github.com/cloudnative-pg/cnpg-playground\n- CloudNativePG Installation \u0026 Upgrades: https://cloudnative-pg.io/documentation/current/installation_upgrade/\n- Litmus Helm chart: https://github.com/litmuschaos/litmus-helm/\n- kube-prometheus-stack: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack\n- CNPG Grafana dashboards: https://github.com/cloudnative-pg/grafana-dashboards\n- License: Apache 2.0 (see `LICENSE`).\n\n---\n\n## 🔧 Monitoring and Observability Tools\n\n### Real-time Monitoring Script\n\nWatch CNPG pods, chaos engines, and cluster events during experiments:\n\n```bash\n# Monitor pod deletions and failovers in real-time\nbash scripts/monitor-cnpg-pods.sh \u003ccluster-name\u003e \u003ccnpg-namespace\u003e \u003cchaos-namespace\u003e \u003ckube-context\u003e\n\n# Example\nbash scripts/monitor-cnpg-pods.sh pg-eu default litmus kind-k8s-eu\n```\n\n**What it shows:**\n\n- CNPG pod status with role labels (primary/replica)\n- Active ChaosEngines in the chaos namespace\n- Recent Kubernetes events (pod deletions, promotions, etc.)\n- Updates every 2 seconds\n\n## 📚 Additional Resources\n\n- **CNPG Documentation:** \u003chttps://cloudnative-pg.io/documentation/\u003e\n- **Litmus Documentation:** \u003chttps://docs.litmuschaos.io/\u003e\n- **Jepsen Documentation:** \u003chttps://jepsen.io/\u003e\n- **PostgreSQL High Availability:** \u003chttps://www.postgresql.org/docs/current/high-availability.html\u003e\n\n---\n\nFollow the sections above to execute chaos tests. Review the logs for analysis, and consult the `/archive` directory for additional documentation if needed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudnative-pg%2Fchaos-testing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudnative-pg%2Fchaos-testing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudnative-pg%2Fchaos-testing/lists"}