{"id":23893861,"url":"https://github.com/iskorotkov/chaos-scheduler","last_synced_at":"2026-04-09T21:01:41.363Z","repository":{"id":39877674,"uuid":"315623322","full_name":"iskorotkov/chaos-scheduler","owner":"iskorotkov","description":"Service for automatic generation and scheduling of the chaos test workflows","archived":false,"fork":false,"pushed_at":"2023-03-07T01:56:58.000Z","size":1109,"stargazers_count":2,"open_issues_count":9,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-04T14:49:38.759Z","etag":null,"topics":["backend","chaos-engineering","docker","frontend","go","kubernetes","web"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iskorotkov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-24T12:20:13.000Z","updated_at":"2022-08-29T16:37:43.000Z","dependencies_parsed_at":"2024-06-21T02:06:15.751Z","dependency_job_id":"bc34ee91-160a-43b3-891b-03ce64bcf887","html_url":"https://github.com/iskorotkov/chaos-scheduler","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iskorotkov%2Fchaos-scheduler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iskorotkov%2Fchaos-scheduler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iskorotkov%2Fchaos-scheduler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iskorotkov%2Fchaos-scheduler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iskorotkov","download_url":"https://codeload.github.com/iskorotkov/chaos-scheduler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240277430,"owners_count":19775893,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","chaos-engineering","docker","frontend","go","kubernetes","web"],"created_at":"2025-01-04T14:49:47.000Z","updated_at":"2026-04-09T21:01:36.321Z","avatar_url":"https://github.com/iskorotkov.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Chaos Scheduler\n\nService for automatic generation and scheduling of the chaos test workflows.\n\n- [Chaos Scheduler](#chaos-scheduler)\n  - [Overview](#overview)\n    - [Failures](#failures)\n      - [Container failures](#container-failures)\n      - [Pod, part of deployment, deployment failures](#pod-part-of-deployment-deployment-failures)\n      - [Node failures](#node-failures)\n      - [Cluster failures](#cluster-failures)\n    - [Targets](#targets)\n    - [Workflows](#workflows)\n  - [Setup](#setup)\n    - [Dependencies](#dependencies)\n    - [Installation](#installation)\n    - [Env vars](#env-vars)\n  - [API](#api)\n    - [REST API](#rest-api)\n    - [Annotations](#annotations)\n  - [Development](#development)\n  - [Project structure](#project-structure)\n\n## Overview\n\nService generates workflows consisting of failures. Each failure has associated target. Each failure is something bad happening to the target (network loss, pod deletion, etc). Target is a part of the system under test (container, pod, deployment, etc). Service fetches list of potential targets from Kubernetes.\n\n### Failures\n\n**Failures selection**. Service picks failures in random order according to several constraints: max number of failures per stage, max chaos score per stage. It's possible to change failures selection by providing another seed.\n\n**Failure score**. Each failure has chaos score — amount of chaos that it will create once executed. Failure score depends on its scale and severity.\n\n#### Container failures\n\nContainer failures target a single container in a pod:\n\n- container CPU hog\n- container memory hog\n- container network corruption\n- container network duplication\n- container network latency\n- container network loss\n\n#### Pod, part of deployment, deployment failures\n\nPod failures target pods, parts of deployments (specified percent of pods in deployment) or entire deployments:\n\n- pod delete\n- pod I/O stress\n\n#### Node failures\n\nNode failures target entire nodes:\n\n- node CPU hog\n- node memory hog\n- node I/O stress\n\n#### Cluster failures\n\nCluster failures target entire cluster. No cluster failures currently supported.\n\n### Targets\n\n**Namespace**. Targets are fetched from Kubernetes. Service fetches only targets from specified namespace to avoid targeting production instances.\n\n**Target selection**. Service randomly picks appropriate target for each failure. It's possible to change target selection by providing another seed.\n\n### Workflows\n\n**Structure**. Each workflow consists of specified number of test stages; each stage consists of several steps (or actions); each step is a failure with associated target.\n\n**Execution order**. Stages are executed in order one at a time. Steps in each stage are executed at the same time.\n\n**Workflow preview**. Service allows previewing generated workflow without launching it.\n\n## Setup\n\n### Dependencies\n\nInstall dependencies before continuing:\n\n- Argo\n- Litmus Chaos\n\nYou also have to create `ServiceAccount` for Litmus Chaos and Argo.\n\n### Installation\n\nMake sure you have a Kubernetes cluster ready.\n\nInstall all dependencies and make sure they work correctly.\n\nTweak env var values in `deploy/scheduler.yaml` file for your environment (optional). Once finished, execute the command in the root folder:\n\n```shell\nkubectl -f deploy/scheduler.yaml\n```\n\n### Env vars\n\nService requires several env vars set (example values are provided in parentheses):\n\n- infrastructure:\n  - `ARGO_SERVER` — Argo server to use (`argo-server.argo.svc:2746`)\n  - `STAGE_MONITOR_IMAGE` — Docker image to use for monitoring crashes of target containers/pods (`iskorotkov/chaos-pods-monitor:v0.4.0`)\n  - `DEVELOPMENT` — whether in development or not (false)\n\n- target:\n\n  - `APP_NS` — namespace where system under test is located (`chaos-app`)\n  - `CHAOS_NS` — namespace where to create workflows (`litmus`)\n  - `APP_LABEL` — label to use for target selection (`app`)\n\n      Service looks for label `{APP_LABEL}: {VALUE}`, where `{VALUE}` will be the name of the target.\n\n      For example, when `APP_LABEL`=`app` the service will look for label `app: {VALUE}`. The target with label `app: nginx` will be named `nginx`.\n\n- workflow:\n\n  - `STAGE_DURATION` — duration of each stage (30s)\n  - `STAGE_INTERVAL` — duration between stages (30s)\n\n      Some failures take seconds to start and can't finish instantly. It's recommended to set interval to 30s or higher to avoid false positives in latter stages.\n\n- failures: [see struct definition](internal/config/config.go) and [example values in manifest file](deploy/scheduler.yaml).\n\n## API\n\n### REST API\n\n- api/v1/workflows\n  - preview — generate and preview test workflow (without launching it)\n  - create — generate and launch test workflow\n\n### Annotations\n\nService adds annotations to generated workflow steps according:\n\n| Key                         | Category       | Type   | Description                                                                |\n| --------------------------- | -------------- | ------ | -------------------------------------------------------------------------- |\n| chaosframework.com/version  | version        | semver | Version of annotations format                                              |\n| chaosframework.com/type     | classification | string | Type of template (failure, utility)                                        |\n| chaosframework.com/severity | classification | string | Failure severity (harmless, light, severe, critical)                       |\n| chaosframework.com/scale    | classification | string | Failure scale (container, pod, deployment part, deployment, node, cluster) |\n\n## Development\n\nTo build project:\n\n```shell\ngo build ./...\n```\n\nTo run tests:\n\n```shell\ngo test ./...\n```\n\n## Project structure\n\n- cmd\n  - scheduler — entry point\n- internal\n  - handlers — request handlers\n  - config — getting config from environment\n- pkg\n  - argo — argo client for executing workflows\n  - k8s — kubernetes client for fetching list of targets\n  - rx — random string, map and slice generation\n  - server — advanced request handling\n  - workflows — workflow creation and execution\n    - generate — test scenario creation\n\n      Scenario is a logical representation of a chaos test, while a workflow is a 1) practical representation of a chaos test; 2) scenario prepared to be executed.\n\n    - assemble — test workflow creation\n    - execution — test workflow execution\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiskorotkov%2Fchaos-scheduler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiskorotkov%2Fchaos-scheduler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiskorotkov%2Fchaos-scheduler/lists"}