{"id":13701862,"url":"https://github.com/tmobile/monarch","last_synced_at":"2025-07-26T14:39:27.874Z","repository":{"id":52602191,"uuid":"175462746","full_name":"tmobile/monarch","owner":"tmobile","description":"App-level Chaos Engineering","archived":false,"fork":false,"pushed_at":"2021-04-27T14:03:02.000Z","size":207,"stargazers_count":28,"open_issues_count":2,"forks_count":4,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-05-15T06:39:42.925Z","etag":null,"topics":["chaos-engineering","chaostoolkit-extension","cloudfoundry"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tmobile.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-13T16:52:17.000Z","updated_at":"2024-02-26T14:26:54.000Z","dependencies_parsed_at":"2022-09-07T06:01:02.916Z","dependency_job_id":null,"html_url":"https://github.com/tmobile/monarch","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/tmobile/monarch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmobile%2Fmonarch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmobile%2Fmonarch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmobile%2Fmonarch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmobile%2Fmonarch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tmobile","download_url":"https://codeload.github.com/tmobile/monarch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmobile%2Fmonarch/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267179573,"owners_count":24048371,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-26T02:00:08.937Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chaos-engineering","chaostoolkit-extension","cloudfoundry"],"created_at":"2024-08-02T21:00:25.523Z","updated_at":"2025-07-26T14:39:27.856Z","avatar_url":"https://github.com/tmobile.png","language":"Python","readme":"# ![Monarch](graphics/banner.png)\n\nThis is a series of tools for [Chaos Toolkit](https://chaostoolkit.org/) (CTK) which can perform targeted experiments\non applications deployed in [Cloud Foundry](https://www.cloudfoundry.org/).\n\n## Available Experiments\n\n- Block general network traffic\n    - Block all incoming traffic to the application\n    - Block all outgoing traffic from the application\n- Block service traffic\n    - Auto-detection of bound services and support for manually specified non-bound services \n    - Block all outgoing traffic from the application to one or more bound services\n    - Block all incoming traffic form the application to one or more bound services\n- Manipulate all network traffic from an application (including to its services)\n    - Latency\n    - Packet loss\n    - Packet duplication\n    - Packet corruption\n- Impose bandwidth restrictions\n    - Application download bandwidth shaping (using queuing)\n    - Application upload bandwidth limiting (using policing)\n- Perform network speedtest from within hosting containers\n- Crash one or more random application instances\n- Kill/start monit processes on hosting diego-cells\n\n\n## Setup\n\n### Build Dockerfile\nIt is recommended that you run Monarch with [Docker](https://www.docker.com/) which you can get\n[here](https://www.docker.com/products/docker-desktop). We have had some issues with cross-platform support for the\nunderlying CLIs.\n\nWith docker up and running, run the following within the root of the git repository:\n```bash\n# FIRST Run\ndocker build -t monarch .\ndocker run -it \\\n    --name monarch \\\n    -v C:\\Users\\\u003cusername\u003e\\Documents\\monarch\\config:/monarch/config # and create the needed files from within.\n    monarch\n\n# Subsequent Runs\ndocker start -ai monarch\n\n# Rebuild Image (You will loose information not in an attached volume)\ndocker container rm monarch\nyes | docker image prune\n# goto FIRST Run ;)\n``` \n\nNote that the config volume is optional and does not need to be mounted, however, even if you do not have any written\nalready, you should mount the volumes to prevent data loss when you destroy the container during the inevitable upgrade\nprocess. Also, if you plan to run tests, make sure to mount the testing config volume as well!\n\nFrom within the docker image, you may now use either the python shell to interact with monarch, or chaostollkit which is\ninstalled automatically when the image is built. You will need to login with cf-cli and bosh-cli before attempting to\nuse monarch.\n\n### Install Locally\nTo be used from your experiment, this package must first be installed in the Python environment where\n[chaostoolkit](https://chaostoolkit.org/) already exists. This package requires at least\n[Python](https://www.python.org/) version 3.5 (3.6 if using the chaostoolkit interfaces directly), so translate `python`\nas `python3` or `pyhton3.5` as appropriate for your OS.\n\nFrom within the source, run:  \n\n```bash\nsudo python setup.py install\n```\n\nOr to install for just your user:\n\n```bash\npython setup.py install --user\n```\n\nNow you should be able to import the package.\n\n```python\nimport monarch\nprint(monarch.__version__)\n```\n\n\n### Third-Party CLI Setup\nIn order to run the script, it will require that you have the\n[Cloud Foundry CLI](https://docs.cloudfoundry.org/cf-cli/install-go-cli.html) installed and the\n[BOSH CLI](https://bosh.io/docs/cli-v2-install/) installed. You will also need to be logged in to the Cloud Foundry CLI \nas a user with permission to access all apps which are to be targeted and logged in as and you will need to be login as\nan admin with the BOSH CLI. This is because the script requires ssh access to the bosh vms to make its changes and also\nprevents applications from needing SSH enabled.\n\n\n### Configuration File\nOnce the CLIs are ready, create (or modify the existing) configuration file. This file is only necessary for CLI use as\nit is included within the experiments for Chaos Toolkit. \n\n- `bosh`: Information about the bosh cli and environment\n    - `cmd`: The bosh-cli command.\n    - `env`: The environment name for the cf deployment (`-e env`).\n    - `cf-dep`: The cloud foundry deployment in the bosh environment.\n    - `cfdot-dc`: The diego-cell to use for `cfdot` queries.\n- `cf`: Information about the cf cli and environment\n    - `cmd`: The cf-cli command.\n- `container-port-whitelist`: List of node ports which should be ignored. These are the external ports on the\ndiego-cells.\n- `service-whitelist`: List of service types which should be ignored. These must be the names displayed in the cf-cli\nmarketplace.\n- `quantum`: The quantum to use when configuring qdisc perturbance. The recommended `6000` should work without issue.\n\nSample config.yml or `cfg` values for Chaos Toolkit.\n\n```yaml\nbosh:\n  cmd: bosh2 # bosh CLI to be used\n  env: bosh-lite # environment alias name or address \n  cf-dep: cf # Bosh deployment name\n  cfdot-dc: diego_cell/0\n  credentials: # Optional; CLI will need to be logged in already if not present\n    user: iamaperson\n    pswd: ideallysomethingsecure\n    cacert: |  # include as needed\n      -----BEGIN CERTIFICATE-----\n      ...\n      -----END CERTIFICATE-----\ncf:\n  cmd: cf\n  credentials: # Optional; CLI will need to be logged in already if not present\n    user: iamaperson\n    pswd: hopefullysomethingdifferent\n    api: cf.example.com\n    skip_ssl_validation: true # add as needed (false by default)\ncontainer-port-whitelist:\n - 22\n - 2222\nhost-port-whitelist: []\nservice-whitelist:\n - logger\nquantum: 6000\n#services:  # custom service definitions, not needed for bound services\n#  - name: google\n#    host: google.com\n#    ports:\n#     - ['tcp', 80]\n#     - ['tcp', 443]\n#     - ['icmp', 'all']\n```\n\n\n## Usage\nThere are two ways you can call these scripts. The first is the Python Shell which will allow you to manually block\nservices or applications and then unblock them at your leisure. The second is through the `actions` and `probes` which\nshould be called by Chaos Toolkit.\n\n### Chaos Toolkit Interface\nIf you have not installed the `monarch` package, then make sure you run Chaos Toolkit from this directory (the root of\nthis repository) using `python -m chaostoolkit run exp.json` or else the `monarch` module will not be found. Otherwise\njust use `chaos run exp.json` from any directory.\n\nCurrently, the Chaos Toolkit interface does not support saving information about what was targeted, which should be okay\nfor the time being as we have yet to observe Cloud Foundry moving app instances as a result of any of these actions.\nThough it is a good reason to be cautious of its use as it simply re-queries again when unblocking, so if something did\nmove, it will not remove the old rule in the location the app is no longer at. If you need to manually verify that all \nof the rules have been removed, you can go through each diego-cell in the Cloud Foundry deployment and run\n`iptables -L | grep DROP` to see if any rules are lingering. (This script *should* be the only source of `DROP` rules).\n\nThe following is a sample, Chaos-Toolkit experiment file to block all traffic to the application.\n\n```json\n{\n  \"version\": \"0.1.0\",\n  \"title\": \"Blocking spring-music makes it unreachable.\",\n  \"description\": \"This is a testing experiment to verify the script's block traffic function works.\",\n  \"tags\": [\"cloudfoundry\", \"bosh\", \"springboot\"],\n  \"configuration\": {\n    \"TODO\": \"Some of this needs to be part of the application configuration since the user of this would not know what the cli commands are for instance.\",\n    \"bosh\": {\n      \"cmd\": \"bosh2\",\n      \"env\": \"tt-stg02\",\n      \"cf-dep\": \"cf-da0ba81cb255ad93a508\",\n      \"cfdot-dc\": \"diego_cell/0\"\n    },\n    \"cf\": {\n      \"cmd\": \"cf\"\n    },\n    \"container-port-whitelist\": [22, 2222],\n    \"host-port-whitelist\": [],\n    \"service-whitelist\": [\"T-Logger\"],\n    \"quantum\": 6000\n  },\n  \"steady-state-hypothesis\": {\n    \"title\": \"We can access the application and other neighboring applications (This should fail because we block all traffic)\",\n    \"probes\": [\n      {\n        \"type\": \"probe\",\n        \"name\": \"spring-music-responds\",\n        \"tolerance\": 200,\n        \"provider\": {\n          \"type\": \"http\",\n          \"url\": \"http://spring-music-interested-bonobo.apps.tt-stg02.cf.t-mobile.com/\"\n        }\n      },\n      {\n        \"type\": \"probe\",\n        \"name\": \"spring-music2-responds\",\n        \"tolerance\": 200,\n        \"provider\": {\n          \"type\": \"http\",\n          \"url\": \"http://spring-music2-lean-sable.apps.tt-stg02.cf.t-mobile.com/\"\n        }\n      }\n    ]\n  },\n  \"method\": [\n    {\n      \"type\": \"action\",\n      \"name\": \"block-traffic\",\n      \"provider\": {\n        \"type\": \"python\",\n        \"module\": \"monarch.pcf.actions\",\n        \"func\": \"block_traffic\",\n        \"arguments\": {\n          \"org\": \"sys-tmo\",\n          \"space\": \"test\",\n          \"appname\": \"spring-music\"\n        }\n      }\n    }\n  ],\n  \"rollbacks\": [\n    {\n      \"type\": \"action\",\n      \"name\": \"unblock-traffic\",\n      \"provider\": {\n        \"type\": \"python\",\n        \"module\": \"monarch.pcf.actions\",\n        \"func\": \"unblock_traffic\",\n        \"arguments\": {\n          \"org\": \"sys-tmo\",\n          \"space\": \"test\",\n          \"appname\": \"spring-music\"\n        }\n      }\n    }\n  ]\n}\n``` \n\n### CLI Interface\nFor now, there is no CLI interface, instead use an interactive python shell session. See bleow.\n\n### From Python Shell\nExample session:\n```python\nfrom monarch.pcf.config import Config\nfrom monarch.pcf.app import App\n\nConfig().load_yaml('config/tt-stg02.yml')\n\napp = App.discover('sys-tmo', 'ce-service-registry', 'spring-music')\n\napp.block()\napp.unblock()\n\napp.crash_random_instance(2) # will require that you rediscover the app once CF brings a new container up\napp = App.discover('sys-tmo', 'ce-service-registry', 'spring-music')\n\napp.block_services('musicdb')\napp.unblock_services()\n\n```\n\n## Tests\nUnit tests are written with pytest and can be run with `./setup.py test`. Before running the tests, you will need to add\n`tests/config/app_test.yml` which is the same as the above configuration with the following appended:\n```yaml\n# ...\ntesting:\n  org: coolkids\n  space: ce-testing\n  appname: spring-music\n  push-app: true\n  db-market-name: p-mysql\n  db-plan: 100mb\n  db-instance-name: musicdb\n```\n\nYou will also need to include the credential sections for the bosh and cf cli configs. If push-app is true, it will\nexpect the `org` and `space` to be pre-existing, but deploy spring-music from scratch (meaning `db-instance-name` and\n`appname` should not already exist). It will perform cleanup after tests are done leaving the space in the state it was\noriginally.\n\nThese tests can be run from within docker using \n\n## License\nMonarch is open-sourced under the terms of section 7 of the Apache 2.0 license and is released AS-IS WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND.\n","funding_links":[],"categories":["3. Fault Injection"],"sub_categories":["Generic Tools"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftmobile%2Fmonarch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftmobile%2Fmonarch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftmobile%2Fmonarch/lists"}