{"id":13820007,"url":"https://github.com/jonog/redalert","last_synced_at":"2025-05-16T07:32:42.523Z","repository":{"id":23551948,"uuid":"26919214","full_name":"jonog/redalert","owner":"jonog","description":":rocket: continuously test all the things - trigger alerts on failure :boom:","archived":false,"fork":false,"pushed_at":"2018-06-08T12:58:31.000Z","size":284,"stargazers_count":91,"open_issues_count":11,"forks_count":16,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-11-19T19:43:48.203Z","etag":null,"topics":["go","metrics","monitoring","ping"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonog.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-11-20T15:45:07.000Z","updated_at":"2024-10-02T05:49:23.000Z","dependencies_parsed_at":"2022-07-10T09:30:42.472Z","dependency_job_id":null,"html_url":"https://github.com/jonog/redalert","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonog%2Fredalert","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonog%2Fredalert/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonog%2Fredalert/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonog%2Fredalert/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonog","download_url":"https://codeload.github.com/jonog/redalert/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254488447,"owners_count":22079429,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","metrics","monitoring","ping"],"created_at":"2024-08-04T08:00:56.807Z","updated_at":"2025-05-16T07:32:41.671Z","avatar_url":"https://github.com/jonog.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"## Redalert\n\n[![Circle CI](https://circleci.com/gh/jonog/redalert.svg?style=svg)](https://circleci.com/gh/jonog/redalert)\n\n[![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=redalert\u0026templateURL=https://s3-ap-southeast-2.amazonaws.com/redalert-cloudformation/redalert.yml)\n\nFor monitoring your infrastructure and sending notifications if stuff is not ok.\n(e.g. pinging your websites/APIs via HTTP GET at specified intervals, and alerting you if there is downtime).\n\n\u003cimg src=\"https://cloud.githubusercontent.com/assets/1314353/23970218/34e46d0a-0a1d-11e7-8af0-6db94f69f0a9.png\" width=\"500\"\u003e\n\n### Features\n\n#### Checks\n* *Website monitoring* \u0026 latency measurement (check type: `web-ping`)\n* *Server metrics* from local machine (check type: `scollector`)\n* *Docker container metrics* (check type: `docker-stats`)\n* *Docker container metrics* from remote host via SSH (check type: `remote-docker`)\n* *Postgres counts/stats* via SQL queries (check type: `postgres`)\n* *TCP connectivity monitoring* \u0026 latency measurement (check type: `tcp`)\n* *Execute local commands* \u0026 capture output (check type: `command`)\n* *Execute remote commands via SSH* \u0026 capture output (check type: `remote-command`)\n* *Run test suite and capture report metrics* via `JUnit XML` format (check type: `test-report`)\n\nChecks will happen at specified intervals or explicit trigger (i.e. trigger check API endpoint).\n\n#### Dashboard and Alerts\n* Alert notifications available on several channels:\n  * sending email (`gmail`)\n  * sending SMS (`twilio`)\n  * posting a message to Slack (`slack`)\n  * unix stream (`stderr`)\n* Provides ping status \u0026 latency info to `stdout`.\n* Adjustable back-off after a check fails (constant, linear, exponential - see notes below).\n* Includes a web UI as indicated by the screenshot above. (visit localhost:8888/, configure port via cli flag)\n* Triggers a failure alert (`redalert`) when a check is failing, and a recovery alert (`greenalert`) when the check has recovered (e.g. a successful ping, following a failing ping).\n* Triggers an alert when specified metric is above/below threshold.\n\n#### Assertions\n* Assertions are used to define criteria for checks to pass or fail:\n* Assert on metrics\n  * source: `metric`\n  * `\u003e` or `greater than`\n  * `\u003e=` or `greater than or equal`\n  * `\u003c` or `less than`\n  * `\u003c=` or `less than or equal`\n  * `==` or `=` or `equals`\n* Assert on metadata\n  * source: `metadata`\n  * `web-ping` returns `status_code`\n* Assert on response\n  * source: `text`\n  * source: `json`\n\n#### API\n\n| Endpoint | Description |\n| --- | --- |\n| `GET /v1/stats` | Retrieve stats for all checks |\n| `POST /v1/checks/{check_id}/disable` | Disable check |\n| `POST /v1/checks/{check_id}/enable` | Enable check |\n| `POST /v1/checks/{check_id}/trigger` | Trigger check |\n\n\n### Design\n\n```\n\n         ┌──────────────────────────────┐\n         │                              │\n   ┌────▶│     Redalert Check Flow      │\n   │     │                              │\n   │     └──────────────────────────────┘\n   │                    │\n   │          @interval or -\u003etrigger   ┌──────────────────────┐\n   │                    │            ┌▶│  error during check  │\n   │                    ▼            │ └──────────────────────┘\n   │        ┌──────────────────────┐ │ ┌──────────────────────┐\n   │        │  is check failing?   │─┤ │  failing assertions  │\n   │        └──────────────────────┘ │ │     * metrics *      │\n   │                    │            └▶│     * metadata *     │\n   │          ┌───YES───┴───NO────┐    │     * response *     │\n   │          │                   │    └──────────────────────┘\n   │          ▼                   ▼\n   │  ┌───────────────┐   ┌───────────────┐\n   │  │send alerts via│   │   is check    │\n   │  │   notifiers   │   │  recovering?  │\n   │  └───────────────┘   └───────────────┘\n   │  ┌───────────────┐          YES\n   │  │adjust backoff │           │\n   │  └───────────────┘           ▼\n   │          │           ┌───────────────┐\n   │          │           │send alerts via│\n   │          │           │   notifiers   │\n   │          │           └───────────────┘\n   │          │           ┌───────────────┐\n   │          │           │ reset backoff │\n   │          │           └───────────────┘\n   │          │                   │\n   │          ▼                   ▼\n   │         ┌──────────────────────┐\n   └─────────│    Event Storage     │\n             └──────────────────────┘\n```\n\n#### Screenshots\n![](https://cloud.githubusercontent.com/assets/1314353/5157264/edb21476-733a-11e4-8452-4b96b443f7ee.jpg)\n\n### Getting started\nRun via Docker:\n```\ndocker run -d -P -v /path/to/config.json:/config.json jonog/redalert\n```\nQuick bootstrap example:\n```\ncurl https://gist.githubusercontent.com/jonog/32c953aedf03edf71acaef53d89ce785/raw/e87f7e933165574e1d441781465223bfe6c3f1aa/sample_redalert_config.json \u003e /tmp/sample_redalert_config.json \u0026\u0026 \\\n    docker run -d -P -v /tmp/sample_redalert_config.json:/config.json --name test_redalert jonog/redalert \u0026\u0026 \\\n    open \"http://$(docker port test_redalert 8888)\"\n```\n\n\n\n#### Usage\nGet started with the `redalert` command:\n```\nUsage:\n  redalert [command]\n\nAvailable Commands:\n  checks      List checks\n  config-sync Sync file and database configurations\n  server      Run checks and server stats\n  version     Print the version number of Redalert\n\nFlags:\n  -d, --config-db string     config database url\n  -f, --config-file string   config file (default \"config.json\")\n  -s, --config-s3 string     config S3\n  -u, --config-url string    config url\n  -h, --help                 help for redalert\n  -p, --port int             port to run web server (default 8888)\n  -r, --rpc-port int         port to run RPC server (default 8889)\n\nUse \"redalert [command] --help\" for more information about a command.\n```\n\n#### Configuration\n\nConfigure servers to monitor \u0026 alert settings via a configuration file:\n* a local file (specified by `-f` or `--config-file`) - defaults to `config.json`\n* a file remotely accessible via HTTP (specified by `-u` or `--config-url`)\n* a file hosted in an AWS S3 bucket (specified by `-s` or `--config-s3`)\n\nTODO: document Postgres configuration option\n\n##### Example config.json\n```\n{\n   \"checks\":[\n      {\n         \"name\":\"Google\",\n         \"type\": \"web-ping\",\n         \"config\": {\n            \"address\":\"http://google.com\"\n         },\n         \"send_alerts\": [\"stderr\"],\n         \"backoff\": {\n            \"type\": \"constant\",\n            \"interval\": 10\n         },\n         \"assertions\": [\n             {\n                 \"comparison\": \"==\",\n                 \"identifier\": \"status_code\",\n                 \"source\": \"metadata\",\n                 \"target\": \"200\"\n             }\n         ]\n      }\n   ],\n   \"notifications\": []\n}\n```\n\n##### Example Larger config.json\n```\n{\n    \"checks\": [\n        {\n            \"name\": \"Demo HTTP Status Check\",\n            \"type\": \"web-ping\",\n            \"config\": {\n                \"address\": \"http://httpstat.us/200\",\n                \"headers\": {\n                    \"X-Api-Key\": \"ABCD1234\"\n                }\n            },\n            \"send_alerts\": [\n                \"stderr\"\n            ],\n            \"backoff\": {\n                \"interval\": 10,\n                \"type\": \"constant\"\n            },\n            \"assertions\": [\n                {\n                    \"comparison\": \"==\",\n                    \"identifier\": \"status_code\",\n                    \"source\": \"metadata\",\n                    \"target\": \"200\"\n                }\n            ]\n        },\n        {\n            \"name\": \"Demo Response Check\",\n            \"type\": \"web-ping\",\n            \"config\": {\n                \"address\": \"http://httpstat.us/400\"\n            },\n            \"send_alerts\": [\n                \"stderr\",\n                \"email\",\n                \"chat\",\n                \"sms\"\n            ],\n            \"backoff\": {\n                \"interval\": 10,\n                \"type\": \"linear\"\n            },\n            \"assertions\": [\n                {\n                    \"comparison\": \"less than\",\n                    \"identifier\": \"latency\",\n                    \"source\": \"metric\",\n                    \"target\": \"1100\"\n                },\n                {\n                    \"comparison\": \"==\",\n                    \"identifier\": \"status_code\",\n                    \"source\": \"metadata\",\n                    \"target\": \"400\"\n                },\n                {\n                    \"comparison\": \"==\",\n                    \"source\": \"text\",\n                    \"target\": \"400 Bad Request\"\n                }\n            ],\n            \"verbose_logging\": true\n        },\n        {\n            \"name\": \"Demo Exponential Backoff\",\n            \"type\": \"web-ping\",\n            \"config\": {\n                \"address\": \"http://httpstat.us/200\"\n            },\n            \"send_alerts\": [\n                \"stderr\"\n            ],\n            \"backoff\": {\n                \"interval\": 10,\n                \"multiplier\": 2,\n                \"type\": \"exponential\"\n            },\n            \"assertions\": [\n                {\n                    \"comparison\": \"==\",\n                    \"identifier\": \"status_code\",\n                    \"source\": \"metadata\",\n                    \"target\": \"500\"\n                }\n            ]\n        },\n        {\n            \"name\": \"Docker Redis\",\n            \"type\": \"tcp\",\n            \"config\": {\n                \"host\": \"192.168.99.100\",\n                \"port\": 1001\n            },\n            \"send_alerts\": [\n                \"stderr\"\n            ],\n            \"backoff\": {\n                \"interval\": 10,\n                \"type\": \"constant\"\n            }\n        },\n        {\n            \"name\": \"Docker stats\",\n            \"type\": \"docker-stats\",\n            \"config\": {},\n            \"send_alerts\": [\n                \"stderr\"\n            ],\n            \"backoff\": {\n                \"interval\": 30,\n                \"type\": \"linear\"\n            }\n        },\n        {\n            \"name\": \"production-docker-host\",\n            \"type\": \"remote-docker\",\n            \"config\": {\n                \"host\": \"ec2-xx-xxx-xx-xxx.ap-southeast-1.compute.amazonaws.com\",\n                \"user\": \"ubuntu\"\n            },\n            \"send_alerts\": [\n                \"stderr\"\n            ],\n            \"backoff\": {\n                \"interval\": 5,\n                \"type\": \"linear\"\n            }\n        },\n        {\n            \"name\": \"scollector-metrics\",\n            \"type\": \"scollector\",\n            \"config\": {\n                \"host\": \"hostname\"\n            },\n            \"send_alerts\": [\n                \"stderr\"\n            ],\n            \"backoff\": {\n                \"interval\": 15,\n                \"type\": \"constant\"\n            }\n        },\n        {\n            \"name\": \"production-db\",\n            \"type\": \"postgres\",\n            \"config\": {\n                \"connection_url\": \"postgres://user:pass@localhost:5432/dbname?sslmode=disable\",\n                \"metric_queries\": [\n                    {\n                        \"metric\": \"client_count\",\n                        \"query\": \"select count(*) from clients\"\n                    }\n                ]\n            },\n            \"send_alerts\": [\n                \"stderr\"\n            ],\n            \"backoff\": {\n                \"interval\": 120,\n                \"type\": \"linear\"\n            }\n        },\n        {\n            \"name\": \"README size\",\n            \"type\": \"command\",\n            \"config\": {\n                \"command\": \"cat README.md | wc -l\",\n                \"output_type\": \"number\"\n            },\n            \"send_alerts\": [\n                \"stderr\"\n            ],\n            \"backoff\": {\n                \"interval\": 10,\n                \"type\": \"constant\"\n            }\n        },\n        {\n            \"name\": \"List files\",\n            \"type\": \"command\",\n            \"config\": {\n                \"command\": \"ls\"\n            },\n            \"send_alerts\": [\n                \"stderr\"\n            ],\n            \"backoff\": {\n                \"interval\": 10,\n                \"type\": \"constant\"\n            }\n        },\n        {\n            \"name\": \"SHH into docker-alpine-sshd\",\n            \"type\": \"remote-command\",\n            \"config\": {\n                \"command\": \"uptime\",\n                \"ssh_auth_options\": {\n                  \"user\": \"root\",\n                  \"password\": \"root\",\n                  \"host\": \"localhost\",\n                  \"port\": 2222\n                }\n            },\n            \"send_alerts\": [\n                \"stderr\"\n            ],\n            \"assertions\": [\n                {\n                    \"comparison\": \"==\",\n                    \"identifier\": \"exit_status\",\n                    \"source\": \"metadata\",\n                    \"target\": \"0\"\n                }\n            ]\n        },\n        {\n            \"name\": \"Run Smoke Tests\",\n            \"type\": \"test-report\",\n            \"config\": {\n                \"command\": \"./run-smoke-tests.sh\"\n            },\n            \"send_alerts\": [\n                \"stderr\"\n            ],\n            \"assertions\": [\n                {\n                    \"comparison\": \"==\",\n                    \"identifier\": \"status\",\n                    \"source\": \"metadata\",\n                    \"target\": \"PASSING\"\n                }\n            ]\n        }\n    ],\n    \"notifications\": [\n        {\n            \"name\": \"email\",\n            \"type\": \"gmail\",\n            \"config\": {\n                \"notification_addresses\": \"\",\n                \"pass\": \"\",\n                \"user\": \"\"\n            }\n        },\n        {\n            \"name\": \"chat\",\n            \"type\": \"slack\",\n            \"config\": {\n                \"channel\": \"#general\",\n                \"icon_emoji\": \":rocket:\",\n                \"username\": \"redalert\",\n                \"webhook_url\": \"\"\n            }\n        },\n        {\n            \"name\": \"sms\",\n            \"type\": \"twilio\",\n            \"config\": {\n                \"account_sid\": \"\",\n                \"auth_token\": \"\",\n                \"notification_numbers\": \"\",\n                \"twilio_number\": \"\"\n            }\n        }\n    ],\n    \"preferences\": {\n        \"notifications\": {\n          \"fail_count_alert_threshold\": 2,\n          \"repeat_fail_alerts\": false\n        }\n    }\n}\n\n```\n\nBuild and run (capture stderr).\n```\ngo build\n\n./redalert 2\u003e errors.log\n```\n\n#### Notification Preferences\n* `fail_count_alert_threshold` controls sending an alert, only after N fails (defaults to 1)\n* `repeat_fail_alerts` controls whether fail alerts are repeated, on consecutive failing checks (defaults to false)\n```\n\"preferences\": {\n  \"notifications\": {\n    \"fail_count_alert_threshold\": 2,\n    \"repeat_fail_alerts\": false\n  }\n}\n```\n\n#### Backoffs\nWhen a server check fails - the next check will be delayed according to the back-off algorithm. By default, there is no delay (i.e. `constant` back-off), with a default interval of 10 seconds between checks. When a failing server returns to normal, the check frequency returns to its original value.\n\n##### Constant\nPinging interval will remain constant. i.e. will not provide any back-off after failure.\n\n##### Linear\nThe pinging interval upon failure will be extended linearly. i.e. `failure count x pinging interval`.\n\n##### Exponential\nWith each failure, the subsequent check will be delayed by the last delayed amount, times a multiplier, resulting in time between checks exponentially increasing. The `multiplier` is set to 2 by default.\n\n#### Note for Gmail\nIf there are errors sending email via gmail - enable `Access for less secure apps` under Account permissions @ https://www.google.com/settings/u/2/security\n\n### Deployment\n\n#### CloudFormation Stacks\n\nSee [redalert-cloudformation](https://github.com/jonog/redalert-cloudformation)\n\n##### EC2 \u0026 ELB\n[![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=redalert\u0026templateURL=https://s3-ap-southeast-2.amazonaws.com/redalert-cloudformation/redalert.yml)\n\n##### EC2 \u0026 ELB \u0026 S3 config\n[![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=redalert\u0026templateURL=https://s3-ap-southeast-2.amazonaws.com/redalert-cloudformation/redalert.yml)\n\n\n### Development\n\n#### Setup\nDependencies:\n* Go dependency manager - [glide](https://github.com/Masterminds/glide)\n* Embedding static assets into binary - [go.rice](https://github.com/GeertJohan/go.rice)\n* `protoc` for gRPC code generation - [gRPC](http://www.grpc.io/docs/quickstart/go.html)\n* Docker-machine for tests\n\n### Credits\nRocket emoji via https://github.com/twitter/twemoji\n\n### Next Features\nSee Github Issues [here](https://github.com/jonog/redalert/issues)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonog%2Fredalert","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonog%2Fredalert","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonog%2Fredalert/lists"}