{"id":13878294,"url":"https://github.com/grosser/kennel","last_synced_at":"2025-03-31T04:04:17.112Z","repository":{"id":27361503,"uuid":"113630604","full_name":"grosser/kennel","owner":"grosser","description":"Datadog monitors/dashboards/slos as code, avoid chaotic management via UI","archived":false,"fork":false,"pushed_at":"2024-09-11T16:44:04.000Z","size":2120,"stargazers_count":131,"open_issues_count":9,"forks_count":41,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-10-30T03:42:52.570Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grosser.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-09T00:55:22.000Z","updated_at":"2024-10-24T13:13:46.000Z","dependencies_parsed_at":"2024-11-12T22:00:58.657Z","dependency_job_id":"65a3d405-0e81-442a-910d-f78817997c7c","html_url":"https://github.com/grosser/kennel","commit_stats":{"total_commits":1010,"total_committers":34,"mean_commits":"29.705882352941178","dds":"0.22970297029702968","last_synced_commit":"76c08f15980e2b02e3fcfc601920c5920086a19f"},"previous_names":[],"tags_count":298,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grosser%2Fkennel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grosser%2Fkennel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grosser%2Fkennel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grosser%2Fkennel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grosser","download_url":"https://codeload.github.com/grosser/kennel/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244868510,"owners_count":20523586,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-06T08:01:45.435Z","updated_at":"2025-03-24T03:03:34.972Z","avatar_url":"https://github.com/grosser.png","language":"Ruby","funding_links":[],"categories":["Dashboards as code","Ruby"],"sub_categories":["Shell into containers"],"readme":"![](template/github/cage.jpg?raw=true)\n\nManage Datadog Monitors / Dashboards / Slos as code\n\n - DRY, searchable, audited, documented\n - Changes are PR reviewed and applied on merge\n - Updating shows diff before applying\n - Automated import of existing resources\n - Resources are grouped into projects that belong to teams and inherit tags\n - No copy-pasting of ids to create new resources\n - Automated cleanup when removing code\n - [Helpers](#helpers) for automating common tasks\n\n### Applying changes\n\n![](template/github/screen.png?raw=true)\n\n### Example code\n```Ruby\n# teams/foo.rb\nmodule Teams\n  class Foo \u003c Kennel::Models::Team\n    defaults(mention: -\u003e { \"@slack-my-team\" })\n  end\nend\n\n# projects/bar.rb\nclass Bar \u003c Kennel::Models::Project\n  defaults(\n    team: -\u003e { Teams::Foo.new }, # use mention and tags from the team\n    tags: -\u003e { super() + [\"project:bar\"] }, # unique tag for all project components\n    parts: -\u003e {\n      [\n        Kennel::Models::Monitor.new(\n          self, # the current project\n          type: -\u003e { \"query alert\" },\n          kennel_id: -\u003e { \"load-too-high\" }, # pick a unique name\n          name: -\u003e { \"Foobar Load too high\" }, # nice descriptive name that will show up in alerts and emails\n          message: -\u003e {\n            \u003c\u003c~TEXT\n              This is bad!\n              #{super()} # inserts mention from team\n            TEXT\n          },\n          query: -\u003e { \"avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} \u003e #{critical}\" },\n          critical: -\u003e { 20 }\n        )\n      ]\n    }\n  )\nend\n```\n\u003c!-- NOT IN template/Readme.md  --\u003e\n\n## Installation\n - create a new private `kennel` repo for your organization (do not fork this repo)\n - use the template folder as starting point:\n    ```Bash\n    git clone git@github.com:your-org/kennel.git\n    git clone git@github.com:grosser/kennel.git seed\n    mv seed/template/* kennel/\n    cd kennel \u0026\u0026 git add . \u0026\u0026 git commit -m 'initial'\n    ```\n - add a basic projects and teams so others can copy-paste to get started\n - setup CI build for your repo (travis and Github Actions supported)\n - uncomment `.travis.yml` section for datadog updates on merge (TODO: example setup for Github Actions)\n - follow `Setup` in your repos Readme.md\n\u003c!-- NOT IN --\u003e\n\n## Structure\n - `projects/` monitors/dashboards/etc scoped by project\n - `teams/` team definitions\n - `parts/` monitors/dashboards/etc that are used by multiple projects\n - `generated/` projects as json, to show current state and proposed changes in PRs\n\n## About the models\n\nKennel provides several classes which act as models for different purposes:\n\n* `Kennel::Models::Dashboard`, `Kennel::Models::Monitor`, `Kennel::Models::Slo`, `Kennel::Models::SyntheticTest`;\n  these models represent the various Datadog objects\n* `Kennel::Models::Project`; a container for a collection of Datadog objects\n* `Kennel::Models::Team`; provides defaults and values (e.g. tags, mentions) for the other models.\n\nAfter loading all the `*.rb` files under `projects/`, Kennel's starting point\nis to find all the subclasses of `Kennel::Models::Project`, and for each one,\ncreate an instance of that subclass (via `.new`) and then call `#parts` on that\ninstance. `parts` should return a collection of the Datadog-objects (Dashboard / Monitor / etc).\n\n### Model Settings\n\nEach of the models defines various settings; for example, a Monitor has `name`, `message`,\n`type`, `query`, `tags`, and many more.\n\nWhen defining a subclass of a model, one can use `defaults` to provide default values for\nthose settings:\n\n```Ruby\nclass MyMonitor \u003c Kennel::Models::Monitor\n  defaults(\n    name: \"Error rate\",\n    type: \"query alert\",\n    critical: 5.0,\n    query: -\u003e {\n      \"some datadog metric expression \u003e #{critical}\"\n    },\n    # ...\n  )\nend\n```\n\nThis is equivalent to defining instance methods of those names, which return those values:\n\n```Ruby\nclass MyMonitor \u003c Kennel::Models::Monitor\n  def name\n    \"Error rate\"\n  end\n\n  def type\n    \"query alert\"\n  end\n\n  def critical\n    5.0\n  end\n\n  def query\n    \"some datadog metric expression \u003e #{critical}\"\n  end\nend\n```\n\nexcept that `defaults` will complain if you try to use a setting name which doesn't\nexist. Note also that you can use either plain values (`critical: 5.0`), or procs\n(`query: -\u003e { ... }`). Using a plain value is equivalent to using a proc which returns\nthat same value; use whichever suits you best.\n\nWhen you _instantiate_ a model class, you can pass settings in the constructor, after\nthe project:\n\n```Ruby\nproject = Kennel::Models::Project.new\nmy_monitor = MyMonitor.new(\n  project,\n  critical: 10.0,\n  message: -\u003e {\n    \u003c\u003c~MESSAGE\n      Something bad is happening and you should be worried.\n\n      #{super()}\n    MESSAGE\n  },\n)\n```\n\nThis works just like `defaults` (it checks the setting names, and it accepts\neither plain values or procs), but it applies just to this instance of the class,\nrather than to the class as a whole (i.e. it defines singleton methods, rather\nthan instance methods).\n\nMost of the examples in this Readme use the proc syntax (`critical: -\u003e { 5.0 }`) but\nfor simple constants you may prefer to use the plain syntax (`critical: 5.0`).\n\n## Workflows\n\u003c!-- ONLY IN template/Readme.md\n\n### Setup\n - clone the repo\n - `gem install bundler \u0026\u0026 bundle install`\n - `cp .env.example .env`\n - open [Datadog API Settings](https://app.datadoghq.com/account/settings#api)\n - create a `API Key` or get an existing one from an admin, then add it to `.env` as `DATADOG_API_KEY`\n - open [Datadog API Settings](https://app.datadoghq.com/personal-settings/application-keys) and create a new key, then add it to `.env` as `DATADOG_APP_KEY=`\n - if you have a custom subdomain, change the `DATADOG_SUBDOMAIN=app` in `.env`\n - verify it works by running `rake plan`, it might show some diff, but should not crash\n--\u003e\n\n### Adding a team\n - `mention` is used for all team monitors via `super()`\n - `renotify_interval` is used for all team monitors (defaults to `0` / off)\n - `tags` is used for all team monitors/dashboards (defaults to `team:\u003cteam-name\u003e`)\n\n```Ruby\n# teams/my_team.rb\nmodule Teams\n  class MyTeam \u003c Kennel::Models::Team\n    defaults(\n      mention: -\u003e { \"@slack-my-team\" }\n    )\n  end\nend\n```\n\n### Adding a new monitor\n - use [datadog monitor UI](https://app.datadoghq.com/monitors#create) to create a monitor\n - see below\n\n### Updating an existing monitor\n - use [datadog monitor UI](https://app.datadoghq.com/monitors/manage) to find a monitor\n - run `URL='https://app.datadoghq.com/monitors/123' bundle exec rake kennel:import` and copy the output\n - find or create a project in `projects/`\n - add the monitor to `parts: [` list, for example:\n  ```Ruby\n  # projects/my_project.rb\n  class MyProject \u003c Kennel::Models::Project\n    defaults(\n      team: -\u003e { Teams::MyTeam.new }, # use existing team or create new one in teams/\n      parts: -\u003e {\n        [\n          Kennel::Models::Monitor.new(\n            self,\n            id: -\u003e { 123456 }, # id from datadog url, not necessary when creating a new monitor\n            type: -\u003e { \"query alert\" },\n            kennel_id: -\u003e { \"load-too-high\" }, # make up a unique name\n            name: -\u003e { \"Foobar Load too high\" }, # nice descriptive name that will show up in alerts and emails\n            message: -\u003e {\n              # Explain what behavior to expect and how to fix the cause\n              # Use #{super()} to add team notifications.\n              \u003c\u003c~TEXT\n                Foobar will be slow and that could cause Barfoo to go down.\n                Add capacity or debug why it is suddenly slow.\n                #{super()}\n              TEXT\n            },\n            query: -\u003e { \"avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} \u003e #{critical}\" }, # replace actual value with #{critical} to keep them in sync\n            critical: -\u003e { 20 }\n          )\n        ]\n      }\n    )\n  end\n  ```\n - run `PROJECT=my_project bundle exec rake plan`, an Update to the existing monitor should be shown (not Create / Delete)\n - alternatively: `bundle exec rake generate` to only locally update the generated `json` files\n - review changes then `git commit`\n - make a PR ... get reviewed ... merge\n - datadog is updated by CI\n\n### Deleting\n\nRemove the code that created the resource. The next update will delete it (see above for PR workflow).\n\n### Adding a new dashboard\n - go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard\n - see below\n\n### Updating an existing dashboard\n - go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to find a dashboard\n - run `URL='https://app.datadoghq.com/dashboard/bet-foo-bar' bundle exec rake kennel:import` and copy the output\n - find or create a project in `projects/`\n - add a dashboard to `parts: [` list, for example:\n  ```Ruby\n  class MyProject \u003c Kennel::Models::Project\n    defaults(\n      team: -\u003e { Teams::MyTeam.new }, # use existing team or create new one in teams/\n      parts: -\u003e {\n        [\n          Kennel::Models::Dashboard.new(\n            self,\n            id: -\u003e { \"abc-def-ghi\" }, # id from datadog url, not needed when creating a new dashboard\n            title: -\u003e { \"My Dashboard\" },\n            description: -\u003e { \"Overview of foobar\" },\n            template_variables: -\u003e { [\"environment\"] }, # see https://docs.datadoghq.com/api/?lang=ruby#timeboards\n            kennel_id: -\u003e { \"overview-dashboard\" }, # make up a unique name\n            layout_type: -\u003e { \"ordered\" },\n            definitions: -\u003e {\n              [ # An array or arrays, each one is a graph in the dashboard, alternatively a hash for finer control\n                [\n                  # title, viz, type, query, edit an existing graph and see the json definition\n                  \"Graph name\", \"timeseries\", \"area\", \"sum:mystats.foobar{$environment}\"\n                ],\n                [\n                  # queries can be an Array as well, this will generate multiple requests\n                  # for a single graph\n                  \"Graph name\", \"timeseries\", \"area\", [\"sum:mystats.foobar{$environment}\", \"sum:mystats.success{$environment}\"],\n                  # add events too ...\n                  events: [{q: \"tags:foobar,deploy\", tags_execution: \"and\"}]\n                ]\n              ]\n            }\n          )\n        ]\n      }\n    )\n  end\n ```\n\n### Updating existing resources with id\nSetting `id` makes kennel take over a manually created datadog resource.\nWhen manually creating to import, it is best to remove the `id` and delete the manually created resource.\n\nWhen an `id` is set and the original resource is deleted, kennel will fail to update,\nremoving the `id` will cause kennel to create a new resource in datadog.\n\n### Organizing projects with many resources\nWhen project files get too long, this structure can keep things bite-sized.\n\n```Ruby\n# projects/project_a/base.rb\nmodule ProjectA\n  class Base \u003c Kennel::Models::Project\n    defaults(\n      kennel_id: -\u003e { \"project_a\" },\n      parts: -\u003e {\n        [\n          Monitors::FooAlert.new(self),\n          ...\n        ]\n      }\n      ...\n\n# projects/project_a/monitors/foo_alert.rb\nmodule ProjectA\n  module Monitors\n    class FooAlert \u003c Kennel::Models::Monitor\n      ...\n```\n\n### Updating a single project or resource\n\n- Use `PROJECT=\u003ckennel_id\u003e` for single project:\n\n  Use the projects `kennel_id` (and if none is set then snake_case of the class name including modules)\n  to refer to the project. For example for `class ProjectA` use `PROJECT=project_a` but for `Foo::ProjectA` use `foo_project_a`.\n\n- Use `TRACKING_ID=\u003cproject-kennel_id\u003e:\u003cresource-kennel_id\u003e` for single resource:\n\n  Use the project kennel_id and the resources kennel_id, for example `class ProjectA` and `FooAlert` would give `project_a:foo_alert`.\n\n### Skipping validations\nSome validations might be too strict for your usecase or just wrong, please [open an issue](https://github.com/grosser/kennel/issues) and\nto unblock use the `validate: -\u003e { false }` option.\n\n### Linking resources with kennel_id\nLink resources with their kennel_id in the format `project kennel_id` + `:` + `resource kennel_id`,\nthis should be used to create dependent resources like monitor + slos,\nso they can be created in a single update and can be re-created if any of them is deleted.\n\n|Resource|Type|Syntax|\n|---|---|---|\n|Dashboard|uptime|`monitor: {id: \"foo:bar\"}`|\n|Dashboard|alert_graph|`alert_id: \"foo:bar\"`|\n|Dashboard|slo|`slo_id: \"foo:bar\"`|\n|Dashboard|timeseries|`queries: [{ data_source: \"slo\", slo_id: \"foo:bar\" }]`|\n|Monitor|composite|`query: -\u003e { \"%{foo:bar} \u0026\u0026 %{foo:baz}\" }`|\n|Monitor|slo alert|`query: -\u003e { \"error_budget(\\\"%{foo:bar}\\\").over(\\\"7d\\\") \u003e 123.0\" }`|\n|Slo|monitor|`monitor_ids: -\u003e [\"foo:bar\"]`|\n\n### Debugging changes locally\n - rebase on updated `master` to not undo other changes\n - figure out project name by converting the class name to snake_case\n - run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project (monitors: remove mentions while debugging to avoid alert spam)\n   - use `PROJECT=foo,bar,...` for multiple projects\n\n### Reuse\nAdd to `parts/\u003cfolder\u003e`.\n\n```Ruby\nmodule Monitors\n  class LoadTooHigh \u003c Kennel::Models::Monitor\n    defaults(\n      name: -\u003e { \"#{project.name} load too high\" },\n      message: -\u003e { \"Shut it down!\" },\n      type: -\u003e { \"query alert\" },\n      query: -\u003e { \"avg(last_5m):avg:system.load.5{hostgroup:#{project.kennel_id}} by {pod} \u003e #{critical}\" }\n    )\n  end\nend\n```\n\nReuse it in multiple projects.\n\n```Ruby\nclass Database \u003c Kennel::Models::Project\n  defaults(\n    team: -\u003e { Kennel::Models::Team.new(mention: -\u003e { '@slack-foo' }, kennel_id: -\u003e { 'foo' }) },\n    parts: -\u003e { [Monitors::LoadTooHigh.new(self, critical: -\u003e { 13 })] }\n  )\nend\n```\n\n## Helpers\n\n### Listing un-muted alerts\nRun `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.\n\n### Validating mentions work\n`rake kennel:validate_mentions` should run as part of CI\n\n### Grepping through all of datadog\n```Bash\nrake kennel:dump \u003e tmp/dump\ncat tmp/dump | grep foo\n```\nfocus on a single type: `TYPE=monitors`\n\nShow full resources or just their urls by pattern:\n```Bash\nrake kennel:dump_grep DUMP=tmp/dump PATTERN=foo URLS=true\nhttps://foo.datadog.com/dasboard/123\nhttps://foo.datadog.com/monitor/123\n```\n\n### Find all monitors with No-Data\n`rake kennel:nodata TAG=team:foo`\n\n### Finding the tracking id of a resource\n\nWhen trying to link resources together, this avoids having to go through datadog UI.\n\n```Bash\nrake kennel:tracking_id ID=123 RESOURCE=monitor\n```\n\n\u003c!-- NOT IN template/Readme.md --\u003e\n\n## Development\n\n### Benchmarking\n- Setting `FORCE_GET_CACHE=true` will cache all get requests, which makes benchmarking improvements more reliable.\n- Setting `STORE=false` will make `rake plan` not update the files on disk and save a bit of time\n\n### Integration testing\n```Bash\nrake play\ncd template\nrake plan\n```\n\nThen make changes to play around, do not commit changes and make sure to revert with a `rake kennel:update_datadog` after deleting everything.\n\nTo make changes via the UI, make a new free datadog account and use it's credentials instead.\n\nAuthor\n======\n[Michael Grosser](http://grosser.it)\u003cbr/\u003e\nmichael@grosser.it\u003cbr/\u003e\nLicense: MIT\u003cbr/\u003e\n![CI](https://github.com/grosser/kennel/workflows/CI/badge.svg)\n\u003c!-- NOT IN --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrosser%2Fkennel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrosser%2Fkennel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrosser%2Fkennel/lists"}