{"id":18983087,"url":"https://github.com/bitkill/prometheus-msk-discovery","last_synced_at":"2025-11-11T09:02:03.886Z","repository":{"id":61830348,"uuid":"550522727","full_name":"bitkill/prometheus-msk-discovery","owner":"bitkill","description":"🧿 discovers msk nodes and adds them to prometheus ","archived":false,"fork":false,"pushed_at":"2024-11-12T10:56:25.000Z","size":131,"stargazers_count":1,"open_issues_count":7,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-01T12:12:02.854Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bitkill.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-12T23:02:57.000Z","updated_at":"2024-11-12T10:56:24.000Z","dependencies_parsed_at":"2023-10-11T10:51:21.853Z","dependency_job_id":"3389f6af-78c8-4d8d-89d9-6c88e167fa46","html_url":"https://github.com/bitkill/prometheus-msk-discovery","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitkill%2Fprometheus-msk-discovery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitkill%2Fprometheus-msk-discovery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitkill%2Fprometheus-msk-discovery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitkill%2Fprometheus-msk-discovery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bitkill","download_url":"https://codeload.github.com/bitkill/prometheus-msk-discovery/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239987520,"owners_count":19729821,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T16:15:52.820Z","updated_at":"2025-11-11T09:02:03.834Z","avatar_url":"https://github.com/bitkill.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Prometheus AWS MSK discovery\n\nHeavily based on [teralytics/prometheus-ecs-discovery](https://github.com/teralytics/prometheus-ecs-discovery),\nbut exports MSK instead of ECS.\n\n## Help\n\nRun `prometheus-msk-discovery --help` to get information.\n\nThe command line parameters that can be used are:\n\n* `-config.cluster` (string) : arn of the MSK cluster to scrape\n* `-config.jmx-metrics` : add the jmx metrics to the out file (default = true) (default true)\n* `-config.node-metrics` : add the jmx metrics to the out file (default = true) (default true)\n* `-config.role-arn` (string) : arn of the role to assume when scraping the AWS API (optional)\n* `-config.scrape-interval` (duration) : interval at which to scrape the AWS API for MSK service discovery information (default 1m0s)\n* `-config.scrape-times` (int) : how many times to scrape before exiting (0 = infinite)\n* `-config.write-to` (string) : path of file to write MSK service discovery information to (default \"msk_file_sd.yml\")\n\n## Usage\n\nFirst, build this program using the usual `go get` mechanism.\n\nThen, run it as follows:\n\n* Ensure the program can write to a directory readable by\n  your Prometheus master instance(s).\n* Export the usual `AWS_REGION`, `AWS_ACCESS_KEY_ID` and\n  `AWS_SECRET_ACCESS_KEY` into the environment of the program,\n  making sure that the keys have access to the MSK API.\n  If the program needs to assume\n  a different role to obtain access, this role's ARN may be\n  passed in via the `--config.role-arn` option. This option also\n  allows for cross-account access, depending on which account\n  the role is defined in.\n* Start the program, using the command line option\n  `-config.write-to` to point the program to the specific\n  folder that your Prometheus master can read from.\n* Add a `file_sd_config` to your Prometheus master:\n\n```\nscrape_configs:\n- job_name: msk\n  file_sd_configs:\n    - files:\n      - /path/to/msk_file_sd.yml\n      refresh_interval: 10m\n  # Drop unwanted labels using the labeldrop action\n  metric_relabel_configs:\n    - regex: cluster_arn\n      action: labeldrop\n```\n\nThat's it.  You should begin seeing the program scraping the\nAWS APIs and writing the discovery file (by default it does\nthat every minute, and by default Prometheus will reload the\nfile the minute it is written).  After reloading your Prometheus\nmaster configuration, this program will begin informing via\nthe discovery file of new targets that Prometheus must scrape.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitkill%2Fprometheus-msk-discovery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbitkill%2Fprometheus-msk-discovery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitkill%2Fprometheus-msk-discovery/lists"}