{"id":15039014,"url":"https://github.com/apptality/aws-ecs-cloudmap-prometheus-sd","last_synced_at":"2025-08-01T08:10:59.791Z","repository":{"id":249456871,"uuid":"831258032","full_name":"apptality/aws-ecs-cloudmap-prometheus-sd","owner":"apptality","description":"A comprehensive solution for dynamic Prometheus service discovery in AWS ECS using OpenTelemetry. This tool enables HTTP-based discovery of ECS tasks and CloudMap services, allowing for flexible filtering, label management, and scalable monitoring. Ideal for teams leveraging AWS ECS.","archived":false,"fork":false,"pushed_at":"2024-09-13T20:14:59.000Z","size":801,"stargazers_count":8,"open_issues_count":1,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-04-10T00:01:29.101Z","etag":null,"topics":["aws","cloudmap","docker","dotnetcore","ecs","fargate","monitoring","prometheus"],"latest_commit_sha":null,"homepage":"https://medium.com/@alxtsbk/scraping-prometheus-metrics-from-aws-ecs-9c8d9a1ca1bd","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apptality.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-20T04:14:41.000Z","updated_at":"2025-01-29T22:02:29.000Z","dependencies_parsed_at":"2024-07-22T08:32:21.836Z","dependency_job_id":"d8273233-19b0-4bdd-99d8-e345d8732c06","html_url":"https://github.com/apptality/aws-ecs-cloudmap-prometheus-sd","commit_stats":null,"previous_names":["apptality/aws-ecs-cloudmap-prometheus-sd"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apptality%2Faws-ecs-cloudmap-prometheus-sd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apptality%2Faws-ecs-cloudmap-prometheus-sd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apptality%2Faws-ecs-cloudmap-prometheus-sd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apptality%2Faws-ecs-cloudmap-prometheus-sd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apptality","download_url":"https://codeload.github.com/apptality/aws-ecs-cloudmap-prometheus-sd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248131324,"owners_count":21052819,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloudmap","docker","dotnetcore","ecs","fargate","monitoring","prometheus"],"created_at":"2024-09-24T20:41:13.784Z","updated_at":"2025-04-10T00:02:37.478Z","avatar_url":"https://github.com/apptality.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS ECS \u0026 CloudMap Prometheus Discovery\n\n[![Build](https://github.com/apptality/aws-ecs-cloudmap-prometheus-sd/actions/workflows/docker.yml/badge.svg?branch=develop)](https://github.com/apptality/aws-ecs-cloudmap-prometheus-sd/actions/workflows/docker.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue)](/LICENSE)\n[![Docker Image Version](https://img.shields.io/docker/v/apptality/aws-ecs-cloudmap-prometheus-discovery?logo=docker\u0026label=Latest%20Version)](https://hub.docker.com/r/apptality/aws-ecs-cloudmap-prometheus-discovery)\n[![Docker Image Size (tag)](https://img.shields.io/docker/image-size/apptality/aws-ecs-cloudmap-prometheus-discovery/latest?logo=docker\u0026label=Image%20Size)](https://hub.docker.com/r/apptality/aws-ecs-cloudmap-prometheus-discovery/tags)\n[![Medium](https://img.shields.io/badge/Medium-Read_blog_post-black?logo=medium)](https://medium.com/@alxtsbk/scraping-prometheus-metrics-from-aws-ecs-9c8d9a1ca1bd)\n\n## Overview\n\nThis application facilitates discovery of ECS and/or CloudMap resources and compatible with [Prometheus HTTP Service Discovery](https://prometheus.io/docs/prometheus/latest/http_sd/#writing-http-service-discovery).\n\nApplication leverages AWS API to dynamically discover:\n\n* [ECS Clusters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/clusters.html), related ECS Services, and ECS Tasks\n* [CloudMap Namespaces](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html), registered Service Connect and Service Discovery services, and their instances\n\nmaking it easier for Prometheus to monitor these services without hardcoding their addresses and ports, or using any kind of [file-based discovery](https://prometheus.io/docs/guides/file-sd/) (thus can be run as a standalone application: as it's own AWS ECS Service, for example).\n\n![AWS ECS \u0026 CloudMap Prometheus Discovery](/docs/diagram.svg)\n\n## Features\n\nBelow is a high level overview of what application is capable of. For the full list of supported configuration parameters and how to override these using [Docker Environment Variables](https://docs.docker.com/reference/cli/docker/container/run/#env) please refer to [appsettings.json](src/Apptality.CloudMapEcsPrometheusDiscovery/appsettings.json)\n\n* **Prometheus Compatibility**: By default, exposes `9001:/prometheus-targets` endpoint which response is compatible with [\u003chttp_sd_config\u003e](https://prometheus.io/docs/prometheus/latest/http_sd/#writing-http-service-discovery) of Prometheus configuration file, as well as with [OpenTelemetry Prometheus Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md) (see example below).\n\n* **ECS Discovery**: Supports discovery of ECS clusters, services, running tasks\n  * Supply `EcsClusters` as semicolon separated string of ECS clusters to include: `\"cluster1;cluster2;\"`\n  * Supports filtering of ECS Services that should be included in the Prometheus response based on Resource Tags (see `EcsServiceSelectorTags` configuration property)\n  * Supports filtering to control which tags should be included as labels in the response (see `EcsTaskTags`, `EcsServiceTags` configuration properties)\n* **AWS CloudMap Integration**: Supports discovery of CloudMap namespaces, services and instances\n  * Supply `CloudMapNamespaces` as semicolon separated string of CloudMap namespaces to include: `\"namespace1;\"`\n  * Supports filtering of CloudMap Services that should be included in the Prometheus response based on Resource Tags (see `CloudMapServiceSelectorTags` configuration property)\n  * Supports filtering to control which tags should be included as labels in the response (see `CloudMapServiceTags`, `CloudMapNamespaceTags` configuration properties)\n* **Scalable and Flexible**: Built to be scalable and flexible, adapting to changes in the service infrastructure\n  * Allows having any number of port-metrics pairs per each running ECS Task (useful when your ECS Task Definition defines multiple containers, which also expose ports and metrics endpoint). See `MetricsPathPortTagPrefix` configuration property for more details.\n  * Allows supplying static set of labels to be added to every Prometheus target (see `ExtraPrometheusLabels` configuration property)\n  * Supports rich mechanism for re-labling, so you don't have to fix your Grafana dashboards. See `RelabelConfigurations` configuration property for more details.\n  * Exposes health check endpoint: `/health`\n\nAt least one of `EcsClusters` or `CloudMapNamespaces` must be provided for application to work. If you would like to include both [Service Connect](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect.html) and [Service Discovery](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html) targets, `CloudMapNamespaces` must be specified (with or without `EcsClusters`). When **both parameters** are specified, the end result is only an intersection of targets that exist in ECS clusters and CloudMap namespaces provided.\n\n\u003e **Permissions**: IAM permissions are required to discover ECS clusters (`ecs:Get*`, `ecs:List*`, `ecs:Describe*`) and CloudMap namespaces (`servicediscovery:Get*`, `servicediscovery:List*`, `servicediscovery:Discover*`, `route53:Get*`)\n\n\n## Usage Example\n\nFor the full example of running this in AWS, please navigate to [/example](/example) folder. Integrates with [OpenTelemetry receivers config](https://opentelemetry.io/docs/collector/configuration/#receivers).\n\n\nRefer to [appsettings.json](src/Apptality.CloudMapEcsPrometheusDiscovery/appsettings.json) for all supported configuration options.\n\nExample run command:\n\n```bash\n  docker run --rm \\\n    # ** REQUIRED PARAMETERS **\n    # At least one of \"EcsClusters\" or \"CloudMapNamespaces\" must be provided.\n    -e DiscoveryOptions__EcsClusters=\"\u003ccluster1\u003e;\u003ccluster2\u003e\" \\\n    -e DiscoveryOptions__CloudMapNamespaces=\"\u003cnamespace1\u003e;\" \\\n    # If running outside of AWS - pass credentials explicitly\n    # If running in AWS - use Task IAM Role.\n    # IMPORTANT: Credentials need to have permissions to describe ECS cluster(optionally CloudMap namespaces)\n    -e AWS_REGION=\"us-west-1\" \\\n    -e AWS_ACCESS_KEY_ID=\"\u003ckey\u003e\" \\\n    -e AWS_SECRET_ACCESS_KEY=\"\u003csecret\u003e\" \\\n    -e AWS_SESSION_TOKEN=\"\u003csession_token\u003e\" \\\n    # ** OPTIONAL PARAMETERS **\n    # Will only include those services, which have 'prom_scrape_target' tag set to 'yes'.\n    # Leave blank to include all services in cluster(s)\n    -e DiscoveryOptions__EcsServiceSelectorTags=\"prom_scrape_target=yes;\" \\\n    # Same as above, but for filtering out CloudMap services\n    # Leave blank ot include all services in namespace(s)\n    -e DiscoveryOptions__CloudMapServiceSelectorTags=\"prom_scrape_target=yes;\" \\\n    # Semicolon separated string of tag keys to include in the service discovery response as metadata.\n    # Supports glob pattern matching using * and ?.\n    -e DiscoveryOptions__EcsTaskTags=\"*\" \\\n    -e DiscoveryOptions__EcsServiceTags=\"*\" \\\n    -e DiscoveryOptions__CloudMapServiceTags=\"AmazonECS*;\" \\\n    -e DiscoveryOptions__CloudMapNamespaceTags=\"*\" \\\n    # Semicolon separated string of labels to include in the service discovery response as metadata.\n    # Will be added to all discovered targets.\n    -e DiscoveryOptions__ExtraPrometheusLabels=\"custom_static_tag=my-static-tag;\" \\\n    # Tag prefix to identify metrics port, [path | \"/metrics\"], [name | \"\"] triplets.\n    # Please refer to the configuration options to learn more.\n    -e DiscoveryOptions__MetricsPathPortTagPrefix=\"METRICS_\" \\\n    # Add new or modify existing labels in the response using token replacements,\n    # to prevent the need of modifying your Grafana dashboards.\n    # Please refer to the configuration options to learn more.\n    -e DiscoveryOptions__RelabelConfigurations=\"cluster_and_service={{_sys_ecs_cluster}}-{{_sys_ecs_service}}\" \\\n    # Instructs .NET application to listen on 9001 inside the container\n    -e ASPNETCORE_URLS=\"http://*:9O01\" \\\n    # ** DOCKER **\n    -p 9001:9001 \\\n    apptality/aws-ecs-cloudmap-prometheus-discovery:latest\n```\n\nExample output:\n\n```json\n[\n  {\n    \"targets\": [\n      \"10.200.10.200:8080\"\n    ],\n    \"labels\": {\n      \"__metrics_path__\": \"/metrics\",\n      \"instance\": \"10.200.10.200\",\n      \"scrape_target_name\": \"app\",\n      \"__meta_cloudmap_service_instance_id\": \"c88nc14799fa46d794c1899612061h3s\",\n      \"__meta_cloudmap_service_name\": \"service-app\",\n      \"__meta_cloudmap_service_type\": \"ServiceConnect\",\n      \"__meta_ecs_cluster\": \"my-ecs-cluster\",\n      \"__meta_ecs_service\": \"my-fargate-application\",\n      \"__meta_ecs_task\": \"arn:aws:ecs:us-west-2:123456789012:task/my-ecs-cluster/c88nc14799fa46d794c1899612061h3s\",\n      \"__meta_ecs_task_definition\": \"arn:aws:ecs:us-west-2:123456789012:task-definition/my-fargate-application:2\",\n      \"_sys_cloudmap_service_instance_id\": \"c88nc14799fa46d794c1899612061h3s\",\n      \"_sys_cloudmap_service_name\": \"my-fargate-application\",\n      \"_sys_cloudmap_service_type\": \"ServiceConnect\",\n      \"_sys_ecs_cluster\": \"my-ecs-cluster\",\n      \"_sys_ecs_service\": \"my-fargate-application\",\n      \"_sys_ecs_task\": \"arn:aws:ecs:us-west-2:123456789012:task/my-ecs-cluster/c88nc14799fa46d794c1899612061h3s\",\n      \"_sys_ecs_task_definition\": \"arn:aws:ecs:us-west-2:123456789012:task-definition/my-fargate-application:2\",\n      \"prom_scrape_target\": \"yes\",\n      \"AmazonECSManaged\": \"true\",\n      \"custom_static_tag\": \"my-static-tag\",\n      \"cluster_and_service\": \"my-ecs-cluster-my-fargate-application\"\n    }\n  },\n  ...\n]\n```\n\n## Response Structure\n\nResponse is returned in [HTTP_SD format](https://prometheus.io/docs/prometheus/latest/http_sd/#http_sd-format) compatible format:\n\n```json\n[\n  {\n    \"targets\": [ \"\u003chost\u003e\", ... ],\n    \"labels\": {\n      \"\u003clabelname\u003e\": \"\u003clabelvalue\u003e\", ...\n    }\n  },\n  ...\n]\n```\n\n### Success\n\nSuccess response is returned with `application/json` HTTP Content Type, and `200` HTTP status code.\n\nSome clarification on labels returned:\n\n* **scrape_target_name** - when any of resource contains `METRICS_NAME_` AWS Resource Tag - this label will have tag value. ECS Tasks can contain multiple containers you would like to scrape from, this label being present helps to denote between such containers when running PromQL queries.\n* **__meta** - labels starting with this prefix are meta labels, and are not included into the resulting set stored Prometheus, but can be used for [re-labeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config).\n* **_sys** - labels starting with this prefix are generated out of AWS Resources properties (ECS, CloudMap). They are prefixed as such in order to prevent conflict with your existing infrastructure configuration, and combined with `RelabelConfigurations` enable powerful manipulations of the resulting labels set.\n\nAnything else returned is either inferred from AWS Resource Tags specified via selectors (`EcsTaskTags`, `EcsServiceTags`, `CloudMapServiceTags`, `CloudMapNamespaceTags`), supplied via `ExtraPrometheusLabels`, or product of `RelabelConfigurations` configurations.\n\nPlease note, that tags are resolved from AWS resource in the following order:\n\n`ECS Task` \u003e `ECS Service` \u003e `CloudMap Service` \u003e `CloudMap Namespace`\n\nmeaning that if ECS Service has tag `\"MyCustomTag=EcsService\"` and CloudMap Service has the same tag, but with different value: `\"MyCustomTag=CloudMap\"`  - the resulting scrape target label will have value of the former:\n\n```json\n[\n  {\n    \"targets\": [ \"\u003chost\u003e\", ... ],\n    \"labels\": {\n      \"MyCustomTag\": \"EcsService\", ...\n    }\n  },\n  ...\n]\n\n```\n\n### Error\n\nWhen application runs into an error, response is returned with `500` HTTP status code:\n\n```json\n{\n  \"type\": \"https://tools.ietf.org/html/rfc9110#section-15.6.1\",\n  \"title\": \"An error occurred while processing your request.\",\n  \"status\": 500\n}\n```\n\nYou'll need to investigate server logs for exception details:\n\n```log\n2024-08-20 21:22:23.242 -03:00 [ERR] HTTP GET /prometheus-targets responded 500 in 99.9711 ms\n2024-08-20 21:22:23.243 -03:00 [ERR] An unhandled exception has occurred while executing the request.\nMicrosoft.Extensions.Options.OptionsValidationException: At least one of 'EcsClusters' or 'CloudMapNamespaces' name must be specified.\n   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)\n...\n```\n\n## Storing this image in ECR\n\n`~/.scripts/dockerhub-to-ecr.sh` script is intended for users who need control over their Docker images in AWS ECR, allowing to pull an image from DockerHub, re-tag it, and push it to an ECR repository with customizable options for platform and tagging.\n\nRun the script specifying your ECR URL as the first positional parameter:\n\n```bash\n# \u003cpaste in your AWS credential\u003e\n\ncd ./scripts\n\nchmod +x dockerhub-to-ecr.sh\n\n./dockerhub-to-ecr.sh \u003ctarget_ecr_repository_url\u003e \\\n    # Defaults to 'apptality/aws-ecs-cloudmap-prometheus-discovery:latest'\n    [source_dockerhub_image_url] \\\n    # Specify explicit value of image tag you want to label your ECR image with.\n    # By default, same tag as on source image will be applied.\n    [target_ecr_repository_tag (Default: same as source)] \\\n    # Specify docker platform to re-push. Amd64/Arm64 available.\n    [docker_image_platform: 'linux/amd64' (Default) | 'linux/arm64']\n```\n\nExample:\n\n```bash\n./dockerhub-to-ecr.sh 123456789012.dkr.ecr.us-west-2.amazonaws.com/tools/ecs-sd\n```\n\nThe above script will:\n\n1. Authenticate with AWS ECR region `us-west-2`\n1. Pull `linux/amd64` platform of `apptality/aws-ecs-cloudmap-prometheus-discovery:latest` locally\n1. Re-tag it to `123456789012.dkr.ecr.us-west-2.amazonaws.com/tools/ecs-sd:latest`\n1. Push `123456789012.dkr.ecr.us-west-2.amazonaws.com/tools/ecs-sd:latest` to ECR\n\n### Positional Parameters\n\n* **`target_ecr_repository_url`**: The full URL of your target AWS ECR repository without tag.\n* **`source_dockerhub_image_url` (optional)**: DockerHub image to pull (default: `apptality/aws-ecs-cloudmap-prometheus-discovery:latest`).\n* **`target_ecr_repository_tag` (optional)**: ECR image tag (defaults to the source image tag).\n* **`docker_image_platform` (optional)**: Image platform (`linux/amd64` by default).\n\n\u003e Note: you need to create destination ECR repository first.\n\n## References\n\nThis application is heavily influenced by the following articles:\n\n* [Metrics collection from Amazon ECS using Amazon Managed Service for Prometheus](https://aws.amazon.com/blogs/opensource/metrics-collection-from-amazon-ecs-using-amazon-managed-service-for-prometheus/)\n* [Metrics and traces collection from Amazon ECS using AWS Distro for OpenTelemetry with dynamic service discovery](https://aws.amazon.com/blogs/containers/metrics-and-traces-collection-from-amazon-ecs-using-aws-distro-for-opentelemetry-with-dynamic-service-discovery/)\n\nDefinitely suggest reading up on what is [AWS Distro for OpenTelemetry (ADOT)](https://aws-otel.github.io/docs/introduction), as it provides open source APIs, libraries, and agents to collect logs, metrics, and traces from your applications.\n\nFor how to integrate AWS AMP (Prometheus) with your Grafana please refer to [Set up Grafana open source or Grafana Enterprise for use with Amazon Managed Service for Prometheus](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-query-standalone-grafana.html) article by AWS.\n\n## Alternatives\n\nThere are definitely some discovery tools that are alternative to current application:\n\n* [Container Insights Prometheus metrics monitoring](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-autodiscovery-ecs.html) discovers your infrastructure using CloudWatch\n  * IMHO - How CloudWatch agent configuration is structured around discovery of resources is NOT idiomatic to AWS. AWS has a powerful, rich support of tags on pretty much any resource. It would be very natural \u0026 AWS'ish to use tagging filters for resource selection instead of complex Regex filters.\n* [aws-cloudmap-prometheus-sd](https://github.com/awslabs/aws-cloudmap-prometheus-sd) plugin by AWS\n* [prometheus-for-ecs](https://github.com/aws-samples/prometheus-for-ecs) plugin by AWS. Definitely dig into [deploy-prometheus](https://github.com/aws-samples/prometheus-for-ecs/tree/main/deploy-prometheus) and [deploy-adot](https://github.com/aws-samples/prometheus-for-ecs/tree/main/deploy-adot) folders of the repository to get a good understanding of how integrations are being set up.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit pull requests or open issues to discuss potential improvements or features.\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapptality%2Faws-ecs-cloudmap-prometheus-sd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapptality%2Faws-ecs-cloudmap-prometheus-sd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapptality%2Faws-ecs-cloudmap-prometheus-sd/lists"}