{"id":18284688,"url":"https://github.com/pplu/aws-managed-prometheus","last_synced_at":"2026-05-02T02:39:08.514Z","repository":{"id":48341162,"uuid":"390802893","full_name":"pplu/aws-managed-prometheus","owner":"pplu","description":"An article about AWS Managed Prometheus","archived":false,"fork":false,"pushed_at":"2021-07-31T06:55:56.000Z","size":10,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-15T00:26:15.836Z","etag":null,"topics":["aws","cloudformation","grafana","grafana-datasource","prometheus","prometheus-metrics","remote-write"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pplu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-29T17:30:08.000Z","updated_at":"2024-09-23T11:17:41.000Z","dependencies_parsed_at":"2022-09-09T13:12:11.837Z","dependency_job_id":null,"html_url":"https://github.com/pplu/aws-managed-prometheus","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplu%2Faws-managed-prometheus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplu%2Faws-managed-prometheus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplu%2Faws-managed-prometheus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplu%2Faws-managed-prometheus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pplu","download_url":"https://codeload.github.com/pplu/aws-managed-prometheus/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247987108,"owners_count":21028891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloudformation","grafana","grafana-datasource","prometheus","prometheus-metrics","remote-write"],"created_at":"2024-11-05T13:14:24.227Z","updated_at":"2026-05-02T02:39:08.445Z","avatar_url":"https://github.com/pplu.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Understanding Amazon Managed Prometheus service\n\nAmazon Managed Prometheus (AMP) service is in Public preview, meaning that we can use it today. Lets have a\nlook at what this service provides.\n\nIf you've ever used Prometheus, you will know that Prometheus is a modern monitoring system. When you\nrun Prometheus, it will poll your systems, recovering metrics from them. Normally, when we configure Prometheus, \nwe take care of the polling configuration (what metrics to gather). Underneath, Prometheus has a metrics \ndatabase where it stores all the data that it gathers. Other systems (like the alerting subsystem) will poll\nthe database to see if any of the configured thresholds are matched, producing an alert.\n\n## Simple Prometheus architectures\n\nYou may be surprised that Amazon Managed Prometheus service is not about the polling part of Prometheus. It is\nabout the storage part of Prometheus. The modular design of Prometheus lets the data gathering happen on one system, \nand the storage on another. By default, when we run Prometheus, it will store all data on the local file system of \nthe machine where it is running. In the Cloud world we like the concept of stateless / immutable systems. Storing all\nthe metrics data on the local filesystem impedes applying this pattern, making us have to take care of managing\ndisk space, I/O, etc on the instance running Prometheus.\n\nAMP really helps us manage our monitoring solution in a different way, since it enables us to write our monitoring metrics \nin an external, managed, Prometheus database.\n\n## I thought AMP was for Kubernetes only\n\nThe AWS documentation for AMP right now only talks about configuring AMP for Kubernetes. I suppose that was their\ntop priority, so getting your non-Kubernetes Prometheus setup to use AMP can be a bit confusing, but the hints\nare there. So lets start.\n\n## AMP Workspaces\n\nEach AMP Workspace that you create is an independent Prometheus database. Each workspace has a unique id of the form\n`ws-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX`. Workspaces can have an alias (a non-unique name) to identify them.\n\n## Configuring Prometheus to write to AMP\n\nPrometheus has a `remote_write` configuration element in the `prometheus.yml` file:\n\n```\nremote_write:\n  url: ...\n```\n\nYour first impulse may be to put the remote write URL of your workspace (this can be found in the Summary section\nof your AMP workspace). This URL is something like `https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/api/v1/remote_write`.\n\nThe problem here is that the remote write URL is a public endpoint authenticated with IAM via AWSv4 signatures, and Prometheus \ndoesn't know how to sign remote write requests.\n\nTo get Prometheus to write to the AMP service we will make it think it is talking to an unauthenticated remote write service, that\nwill just sign the requests, and send them to AMP. We will use the AWS sigv4-proxy:\n\nRun the sigv4-proxy on your Prometheus machine:\n\n```\ndocker run -d --network=host --name promsigner public.ecr.aws/aws-observability/aws-sigv4-proxy:1.0 --name aps --region us-east-1 --host aps-workspaces.us-east-1.amazonaws.com --port :8005\n```\n\nNote: The AMP service is refered to as `aps` in IAM and uses a `aps-workspaces` name in its endpoint. This can be a bit confusing at times, and it's not a typo.\n\nThe signing proxy need some type of IAM credentials to work. Configuring an IAM Role for the instance is the best solution in this case (maybe you already have an\ninstance role configure for Prometheus to call the EC2 API, for example).\n\n```\nPolicyName: prometheusdatastore\nPolicyDocument:\n Version: \"2012-10-17\"\n Statement:\n - Sid: AllowRemote\n   Effect: Allow\n   Action: [ \"aps:GetLabels\", \"aps:GetMetricdata\", \"aps:GetSeries\", \"aps:QueryMetrics\", \"aps:RemoteWrite\" ]\n   Resource: \"*\"\n```\n\nNow we will configure the `remote_write` section of the Prometheus configuration:\n\n```\nremote_write:\n- url: 'http://localhost:8005/workspaces/AMP_WORKSPACE_ID/api/v1/remote_write'\n```\nRemember to substitute the `AMP_WORKSPACE_ID` for the Id of your workspace.\n\nAfter that, Prometheus will start writing to AMP.\n\n## Getting data out of Prometheus\n\nA very typical setup is to have a Grafana server for building our dashboards (since the Prometheus\nweb interface is very limited).\n\nLets see how to get Grafana to read from AMP:\n\nIn this case, Grafana knows how to authenticate requests to the AMP endpoints, but you have to\nactivate this option (it won't appear in the Grafana datasource configuration interface):\n\nIn `grafana.ini` add:\n\n```\n[aws]\nallowed_auth_providers=default,ec2_iam_role\n[auth]\nsigv4_auth_enabled=true\n```\n\nThis will make a new option appear in the Prometheus datasource configuration for Grafana: \n`SigV4 auth`. Activate this authentication option, and configure the default region in the \n`SigV4 Auth Details` section.\n\nIn `URL`, introduce the endpoint of your AMP Workspace: `https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/`\n (note that the URL doesn't have the trailing `api/` part that the write and the query endpoints have).\n\nNote that optionally we could not configure the AWSv4 auth in Grafana, pointing the datasource URL to \n`http://localhost:8005/workspaces/AMP_WORKSPACE_ID/` (note the http at the start), so Grafana would also \nuse the sigv4 proxy, but I think it's better that Grafana handles everything natively.\n\n## Is it working?\n\nBy now your Prometheus should be pushing its metrics to AMP, and Grafana should be reading those metrics. \n\n## Going the extra mile\n\nBy now you should be capable, with a bit more work to adapt your monitoring instance to immutability. On instance\nstartup, you can pull config files and start Prometheus and Grafana. You will need to investigate:\n\n - Grafana datasource provisioning\n - Grafana dashboard provisioning\n\nThat is left as an excercise to the reader. After doing this excercise, you will find that you can simply\nthrow away instances with old configurations, starting new instances with new configuration. Your monitoring\ninfrastructure will also be more robust thanks to the fact that Autoscaling can substitute your failed \nmonitoring instance.\n\n## No AMP Resources in CloudFormation\n\nAs of this writing, there was no CloudFormation support for creating AMP Workspaces via CloudFormation. I've built a\n[CloudFormation Custom Resource](amp.yaml) to fill that gap.\n\n# Conclusions\n\nAlthough it is not explicitly documented, we can get AMP to work outside of a Kubernetes environment and get\nthe benefits of not having to manage the metrics storage of Prometheus.\n\nThere are, though, a couple of things to remark:\n\n - sigv4-proxy is only distributed as a Docker container. Being a stand-alone Go program, It would be useful\n   that AWS also distribute it in non-docker format also.\n - I hope AWS gets v4 signatures support into Prometheus to make things simpler (no need for v4sig-proxy).\n - No native CloudFormation support for AMP: services should have CloudFormation support at Launch. The \n   community can \"fill in gaps\", but it is increasingly frustrating to work on things that AWS should do.\n - You can look at the Managed Grafana Enterprise AWS Offering, but it will probably be more expensive\n   than hosting your own server with Grafana Community and a small instance.\n\n# Author, Copyright and License\n\nThis article was authored by Jose Luis Martinez Torres.\n\nThis article is (c) 2021 Jose Luis Martinez Torres, Licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).\n\nThe canonical, up-to-date source is [GitHub](https://github.com/pplu/aws-managed-prometheus). Feel free to contribute back.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpplu%2Faws-managed-prometheus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpplu%2Faws-managed-prometheus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpplu%2Faws-managed-prometheus/lists"}