Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/benfoster/peekaqueue
Drop-in docker container for monitoring SQS queues
https://github.com/benfoster/peekaqueue
sqs
Last synced: about 1 month ago
JSON representation
Drop-in docker container for monitoring SQS queues
- Host: GitHub
- URL: https://github.com/benfoster/peekaqueue
- Owner: benfoster
- License: mit
- Created: 2019-04-29T19:06:02.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2020-08-31T11:45:56.000Z (about 4 years ago)
- Last Synced: 2023-03-23T22:01:30.423Z (over 1 year ago)
- Topics: sqs
- Language: C#
- Size: 20.5 KB
- Stars: 2
- Watchers: 2
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# peekaqueue
Peekaqueue provides a drop-in docker container for monitoring SQS queues and enabling queue-based auto-scaling of ECS services.
Unlike the default SQS metrics which are only published to CloudWatch every 5 minutes, peekaqueue can be configured to gather queue metrics much more frequently.
## Features
- Gets SQS queue metrics and exposes them via a [Prometheus](https://prometheus.io/) metrics endpoint
- Optionally retrieves the desired/pending/running counts of queue consumers running in ECS
- Optionally publishes the "backlog-per-instance" for your queue consumers to CloudWatch in order to configure auto-scaling
### Prometheus metrics
Peekaqueue can be used as a [Prometheus](https://prometheus.io/) exporter. By default, the following SQS queue attributes are exposed as metrics at http://localhost:5000/metrics:
| SQS Attribute | Metric Name | Labels |
|-----------------------------------------|----------------------------------|---------|
| `ApproximateNumberOfMessages` | `sqs_available_messages_count` | `queue` |
| `ApproximateNumberOfMessagesDelayed` | `sqs_delayed_messages_count` | `queue` |
| `ApproximateNumberOfMessagesNotVisible` | `sqs_not_visible_messages_count` | `queue` |
### Monitoring Queue Consumers
One of the main goals of peekaqueue is to also monitor the consumers of SQS queues. It assumes that consumers are dockerized applications running in ECS. When configured, the following Prometheus metrics are exposed:
| ECS Service Attribute | Metric Name | Labels |
|-----------------------|-----------------------------|------------------------------|
| `DesiredCount` | `ecs_service_desired_count` | `ecs_cluster`, `ecs_service` |
| `PendingCount` | `ecs_service_pending_count` | `ecs_cluster`, `ecs_service` |
| `RunningCount` | `ecs_service_running_count` | `ecs_cluster`, `ecs_service` |
### Calculating Consumer Backlogs
To auto-scale queue consumers based on the number of messages, we need to determine the backlog-per-instance, that is how many messages are available per consumer instance (`sqs_available_messages_count/ecs_service_running_count`).
This value is exposed as the following Prometheus metric:
| Metric Name | Labels |
|-----------------------------|---------------------------------------|
| `ecs_service_backlog_count` | `ecs_cluster`, `ecs_service`, `queue` |
In order to use this metric for auto-scaling it is published to CloudWatch as `EcsServiceBacklog` under the `Custom` metrics namespace (configurable). It includes the following CloudWatch metric dimensions:
- Queue
- EcsCluster
- EcsService
## Configuration
In most cases peekaqueue can be deployed direct to ECS using the images hosted on [Docker Hub](https://hub.docker.com/r/benfoster/peekaqueue) and configured using the following environment variables:
| Environment Variable | Data Type | Default Value | Description |
|--------------------------------------------------------------|-----------|-------------------------|-----------------------------------------------------------------------------------------------------------|
| `PEEKAQ_Monitoring__IntervalInSeconds` | `int` | `15` | The interval in seconds in which peekaqueue will collect stats about your queues and consumers |
| `PEEKAQ_Monitoring__MetricsEndpointPort` | `int` | `5000` | The port that the prometheus metrics will run on |
| `PEEKAQ_Monitoring__MetricsEndpointPath` | `string` | `metrics/` | The path of the metrics endpoint. Must end in a trailing `/` |
| `PEEKAQ_Monitoring__CloudWatchNamespace` | `string` | `Custom` | The CloudWatch namespace for publishing metrics to |
| `AWS_Region` | `string` | | The AWS region in which your resources reside e.g. `eu-west-2` |
| `PEEKAQ_Monitoring__Queues__X__Name` | `string` | | The name of the queue you wish to monitor. `X` is the 0-based array indexer of the queues. |
| `PEEKAQ_Monitoring__Queues__X__ConsumerCluster` | `string` | | The name of the ECS cluster in which your queue consumer resides |
| `PEEKAQ_Monitoring__Queues__X__ConsumerService` | `string` | | The name of the ECS service in which your queue consumer resides |
| `PEEKAQ_Serilog__WriteTo__0__Args__restrictedToMinimumLevel` | `string` | `Information` | Configures the console logging level. It it recommended to set this to `Error` when running in production |
| `PEEKAQ_Serilog__WriteTo__1__Args__restrictedToMinimumLevel` | `string` | `Information` | If using [Seq](https://datalust.co/seq logging), configures the Seq logging level |
| `PEEKAQ_Serilog__WriteTo__1__Args__serverUrl` | `string` | `http://localhost:5341` | The Seq server URL |
| `PEEKAQ_Serilog__WriteTo__1__Args__apiKey` | `string` | | API key for authenticating with Seq |
If you are building the image from scratch you can instead update `appsettings.json` as required.
## AWS Permissions and Credentials
It is assumed that you will run peekaqueue as an ECS service which should be given an ECS task role with the following permissions:
- sqs:GetQueueUrl
- sqs:GetQueueAttributes
- cloudwatch:PutMetricData
- ecs:DescribeServices
Below is an example policy that provides access to a single queue:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"sqs:GetQueueUrl",
"sqs:GetQueueAttributes"
],
"Resource": "arn:aws:sqs:eu-west-2:*:my-queue-name"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData",
"ecs:DescribeServices"
],
"Resource": "*"
}
]
}
```
## Running Locally
To test peekaqueue on your local machine against AWS, you must provide your AWS access keys:
```
docker run --rm -it \
-e AWS_REGION=eu-west-2 \
-e AWS_ACCESS_KEY_ID=YOURKEYID \
-e AWS_SECRET_ACCESS_KEY=YOURKEY \
-e "PEEKAQ_MONITORING__QUEUES__0__Name=QUEUENAME" \
-p 5000:5000 benfoster/peekaqueue:latest
```
## Developing peekaqueue
Peekqueue is a .NET Core 2.2 application. You can download the .NET Core runtime and tools [here](http://dot.net).