https://github.com/ev2900/kinesis_data_stream_hot_shard_demo

Demo a Kinesis Data Stream with a hot shard. Use CloudWatch to view the per shard message count
https://github.com/ev2900/kinesis_data_stream_hot_shard_demo

aws cloudwatch cloudwatch-dashboard cloudwatch-metrics kinesis kinesis-shards

Last synced: about 2 months ago
JSON representation

Demo a Kinesis Data Stream with a hot shard. Use CloudWatch to view the per shard message count

• Host: GitHub
• URL: https://github.com/ev2900/kinesis_data_stream_hot_shard_demo
• Owner: ev2900
• Created: 2022-05-29T13:47:02.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2025-02-09T01:16:04.000Z (4 months ago)
• Last Synced: 2025-04-10T03:14:30.538Z (about 2 months ago)
• Topics: aws, cloudwatch, cloudwatch-dashboard, cloudwatch-metrics, kinesis, kinesis-shards
• Language: Python
• Homepage:
• Size: 110 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Kinesis Data Stream CloudWatch

  

A demo that will intially deploy a Kinesis data stream with 4 shards and sample python scripts run in Cloud9 that purposly send a higher volume of messages to a single shard on Kinesis. This sets up a scenario with where a Kinesis data stream has a *hot* shard. Subseqently you enable enhanced monitoring on the Kinesis data stream and deploy a CloudWatch dashboard to identify the *hot* shard

## Instructions

1. Deploy CloudFormation for Kinesis + Cloud9

[![Launch CloudFormation Stack](https://sharkech-public.s3.amazonaws.com/misc-public/cloudformation-launch-stack.png)](https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=kinesis-cloud9&templateURL=https://sharkech-public.s3.amazonaws.com/misc-public/kinesis_Cloud9.yaml)

The CloudFormation will deploy the following architecture



2. Enable enhanced shard-level metrics on the Kinesis data stream

* Navigate to the [Kinesis Data Stream Console](https://us-east-1.console.aws.amazon.com/kinesis/home?region=us-east-1#/streams/list)

* Click on *hot-shard-data-stream*

* Navigate to the *Configuration* tab

* Enable all *Enhanced (shard-level) metrics*

3. Send data to Kinesis via. Python scripts in Cloud9

* Navigate to the [Cloud9 console](https://us-east-1.console.aws.amazon.com/cloud9/home?region=us-east-1#)

* Click on the *kinesis-cloud9* enviorment

* Run each script in a seperate Cloud9 terminal

  * ```python Kinesis_Data_Stream_Monitoring/Data_Producer/2a_send_data_to_a_shard_0.py```

  * ```python Kinesis_Data_Stream_Monitoring/Data_Producer/2b_send_data_to_a_shard_1.py```

  * ```python Kinesis_Data_Stream_Monitoring/Data_Producer/2c_send_data_to_a_shard_2.py```

  * ```python Kinesis_Data_Stream_Monitoring/Data_Producer/2d_send_data_to_a_shard_3.py```

4. Deploy CloudWatch dashboard

The CloudFormation will deploy a CloudWatch dashboard

[![Launch CloudFormation Stack](https://sharkech-public.s3.amazonaws.com/misc-public/cloudformation-launch-stack.png)](https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=cloudwatch-dashboard&templateURL=https://sharkech-public.s3.amazonaws.com/misc-public/cloud_watch_dashboard.yaml)

The cloudwatch dashboard will produce a graph that looks something like the image below. Notice that the message count is higher for shard-0000 ... compared to the other shards



shard-0000 ... is our hot shard!