{"id":15069073,"url":"https://github.com/aws-samples/amazon-kinesis-timestream-grafana","last_synced_at":"2025-07-18T20:37:12.706Z","repository":{"id":54412058,"uuid":"327882563","full_name":"aws-samples/amazon-kinesis-timestream-grafana","owner":"aws-samples","description":"Sample serverless application that reads events from Amazon Kinesis Data Streams and batches records to Amazon Timestream via Apache Flink app, visualizing results via Grafana.","archived":false,"fork":false,"pushed_at":"2021-12-14T13:28:59.000Z","size":316,"stargazers_count":30,"open_issues_count":0,"forks_count":8,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-05T08:17:37.748Z","etag":null,"topics":["amazon-kinesis","amazon-kinesis-data-analytics-flink","amazon-kinesis-streams","amazon-timestream","amazon-web-services","analytics","aws","aws-cdk","aws-lambda","gradle","grafa","java","java-11","kotlin"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit-0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aws-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-01-08T11:22:13.000Z","updated_at":"2025-03-12T15:21:08.000Z","dependencies_parsed_at":"2022-08-13T14:50:52.871Z","dependency_job_id":null,"html_url":"https://github.com/aws-samples/amazon-kinesis-timestream-grafana","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/aws-samples/amazon-kinesis-timestream-grafana","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Famazon-kinesis-timestream-grafana","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Famazon-kinesis-timestream-grafana/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Famazon-kinesis-timestream-grafana/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Famazon-kinesis-timestream-grafana/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aws-samples","download_url":"https://codeload.github.com/aws-samples/amazon-kinesis-timestream-grafana/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Famazon-kinesis-timestream-grafana/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265829225,"owners_count":23835090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-kinesis","amazon-kinesis-data-analytics-flink","amazon-kinesis-streams","amazon-timestream","amazon-web-services","analytics","aws","aws-cdk","aws-lambda","gradle","grafa","java","java-11","kotlin"],"created_at":"2024-09-25T01:40:19.396Z","updated_at":"2025-07-18T20:37:12.410Z","avatar_url":"https://github.com/aws-samples.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Amazon Timestream with Amazon Kinesis and Grafana Demo\n\nSample application that reads events from Amazon Kinesis Data Streams and batches records to Amazon Timestream,\nvisualizing results via Grafana.\n\n### Overall architecture\n\nThe overall serverless architecture will work on events streamed to [Amazon Kinesis](https://aws.amazon.com/kinesis/)\nData Streams, pulled by an [Apache Flink](https://flink.apache.org/)\nanalytics application hosted on [Amazon Kinesis Data Analytics](https://aws.amazon.com/de/kinesis/data-analytics/)\nto be batch-inserted in [Amazon Timestream](https://aws.amazon.com/timestream/) database. Finally, data will be\nvisualized directly from the Timestream database by a Grafana dashboard using the\n[Timestream datasource plugin](https://grafana.com/grafana/plugins/grafana-timestream-datasource).\n\n![Data sent from IoT device, to Amazon Kinesis Data Stream to Amazon Kinesis Data Analytics for Apache Flink to Amazon Timestream then visualized through Grafana ](assets/OverallArchitecture.svg)\n\nThe sample setup will assume that events are being streamed via Amazon Kinesis service. However, this is not a\nprecondition as any other streaming service like Kafka provisioned on EC2 or\nusing [Amazon Managed Streaming for Apache Kafka(Amazon MSK)](https://aws.amazon.com/msk/) can be used in a similar\nsetup. To simulate devices streaming data to the Kinesis data stream, an AWS Lambda function produces events and pushes\nthem to Kinesis data stream, the events streamed will contain information similar to the below sample.\n\n|DeviceID                                |Timestamp                     |temperature |humidity |voltage | watt |\n|---------------------------------------|:--------------------------:|:----------:|:-------:|:------:|-----:| \n|b974f43a-2f04-11eb-adc1-0242ac120002    |2020-09-13 11:49:42.5352820 |15.5        |70.3     |39.7    |301.44|\n\nIn Timestream data will be modeled as follows\n\n|DeviceID (Dimension)                    | measure_value::double    | measure_name | measure_type |time                       |\n|----------------------------------------|:------------------------:|:------------:|:-------------|:-------------------------:|\n|b974f43a-2f04-11eb-adc1-0242ac120002    |15.5                      |temperature   |DOUBLE        |2020-09-13 11:49:42.5352820|\n|b974f43a-2f04-11eb-adc1-0242ac120002    |70.3                      |humidity      |DOUBLE        |2020-09-13 11:49:42.5352820|\n|b974f43a-2f04-11eb-adc1-0242ac120002    |39.7                      |voltage       |DOUBLE        |2020-09-13 11:49:42.5352820|\n|b974f43a-2f04-11eb-adc1-0242ac120002    |301.4                     |watt          |DOUBLE        |2020-09-13 11:49:42.5352820|\n\nThe Device ID is mapped as a\n[Dimension](https://docs.aws.amazon.com/timestream/latest/developerguide/API_Dimension.html)\nand the Property fields measured mapped as\n[`measure_name`](https://docs.aws.amazon.com/timestream/latest/developerguide/concepts.html)\n, finally the value of the measure is mapped to the\n[`measure_value`](https://docs.aws.amazon.com/timestream/latest/developerguide/concepts.html)\n.`datatype` is set as double in this case.\n[Check the best practices on mapping](https://docs.aws.amazon.com/timestream/latest/developerguide/best-practices.html#data-modeling)\nand data modeling for a better insight on mapping your data.\n\n## Getting started\n\n### 1. Building and packaging Amazon Kinesis Data Analytics for Apache Flink application\n\nThis project provides 2 sample applications built with different toolsets. You can use either one of those as the\napplication to be deployed.\n\n1. To build an application using Java and Apache Maven, refer to [instructions here](./analytics/README.md)\n2. To build an application using Kotlin and Gradle, refer to [instructions here](./analytics-kotlin/README.md)\n\n### 2. Deploy infrastructure\n\nInfrastructure deployment will automatically use packaged application jar and upload it to an\n[Amazon S3](https://aws.amazon.com/s3/) bucket. The infrastructure utilizes multiple stacks built using an AWS CDK\nproject with Python3 language. For more information on working with the CDK and Python,\n[check the following guide](https://docs.aws.amazon.com/cdk/latest/guide/work-with-cdk-python.html). To deploy all\nstacks use the `--all` option when invoking `cdk deploy`\n\n1. Navigate to cdk folder\n1. Follow [instructions here](cdk/README.md) to create virtual environment and build stacks\n1. Make sure CDK environment is bootstrapped in the account and region you're deploying stacks to, as the stacks utilize\n   assets to deploy the Kinesis Data Analytics Flink application\n   ```shell\n   $ cdk bootstrap\n   ```\n1. Deploy infrastructure using packaged applications\n   1. To deploy infrastructure and use `Java` application as basis for Kinesis analytics application, you can directly\n      deploy the CDK stacks\n      ```shell\n      $ cdk deploy --all\n      ```\n   1. To deploy infrastructure and use `Kotlin` application as basis for Kinesis analytics application, you can\n      customize the stacks using [context variables](https://docs.aws.amazon.com/cdk/latest/guide/context.html)\n      ```shell\n      $ cdk deploy --context kda_path=../analytics-kotlin/build/libs/analytics-timestream-kotlin-sample-all.jar --all\n      ```\n\n#### Check deployed resource and kick-off pipeline\n\nOnce CDK stacks are deployed successfully you can check created AWS resources. You can directly run\nscript [`./setup.sh`](setup.sh) or follow below instructions.\n\n1. Amazon Kinesis Data Stream\n\n   Deployed through stack `amazon-kinesis-stream-stack` and is ready to receive events from sample producer.\n\n   ![Amazon EventBridge 1 min rate event triggers AWS Lambda to send JSON data to Amazon Kinesis Data Stream](assets/DeviceSimulatorSampleProducer.svg)\n\n   Producer resources are deployed through stack `sample-kinesis-stream-producer`. You can check the Lambda function\n   monitoring and logs to make sure it's being regularly called and sending events to the stream.\n\n   For more information on how the producer\n   works [check documentation](cdk/stacks/sample_kinesis_stream_producer/README.md).\n\n1. Amazon Kinesis Data Analytics for Apache Flink Application\n\n   Deployed through stack `amazon-kinesis-analytics`. Although the application is created through the stack it would\n   still not be running.\n\n   To run the application and kick-off the pipeline, simply pickup the application name from the stack\n   output `KdaApplicationName`.\n\n   Follow instructions\n   under [Run the Application section](https://docs.aws.amazon.com/kinesisanalytics/latest/java/get-started-exercise.html#get-started-exercise-7)\n   or run the following command\n    ```shell\n    $ aws kinesisanalyticsv2 start-application --application-name amazon-kinesis-analytics \\\n      --run-configuration '{ \"ApplicationRestoreConfiguration\": { \"ApplicationRestoreType\": \"SKIP_RESTORE_FROM_SNAPSHOT\" } }'\n    ```\n1. Amazon S3 bucket to store the JAR package for the application and application role\n\n   Deployed through stack `flink-source-bucket`\n1. Amazon Timestream Database\n\n   Deployed through stack `amazon-timestream`\n1. Grafana dashboard deployment\n\n   Deployed through stack `grafana`. To check the created Grafana check the output `MyFargateServiceServiceURL...`.\n   1. Grafana deployed using Amazon ECS on AWS Fargate for compute\n   1. Amazon Elastic File System (Amazon EFS) for storage\n   1. AWS SecretsManager Secret storing Grafana `admin` user password. Check the stack output `GrafanaAdminSecret` for\n      AWS Secrets Manager Secret's id storing the Grafana admin user's password.\n\n### 3. Visualizing Amazon Timestream values through Grafana\n\nTo help you get started with data visualization, we have created a sample dashboard in Grafana that visualizes data sent\nto Timestream from the sample producer. If you invoked the [`setup.sh`](setup.sh) script, it would have automatically\nperformed these steps.\n\nYou can also check the\nfollowing [video tutorial](https://docs.aws.amazon.com/timestream/latest/developerguide/Grafana.html#Grafana.video-tutorial)\nor [complete guide](https://docs.aws.amazon.com/timestream/latest/developerguide/Grafana.html) for more information.\n\n1. Install Grafana Datasource plugin\n\n   The [Grafana Amazon Timestream datasource plugin](https://grafana.com/grafana/plugins/grafana-timestream-datasource/installation)\n   is automatically installed through the infrastructure stack.\n1. Create Grafana Datasource\n\n   To create a Timeseries data source, go to Datasources, click on Add Datasource, search for Timestream, and select the\n   Timestream datasource. You can also\n   use [programmatic means described here](https://grafana.com/docs/grafana/latest/http_api/data_source/).\n\n   To use the programmatic means to create a data source:\n   1. Create an API token to be used. Make sure to replace `\u003cstack grafana.GrafanaAdminSecret.secret_value\u003e` with the\n      sceret value, escaping any characters in the password like `$` or `\"` with a preceding `\\`. And\n      replace `\u003cstack grafana.MyFargateServiceServiceURL...\u003e` with the value from stack output.\n   ```shell\n   $ grafana_token=$(\\\n    curl -X POST -u \"admin:\u003cstack grafana.GrafanaAdminSecret.secret_value\u003e\" \\\n    -H \"Content-Type: application/json\" \\\n    -d '{\"name\":\"apikeycurl\", \"role\": \"Admin\"}' \\\n    \u003cstack grafana.MyFargateServiceServiceURL...\u003e/api/auth/keys \\\n    | jq -r .key)\n   ```\n   1. Create Amazon Timestream datasource\n   ```shell\n   $ curl -X POST --insecure \\\n    -H \"Content-Type: application/json\" -H \"Authorization: Bearer ${grafana_token}\" \\\n    -d @./cdk/stacks/grafana/datasource.json \\\n    \u003cstack grafana.MyFargateServiceServiceURL...\u003e/api/datasources\n   ```\n1. Create Grafana Dashboard\n\n   Grafana dashboards can be exported and imported as `JSON`. You can find a\n   dashboard [sample json in here](cdk/stacks/grafana/dashboard.json). The defined dashboard provides sample variables\n   and visualizations by querying directly from Timestream database. The dashboard assumes the Timestream datasource is\n   your default datasource.\n\n   To import sample dashboard you\n   can [follow instructions here](https://grafana.com/docs/grafana/latest/dashboards/export-import/#importing-a-dashboard)\n   or use [programmatic means described here](https://grafana.com/docs/grafana/latest/http_api/dashboard/).\n\n   To use the programmatic means to create a dashboard:\n   ```shell\n   $ curl -X POST --insecure \\\n    -H \"Content-Type: application/json\" -H \"Authorization: Bearer ${grafana_token}\" \\\n    -d @./cdk/stacks/grafana/dashboard.json \\\n    \u003cstack grafana.MyFargateServiceServiceURL...\u003e/api/dashboards/db\n   ```\n   1. To check created dashboard navigate to \u003cstack grafana.MyFargateServiceServiceURL...\u003e\n   1. Login using admin credentials.\n   1. Check created dashboard under \"Dashboards\" -\u003e \"Manage\"\n      ![Grafana dashboards screenshot](assets/GrafanaDashboard.png)\n\n## Clean up\n\nTo delete all created stack resources you can use\n\n```shell\n  $ cdk destroy --all\n```\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis library is licensed under the MIT-0 License. See the LICENSE file.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws-samples%2Famazon-kinesis-timestream-grafana","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faws-samples%2Famazon-kinesis-timestream-grafana","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws-samples%2Famazon-kinesis-timestream-grafana/lists"}