https://github.com/sourcefuse/terraform-aws-arc-kinesis-stream
Terraform Module to provision Kinesis Streams
https://github.com/sourcefuse/terraform-aws-arc-kinesis-stream
Last synced: 4 months ago
JSON representation
Terraform Module to provision Kinesis Streams
- Host: GitHub
- URL: https://github.com/sourcefuse/terraform-aws-arc-kinesis-stream
- Owner: sourcefuse
- License: apache-2.0
- Created: 2024-06-28T12:54:04.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-08-29T16:43:31.000Z (almost 2 years ago)
- Last Synced: 2025-08-01T05:55:58.794Z (11 months ago)
- Language: HCL
- Homepage:
- Size: 4.13 MB
- Stars: 0
- Watchers: 5
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
[](#contributors-)
# [terraform-aws-arc-kinesis-stream](https://github.com/sourcefuse/terraform-aws-arc-kinesis-stream)
 
[](https://sonarcloud.io/summary/new_code?id=sourcefuse_terraform-aws-arc-kinesis-stream)
[](https://github.com/sourcefuse/terraform-aws-arc-kinesis-stream/actions/workflows/snyk.yaml)
## Overview
For more information about this repository and its usage, please see [Terraform AWS ARC GitHub Kinesis Module Usage Guide](https://github.com/sourcefuse/terraform-aws-arc-kinesis-stream/blob/main/docs/module-usage-guide/README.md).
## Introduction
SourceFuse's AWS Reference Architecture (ARC) Terraform module simplifies the creation and management of essential AWS infrastructure components. It is designed to provision and configure the following resources:
1. Amazon Kinesis Data Stream (aws_kinesis_stream):
- Name: A unique name for the Kinesis stream, generated using a local variable.
- Shard Count: The number of shards provisioned for the stream. This can be dynamically set based on the stream mode (PROVISIONED or ON_DEMAND).
- Retention Period: The length of time that data records are accessible after they are added to the stream.
- Shard Level Metrics: Metrics to be enabled for monitoring the stream at the shard level.
- Enforce Consumer Deletion: A flag to enforce consumer deletion when the stream is deleted.
- Encryption Type: The encryption type to be used for the stream.
- KMS Key ID: The KMS key ID for encrypting data records in the stream.
- Stream Mode Details: Dynamic configuration block to set the stream mode (PROVISIONED or ON_DEMAND).
- Tags: Key-value pairs to tag the stream for identification and management.
2. Amazon Kinesis Stream Consumer (aws_kinesis_stream_consumer):
- Count: The number of consumer instances to create, determined by the var.consumer_count variables.
- Name: A unique name for each consumer, generated using a local variable and the count index
- Stream ARN: The Amazon Resource Name (ARN) of the Kinesis stream to which the consumer is attached.
### Prerequisites
Before using this module, ensure you have the following:
- AWS credentials configured.
- Terraform installed.
- A working knowledge of Terraform.
## Getting Started
1. **Define the Module**
Initially, it's essential to define a Terraform module, which is organized as a distinct directory encompassing Terraform configuration files. Within this module directory, input variables and output values must be defined in the variables.tf and outputs.tf files, respectively. The following illustrates an example directory structure:
```plaintext
ec2/
|-- main.tf
|-- variables.tf
|-- outputs.tf
```
2. **Define Input Variables**
Inside the `variables.tf` or in `*.tfvars` file, you should define values for the variables that the module requires.
3. **Use the Module in Your Main Configuration**
In your main Terraform configuration file (e.g., main.tf), you can use the module. Specify the source of the module, and version, For Example
```hcl
module "kinesis" {
source = "sourcefuse/arc-kinesis-stream/aws"
version = "0.0.1"
name = "${var.namespace}-${var.environment}-kinesis"
shard_count = var.shard_count
consumer_count = var.consumer_count
tags = module.tags.tags
}
```
4. **Output Values**
Inside the `outputs.tf` file of the module, you can define output values that can be referenced in the main configuration. For example:
```hcl
output "name" {
description = "Name of the Kinesis stream."
value = module.kinesis.name
}
output "shard_count" {
description = "Number of shards provisioned."
value = module.kinesis.shard_count
}
output "stream_arn" {
description = "ARN of the Kinesis stream."
value = module.kinesis.stream_arn
}
```
## First Time Usage
***uncomment the backend block in [main.tf](./example/main.tf)***
```shell
terraform init -backend-config=config.dev.hcl
```
***If testing locally, `terraform init` should be fine***
Create a `dev` workspace
```shell
terraform workspace new dev
```
Plan Terraform
```shell
terraform plan -var-file terraform.tfvars
```
Apply Terraform
```shell
terraform apply -var-file terraform.tfvars
```
## Production Setup
```shell
terraform init -backend-config=config.prod.hcl
```
Create a `prod` workspace
```shell
terraform workspace new prod
```
Plan Terraform
```shell
terraform plan -var-file prod.tfvars
```
Apply Terraform
```shell
terraform apply -var-file prod.tfvars
```
## Cleanup
Destroy Terraform
```shell
terraform destroy -var-file dev.tfvars
```
## Requirements
| Name | Version |
|------|---------|
| [terraform](#requirement\_terraform) | ~> 1.5, < 2.0.0 |
| [aws](#requirement\_aws) | >= 4.0, < 6.0 |
## Providers
| Name | Version |
|------|---------|
| [aws](#provider\_aws) | 5.58.0 |
## Modules
No modules.
## Resources
| Name | Type |
|------|------|
| [aws_kinesis_stream.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_stream) | resource |
| [aws_kinesis_stream_consumer.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_stream_consumer) | resource |
## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| [consumer\_count](#input\_consumer\_count) | Number of consumers to register with Kinesis stream | `number` | `0` | no |
| [encryption\_type](#input\_encryption\_type) | The encryption type to use. Acceptable values are `NONE` and `KMS`. | `string` | `"KMS"` | no |
| [enforce\_consumer\_deletion](#input\_enforce\_consumer\_deletion) | A boolean that indicates all registered consumers should be deregistered from the stream so that the stream can be destroyed without error. | `bool` | `true` | no |
| [kms\_key\_id](#input\_kms\_key\_id) | The GUID for the customer-managed KMS key to use for encryption. | `string` | `"alias/aws/kinesis"` | no |
| [name](#input\_name) | Name of kinesis-stream | `string` | n/a | yes |
| [retention\_period](#input\_retention\_period) | Length of time data records are accessible after they are added to the stream. The maximum value of a stream's retention period is 168 hours. Minimum value is 24. Default is 24. | `number` | `24` | no |
| [shard\_count](#input\_shard\_count) | The number of shards that the stream will use | `number` | `1` | no |
| [shard\_level\_metrics](#input\_shard\_level\_metrics) | A list of shard-level CloudWatch metrics which can be enabled for the stream. | `list(string)` |
[
"IncomingBytes",
"OutgoingBytes"
]
| no |
| [stream\_mode](#input\_stream\_mode) | Specifies the capacity mode of the stream. Must be either `PROVISIONED` or `ON_DEMAND`. If `ON_DEMAND` is used, then `shard_count` is ignored. | `string` | `null` | no |
| [tags](#input\_tags) | Tags for AWS elasticache redis | `map(string)` | n/a | yes |
## Outputs
| Name | Description |
|------|-------------|
| [consumers](#output\_consumers) | List of consumers registered with Kinesis stream. |
| [name](#output\_name) | Name of the Kinesis stream. |
| [shard\_count](#output\_shard\_count) | Number of shards provisioned. |
| [stream\_arn](#output\_stream\_arn) | ARN of the Kinesis stream. |
### Git commits
while Contributing or doing git commit please specify the breaking change in your commit message whether its major,minor or patch
For Example
```sh
git commit -m "your commit message #major"
```
By specifying this , it will bump the version and if you dont specify this in your commit message then by default it will consider patch and will bump that accordingly
## Development
### Prerequisites
- [terraform](https://learn.hashicorp.com/terraform/getting-started/install#installing-terraform)
- [terraform-docs](https://github.com/segmentio/terraform-docs)
- [pre-commit](https://pre-commit.com/#install)
- [golang](https://golang.org/doc/install#install)
- [golint](https://github.com/golang/lint#installation)
### Configurations
- Configure pre-commit hooks
```sh
pre-commit install
```
### Tests
- Tests are available in `test` directory
- Configure the dependencies
```sh
cd test/
go mod init github.com/sourcefuse/terraform-aws-refarch-
go get github.com/gruntwork-io/terratest/modules/terraform
```
- Now execute the test
```sh
go test -timeout 30m
```
## Authors
This project is authored by:
- SourceFuse
## Contributors ✨
Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!
