https://github.com/stacktape/starter-kafka-stream-processing
Stacktape starter project
https://github.com/stacktape/starter-kafka-stream-processing
Last synced: 3 months ago
JSON representation
Stacktape starter project
- Host: GitHub
- URL: https://github.com/stacktape/starter-kafka-stream-processing
- Owner: stacktape
- Created: 2023-08-16T12:52:34.000Z (almost 2 years ago)
- Default Branch: master
- Last Pushed: 2025-01-20T01:56:38.000Z (4 months ago)
- Last Synced: 2025-01-20T02:41:45.904Z (4 months ago)
- Language: TypeScript
- Homepage: https://stacktape.com
- Size: 5.86 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Kafka stream processing
> [!TIP]
> To deploy this project using **GUI-based flow**, navigate to [console](https://console.stacktape.com/create-new-project/git-project-using-console?name=my-stacktape-app&repositoryType=public&repositoryUrl=https://github.com/stacktape/starter-kafka-stream-processing)
- simple Lambda-based stream-processing app.
- The application uses 2 [Lambda functions](https://docs.stacktape.com/compute-resources/lambda-functions/),
[Upstash Kafka topic](https://docs.stacktape.com/resources/upstash-kafka-topics/). The first Lambda writes the message
from a HTTP request to the Kafka topic. The second lambda function processes the batched messages.
- This project includes a pre-configured [stacktape.yml configuration](stacktape.yml).
The configured infrastructure is described in the [stack description section](#stack-description)
## Pricing
- The infrastructure required for this application uses exclusively "serverless", pay-per-use infrastructure. If your load won't get high, these costs will be close to $0.
## Prerequisites
1. **AWS account**. If you don't have one, [create new account here](https://portal.aws.amazon.com/billing/signup).
2. **Stacktape account**. If you don't have one, [create new account here](https://console.stacktape.com/sign-up).
3. **Stacktape installed**.
Install on Windows (Powershell)
```bash
iwr https://installs.stacktape.com/windows.ps1 -useb | iex
```
Install on Linux
```bash
curl -L https://installs.stacktape.com/linux.sh | sh
```
Install on MacOS
```bash
curl -L https://installs.stacktape.com/macos.sh | sh
```
Install on MacOS ARM (Apple silicon)
```bash
curl -L https://installs.stacktape.com/macos-arm.sh | sh
```
4. **Upstash account**. If you don't have one, [create new account here](https://console.upstash.com/login).
## 1. Generate your project
To initialize the project, use
```bash
stacktape init --starterId kafka-stream-processing
```
## 2. Before deploy
- Fill in your Upstash credentials in the `providerConfig.upstash` section of the stacktape.yml config file. You can get your API key in the [Upstash console](https://console.upstash.com/account/api).
## 3. Deploy your stack
The deployment will take ~5-15 minutes. Subsequent deploys will be significantly faster.
Deploy from local machine
The deployment from local machine will build and deploy the application from your system. This means you also need to have:
- Docker. To install Docker on your system, you can follow [this guide](https://docs.docker.com/get-docker/).- Node.js installed.
To perform the deployment, use the following command:
```bash
stacktape deploy --stage <> --region <>
```
`stage` is an arbitrary name of your environment (for example **staging**, **production** or **dev-john**)
`region` is the AWS region, where your stack will be deployed to. All the available regions are listed below.
| Region name & Location | code |
| -------------------------- | -------------- |
| Europe (Ireland) | eu-west-1 |
| Europe (London) | eu-west-2 |
| Europe (Frankfurt) | eu-central-1 |
| Europe (Milan) | eu-south-1 |
| Europe (Paris) | eu-west-3 |
| Europe (Stockholm) | eu-north-1 |
| US East (Ohio) | us-east-2 |
| US East (N. Virginia) | us-east-1 |
| US West (N. California) | us-west-1 |
| US West (Oregon) | us-west-2 |
| Canada (Central) | ca-central-1 |
| Africa (Cape Town) | af-south-1 |
| Asia Pacific (Hong Kong) | ap-east-1 |
| Asia Pacific (Mumbai) | ap-south-1 |
| Asia Pacific (Osaka-Local) | ap-northeast-3 |
| Asia Pacific (Seoul) | ap-northeast-2 |
| Asia Pacific (Singapore) | ap-southeast-1 |
| Asia Pacific (Sydney) | ap-southeast-2 |
| Asia Pacific (Tokyo) | ap-northeast-1 |
| China (Beijing) | cn-north-1 |
| China (Ningxia) | cn-northwest-1 |
| Middle East (Bahrain) | me-south-1 |
| South America (São Paulo) | sa-east-1 |
Deploy using AWS CodeBuild pipeline
Deployment using AWS CodeBuild will build and deploy your application inside [AWS CodeBuild pipeline](https://aws.amazon.com/codebuild/). To perform the deployment, use
```bash
stacktape codebuild:deploy --stage <> --region <>
```
`stage` is an arbitrary name of your environment (for example **staging**, **production** or **dev-john**)
`region` is the AWS region, where your stack will be deployed to. All the available regions are listed below.
| Region name & Location | code |
| -------------------------- | -------------- |
| Europe (Ireland) | eu-west-1 |
| Europe (London) | eu-west-2 |
| Europe (Frankfurt) | eu-central-1 |
| Europe (Milan) | eu-south-1 |
| Europe (Paris) | eu-west-3 |
| Europe (Stockholm) | eu-north-1 |
| US East (Ohio) | us-east-2 |
| US East (N. Virginia) | us-east-1 |
| US West (N. California) | us-west-1 |
| US West (Oregon) | us-west-2 |
| Canada (Central) | ca-central-1 |
| Africa (Cape Town) | af-south-1 |
| Asia Pacific (Hong Kong) | ap-east-1 |
| Asia Pacific (Mumbai) | ap-south-1 |
| Asia Pacific (Osaka-Local) | ap-northeast-3 |
| Asia Pacific (Seoul) | ap-northeast-2 |
| Asia Pacific (Singapore) | ap-southeast-1 |
| Asia Pacific (Sydney) | ap-southeast-2 |
| Asia Pacific (Tokyo) | ap-northeast-1 |
| China (Beijing) | cn-north-1 |
| China (Ningxia) | cn-northwest-1 |
| Middle East (Bahrain) | me-south-1 |
| South America (São Paulo) | sa-east-1 |
Deploy using Github actions CI/CD pipeline
1. If you don't have one, create a new repository at https://github.com/new
2. Create Github repository secrets: https://docs.stacktape.com/user-guides/ci-cd/#2-create-github-repository-secrets
3. Replace `<>` and `<>` in the .github/workflows/deploy.yml file.
4. `git init --initial-branch=main`
5. `git add .`
6. `git commit -m "setup stacktape project"`
7. `git remote add origin [email protected]:<>/<>.git`
8. `git push -u origin main`
9. To monitor the deployment progress, navigate to your github project and select the Actions tab
`stage` is an arbitrary name of your environment (for example **staging**, **production** or **dev-john**)
`region` is the AWS region, where your stack will be deployed to. All the available regions are listed below.
| Region name & Location | code |
| -------------------------- | -------------- |
| Europe (Ireland) | eu-west-1 |
| Europe (London) | eu-west-2 |
| Europe (Frankfurt) | eu-central-1 |
| Europe (Milan) | eu-south-1 |
| Europe (Paris) | eu-west-3 |
| Europe (Stockholm) | eu-north-1 |
| US East (Ohio) | us-east-2 |
| US East (N. Virginia) | us-east-1 |
| US West (N. California) | us-west-1 |
| US West (Oregon) | us-west-2 |
| Canada (Central) | ca-central-1 |
| Africa (Cape Town) | af-south-1 |
| Asia Pacific (Hong Kong) | ap-east-1 |
| Asia Pacific (Mumbai) | ap-south-1 |
| Asia Pacific (Osaka-Local) | ap-northeast-3 |
| Asia Pacific (Seoul) | ap-northeast-2 |
| Asia Pacific (Singapore) | ap-southeast-1 |
| Asia Pacific (Sydney) | ap-southeast-2 |
| Asia Pacific (Tokyo) | ap-northeast-1 |
| China (Beijing) | cn-north-1 |
| China (Ningxia) | cn-northwest-1 |
| Middle East (Bahrain) | me-south-1 |
| South America (São Paulo) | sa-east-1 |
Deploy using Gitlab CI pipeline
1. If you don't have one, create a new repository at https://gitlab.com/projects/new
2. Create Gitlab repository secrets: https://docs.stacktape.com/user-guides/ci-cd/#2-create-gitlab-repository-secrets
3. replace `<>` and `<>` in the .gitlab-ci.yml file.
4. `git init --initial-branch=main`
5. `git add .`
6. `git commit -m "setup stacktape project"`
7. `git remote add origin [email protected]:<>/<>.git`
8. `git push -u origin main`
9. `To monitor the deployment progress, navigate to your gitlab project and select CI/CD->jobs`
`stage` is an arbitrary name of your environment (for example **staging**, **production** or **dev-john**)
`region` is the AWS region, where your stack will be deployed to. All the available regions are listed below.
| Region name & Location | code |
| -------------------------- | -------------- |
| Europe (Ireland) | eu-west-1 |
| Europe (London) | eu-west-2 |
| Europe (Frankfurt) | eu-central-1 |
| Europe (Milan) | eu-south-1 |
| Europe (Paris) | eu-west-3 |
| Europe (Stockholm) | eu-north-1 |
| US East (Ohio) | us-east-2 |
| US East (N. Virginia) | us-east-1 |
| US West (N. California) | us-west-1 |
| US West (Oregon) | us-west-2 |
| Canada (Central) | ca-central-1 |
| Africa (Cape Town) | af-south-1 |
| Asia Pacific (Hong Kong) | ap-east-1 |
| Asia Pacific (Mumbai) | ap-south-1 |
| Asia Pacific (Osaka-Local) | ap-northeast-3 |
| Asia Pacific (Seoul) | ap-northeast-2 |
| Asia Pacific (Singapore) | ap-southeast-1 |
| Asia Pacific (Sydney) | ap-southeast-2 |
| Asia Pacific (Tokyo) | ap-northeast-1 |
| China (Beijing) | cn-north-1 |
| China (Ningxia) | cn-northwest-1 |
| Middle East (Bahrain) | me-south-1 |
| South America (São Paulo) | sa-east-1 |
## 4. Test your application
After a successful deployment, some information about the stack will be printed to the terminal (**URLs** of the deployed services, links to **logs**, **metrics**, etc.).
1. Make multiple requests to https://YOUR_MAIN_GATEWAY_URL/produce/YOUR_MESSAGE
2. Go to kafkaConsumer -> logs (AWS link will be printed to the console) and see how the messages are handled in
batches.
## 5. Run the application in development mode
To run functions in the development mode (remotely on AWS), you can use the
[dev command](https://docs.stacktape.com/cli/commands/dev/). For example, to develop and debug lambda function `kafkaProducer`, you can use
```bash
stacktape dev --region <> --stage <> --resourceName kafkaProducer
```
The command will:
- quickly re-build and re-deploy your new function code
- watch for the function logs and pretty-print them to the terminal
The function is rebuilt and redeployed, when you either:
- type `rs + enter` to the terminal
- use the `--watch` option and one of your source code files changes
## 6. Hotswap deploys
- Stacktape deployments use [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) under the hood. It
brings a lot of guarantees and convenience, but can be slow for certain use-cases.
- To speed up the deployment, you can use the `--hotSwap` flag that avoids Cloudformation.
- Hotswap deployments work only for source code changes (for lambda function, containers and batch jobs) and for content uploads to buckets.
- If the update deployment is not hot-swappable, Stacktape will automatically fall back to using a Cloudformation deployment.
```bash
stacktape deploy --hotSwap --stage <> --region <>
```
## 7. Delete your stack
- If you no longer want to use your stack, you can delete it.
- Stacktape will automatically delete every infrastructure resource and deployment artifact associated with your stack.
```bash
stacktape delete --stage <> --region <>
```
# Stack description
Stacktape uses a simple `stacktape.yml` configuration file to describe infrastructure resources, packaging, deployment
pipeline and other aspects of your project.
You can deploy your project to multiple environments (stages) - for
example `production`, `staging` or `dev-john`. A stack is a running instance of an project. It consists of your application
code (if any) and the infrastructure resources required to run it.
The configuration for this project is described below.
## 1. Resources
- Every resource must have an arbitrary, alphanumeric name (A-z0-9).
- Stacktape resources consist of multiple underlying AWS or 3rd party resources.
### 1.1 HTTP API Gateway
API Gateway receives requests and invokes our kafkaProducer function.
You can specify [more properties](https://docs.stacktape.com/resources/http-api-gateways/) on the gateway, in our case
we are going with the default setup.
```yml
resources:
mainApiGateway:
type: http-api-gateway
```
### 1.2 Upstash Kafka topic
Upstash Kafka topic is a message hub within our application. Both consumer and producer functions are using it.
You can specify [more properties](https://docs.stacktape.com/resources/upstash-kafka-topics/) on the topic, i.e specify
custom Upstash Kafka cluster or set the number of topic partitions. In this case, we are keeping the defaults.
```yml
kafkaTopic:
type: upstash-kafka-topic
```
### 1.3 Functions
Core of our application consists of two serverless functions:
- **producer function** - produces messages into kafka topic (triggered by HTTP API gateway)
- **consumer function** - consumes messages from kafka topic (triggered when there are messages in the topic)
#### 1.3.1 Producer function
Producer function is configured as follows:
- **Packaging** - determines how the lambda artifact is built. The easiest and most optimized way to build the lambda
from Typescript/Javascript is using `stacktape-lambda-buildpack`. We only need to configure `entryfilePath`. Stacktape
automatically transpiles and builds the application code with all of its dependencies, creates the lambda zip
artifact, and uploads it to a pre-created S3 bucket on AWS. You can also use
[other types of packaging](https://docs.stacktape.com/configuration/packaging/#packaging-lambda-functions).
- **ConnectTo list** - we are adding the topic `kafkaTopic` into `connectTo` list. By doing this, Stacktape will
automatically inject relevant environment variables into the container runtime (such as its address, rest token, etc.)
- **Events** - Events determine how is function triggered. In this case, we are triggering the function when an event
(HTTP request) is delivered to the HTTP API gateway to URL path `/produce/{message}`, where `{message}` is a path
parameter(message can be arbitrary value). The event(request) including the path parameter is passed to the function
handler as an argument.
```yml
kafkaProducer:
type: function
properties:
packaging:
type: stacktape-lambda-buildpack
properties:
entryfilePath: ./src/kafka-producer.ts
events:
- type: http-api-gateway
properties:
httpApiGatewayName: mainApiGateway
path: /produce/{message}
method: GET
connectTo:
- kafkaTopic
```
#### 1.3.2 Consumer function
Consumer function is configured as follows:
- **Packaging** - determines how the lambda artifact is built. The easiest and most optimized way to build the lambda
from Typescript/Javascript is using `stacktape-lambda-buildpack`. We only need to configure `entryfilePath`. Stacktape
automatically transpiles and builds the application code with all of its dependencies, creates the lambda zip
artifact, and uploads it to a pre-created S3 bucket on AWS. You can also use
[other types of packaging](https://docs.stacktape.com/configuration/packaging/#packaging-lambda-functions).
- **Events** - Events determine how is function triggered(invoked). In this case, we are triggering the function when
there are messages in the upstash kafka topic. We are also configuring `batchSize`(amount of messages that can be
included in one invocation) and `maxBatchWindowSeconds`(amount of seconds, messages are collected, before the function
is invoked).
```yml
kafkaConsumer:
type: function
properties:
packaging:
type: stacktape-lambda-buildpack
properties:
entryfilePath: ./src/kafka-consumer.ts
events:
- type: kafka-topic
properties:
upstashKafkaTopic: kafkaTopic
batchSize: 5
maxBatchWindowSeconds: 10
```