{"id":19447069,"url":"https://github.com/stacktape/starter-kafka-stream-processing","last_synced_at":"2026-04-18T11:33:18.467Z","repository":{"id":226143552,"uuid":"679259765","full_name":"stacktape/starter-kafka-stream-processing","owner":"stacktape","description":"Stacktape starter project","archived":false,"fork":false,"pushed_at":"2025-01-20T01:56:38.000Z","size":6,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-25T08:45:09.526Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://stacktape.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stacktape.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-16T12:52:34.000Z","updated_at":"2025-01-20T01:56:39.000Z","dependencies_parsed_at":"2024-04-13T20:22:40.109Z","dependency_job_id":"22d344e8-cc7e-4f88-aeb5-fbf80960189f","html_url":"https://github.com/stacktape/starter-kafka-stream-processing","commit_stats":null,"previous_names":["stacktape/starter-kafka-stream-processing"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/stacktape/starter-kafka-stream-processing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stacktape%2Fstarter-kafka-stream-processing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stacktape%2Fstarter-kafka-stream-processing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stacktape%2Fstarter-kafka-stream-processing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stacktape%2Fstarter-kafka-stream-processing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stacktape","download_url":"https://codeload.github.com/stacktape/starter-kafka-stream-processing/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stacktape%2Fstarter-kafka-stream-processing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31967016,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T16:16:19.667Z","updated_at":"2026-04-18T11:33:18.461Z","avatar_url":"https://github.com/stacktape.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Kafka stream processing\n\n\u003e [!TIP]\n\u003e To deploy this project using **GUI-based flow**, navigate to [console](https://console.stacktape.com/create-new-project/git-project-using-console?name=my-stacktape-app\u0026repositoryType=public\u0026repositoryUrl=https://github.com/stacktape/starter-kafka-stream-processing)\n\n- simple Lambda-based stream-processing app.\n- The application uses 2 [Lambda functions](https://docs.stacktape.com/compute-resources/lambda-functions/),\n  [Upstash Kafka topic](https://docs.stacktape.com/resources/upstash-kafka-topics/). The first Lambda writes the message\n  from a HTTP request to the Kafka topic. The second lambda function processes the batched messages.\n- This project includes a pre-configured [stacktape.yml configuration](stacktape.yml).\nThe configured infrastructure is described in the [stack description section](#stack-description)\n\n## Pricing\n\n\n- The infrastructure required for this application uses exclusively \"serverless\", pay-per-use infrastructure. If your load won't get high, these costs will be close to $0.\n\n## Prerequisites\n\n1. **AWS account**. If you don't have one, [create new account here](https://portal.aws.amazon.com/billing/signup).\n\n2. **Stacktape account**. If you don't have one, [create new account here](https://console.stacktape.com/sign-up).\n\n3. **Stacktape installed**.\n\n  \u003cdetails\u003e\n  \u003csummary\u003eInstall on Windows (Powershell)\u003c/summary\u003e\n\n  ```bash\n  iwr https://installs.stacktape.com/windows.ps1 -useb | iex\n  ```\n\n  \u003c/details\u003e\n  \u003cdetails\u003e\n  \u003csummary\u003eInstall on Linux\u003c/summary\u003e\n\n  ```bash\n  curl -L https://installs.stacktape.com/linux.sh | sh\n  ```\n\n  \u003c/details\u003e\n  \u003cdetails\u003e\n  \u003csummary\u003eInstall on MacOS\u003c/summary\u003e\n\n  ```bash\n  curl -L https://installs.stacktape.com/macos.sh | sh\n  ```\n\n  \u003c/details\u003e\n  \u003cdetails\u003e\n  \u003csummary\u003eInstall on MacOS ARM (Apple silicon)\u003c/summary\u003e\n\n  ```bash\n  curl -L https://installs.stacktape.com/macos-arm.sh | sh\n  ```\n\n  \u003c/details\u003e\n\n\n4. **Upstash account**. If you don't have one, [create new account here](https://console.upstash.com/login).\n\n## 1. Generate your project\nTo initialize the project, use\n\n```bash\nstacktape init --starterId kafka-stream-processing\n```\n\n## 2. Before deploy\n- Fill in your Upstash credentials in the `providerConfig.upstash` section of the stacktape.yml config file. You can get your API key in the [Upstash console](https://console.upstash.com/account/api).\n\n## 3. Deploy your stack\n\nThe deployment will take ~5-15 minutes. Subsequent deploys will be significantly faster.\n\n\u003cdetails\u003e\n\u003csummary\u003eDeploy from local machine\u003c/summary\u003e\n\n\u003cbr /\u003e\n\nThe deployment from local machine will build and deploy the application from your system. This means you also need to have:\n- Docker. To install Docker on your system, you can follow [this guide](https://docs.docker.com/get-docker/).- Node.js installed.\n\n\u003cbr /\u003e\n\nTo perform the deployment, use the following command:\n\n```bash\nstacktape deploy --projectName \u003c\u003cproject-name\u003e\u003e --stage \u003c\u003cstage\u003e\u003e --region \u003c\u003cregion\u003e\u003e\n```\n\n`stage` is an arbitrary name of your environment (for example **staging**, **production** or **dev-john**)\n\n`region` is the AWS region, where your stack will be deployed to. All the available regions are listed below.\n\n`projectName` is the name of your project. You can create it in the console or interactively using CLI.\n\n\u003cbr /\u003e\n\n| Region name \u0026 Location     | code           |\n  | -------------------------- | -------------- |\n  | Europe (Ireland)           | eu-west-1      |\n  | Europe (London)            | eu-west-2      |\n  | Europe (Frankfurt)         | eu-central-1   |\n  | Europe (Milan)             | eu-south-1     |\n  | Europe (Paris)             | eu-west-3      |\n  | Europe (Stockholm)         | eu-north-1     |\n  | US East (Ohio)             | us-east-2      |\n  | US East (N. Virginia)      | us-east-1      |\n  | US West (N. California)    | us-west-1      |\n  | US West (Oregon)           | us-west-2      |\n  | Canada (Central)           | ca-central-1   |\n  | Africa (Cape Town)         | af-south-1     |\n  | Asia Pacific (Hong Kong)   | ap-east-1      |\n  | Asia Pacific (Mumbai)      | ap-south-1     |\n  | Asia Pacific (Osaka-Local) | ap-northeast-3 |\n  | Asia Pacific (Seoul)       | ap-northeast-2 |\n  | Asia Pacific (Singapore)   | ap-southeast-1 |\n  | Asia Pacific (Sydney)      | ap-southeast-2 |\n  | Asia Pacific (Tokyo)       | ap-northeast-1 |\n  | China (Beijing)            | cn-north-1     |\n  | China (Ningxia)            | cn-northwest-1 |\n  | Middle East (Bahrain)      | me-south-1     |\n  | South America (São Paulo)  | sa-east-1      |\n\n\u003c/details\u003e\n\u003cdetails\u003e\n\u003csummary\u003eDeploy using AWS CodeBuild pipeline\u003c/summary\u003e\n\n\u003cbr /\u003e\n\nDeployment using AWS CodeBuild will build and deploy your application inside [AWS CodeBuild pipeline](https://aws.amazon.com/codebuild/). To perform the deployment, use\n\n```bash\nstacktape codebuild:deploy --stage \u003c\u003cstage\u003e\u003e --region \u003c\u003cregion\u003e\u003e --projectName \u003c\u003cproject-name\u003e\u003e\n```\n\n`stage` is an arbitrary name of your environment (for example **staging**, **production** or **dev-john**)\n\n`region` is the AWS region, where your stack will be deployed to. All the available regions are listed below.\n\n`projectName` is the name of your project. You can create it in the console or interactively using CLI.\n\n\u003cbr /\u003e\n\n| Region name \u0026 Location     | code           |\n  | -------------------------- | -------------- |\n  | Europe (Ireland)           | eu-west-1      |\n  | Europe (London)            | eu-west-2      |\n  | Europe (Frankfurt)         | eu-central-1   |\n  | Europe (Milan)             | eu-south-1     |\n  | Europe (Paris)             | eu-west-3      |\n  | Europe (Stockholm)         | eu-north-1     |\n  | US East (Ohio)             | us-east-2      |\n  | US East (N. Virginia)      | us-east-1      |\n  | US West (N. California)    | us-west-1      |\n  | US West (Oregon)           | us-west-2      |\n  | Canada (Central)           | ca-central-1   |\n  | Africa (Cape Town)         | af-south-1     |\n  | Asia Pacific (Hong Kong)   | ap-east-1      |\n  | Asia Pacific (Mumbai)      | ap-south-1     |\n  | Asia Pacific (Osaka-Local) | ap-northeast-3 |\n  | Asia Pacific (Seoul)       | ap-northeast-2 |\n  | Asia Pacific (Singapore)   | ap-southeast-1 |\n  | Asia Pacific (Sydney)      | ap-southeast-2 |\n  | Asia Pacific (Tokyo)       | ap-northeast-1 |\n  | China (Beijing)            | cn-north-1     |\n  | China (Ningxia)            | cn-northwest-1 |\n  | Middle East (Bahrain)      | me-south-1     |\n  | South America (São Paulo)  | sa-east-1      |\n\n\u003c/details\u003e\n\u003cdetails\u003e\n\u003csummary\u003eDeploy using Github actions CI/CD pipeline\u003c/summary\u003e\n\n\u003cbr /\u003e\n\n1. If you don't have one, create a new repository at https://github.com/new\n2. Create Github repository secrets: https://docs.stacktape.com/user-guides/ci-cd/#2-create-github-repository-secrets\n3. Replace `\u003c\u003cstage\u003e\u003e` and `\u003c\u003cregion\u003e\u003e` in the .github/workflows/deploy.yml file.\n4. `git init --initial-branch=main`\n5. `git add .`\n6. `git commit -m \"setup stacktape project\"`\n7. `git remote add origin git@github.com:\u003c\u003cnamespace-name\u003e\u003e/\u003c\u003crepo-name\u003e\u003e.git`\n8. `git push -u origin main`\n9. To monitor the deployment progress, navigate to your github project and select the Actions tab\n\n`stage` is an arbitrary name of your environment (for example **staging**, **production** or **dev-john**)\n\n`region` is the AWS region, where your stack will be deployed to. All the available regions are listed below.\n\n`projectName` is the name of your project. You can create it in the console or interactively using CLI.\n\n\u003cbr /\u003e\n\n| Region name \u0026 Location     | code           |\n  | -------------------------- | -------------- |\n  | Europe (Ireland)           | eu-west-1      |\n  | Europe (London)            | eu-west-2      |\n  | Europe (Frankfurt)         | eu-central-1   |\n  | Europe (Milan)             | eu-south-1     |\n  | Europe (Paris)             | eu-west-3      |\n  | Europe (Stockholm)         | eu-north-1     |\n  | US East (Ohio)             | us-east-2      |\n  | US East (N. Virginia)      | us-east-1      |\n  | US West (N. California)    | us-west-1      |\n  | US West (Oregon)           | us-west-2      |\n  | Canada (Central)           | ca-central-1   |\n  | Africa (Cape Town)         | af-south-1     |\n  | Asia Pacific (Hong Kong)   | ap-east-1      |\n  | Asia Pacific (Mumbai)      | ap-south-1     |\n  | Asia Pacific (Osaka-Local) | ap-northeast-3 |\n  | Asia Pacific (Seoul)       | ap-northeast-2 |\n  | Asia Pacific (Singapore)   | ap-southeast-1 |\n  | Asia Pacific (Sydney)      | ap-southeast-2 |\n  | Asia Pacific (Tokyo)       | ap-northeast-1 |\n  | China (Beijing)            | cn-north-1     |\n  | China (Ningxia)            | cn-northwest-1 |\n  | Middle East (Bahrain)      | me-south-1     |\n  | South America (São Paulo)  | sa-east-1      |\n\n\u003c/details\u003e\n\u003cdetails\u003e\n\u003csummary\u003eDeploy using Gitlab CI pipeline\u003c/summary\u003e\n\n\u003cbr /\u003e\n\n1. If you don't have one, create a new repository at https://gitlab.com/projects/new\n2. Create Gitlab repository secrets: https://docs.stacktape.com/user-guides/ci-cd/#2-create-gitlab-repository-secrets\n3. replace `\u003c\u003cstage\u003e\u003e` and `\u003c\u003cregion\u003e\u003e` in the .gitlab-ci.yml file.\n4. `git init --initial-branch=main`\n5. `git add .`\n6. `git commit -m \"setup stacktape project\"`\n7. `git remote add origin git@gitlab.com:\u003c\u003cnamespace-name\u003e\u003e/\u003c\u003crepo-name\u003e\u003e.git`\n8. `git push -u origin main`\n9. `To monitor the deployment progress, navigate to your gitlab project and select CI/CD-\u003ejobs`\n\n`stage` is an arbitrary name of your environment (for example **staging**, **production** or **dev-john**)\n\n`region` is the AWS region, where your stack will be deployed to. All the available regions are listed below.\n\n`projectName` is the name of your project. You can create it in the console or interactively using CLI.\n\n\u003cbr /\u003e\n\n| Region name \u0026 Location     | code           |\n  | -------------------------- | -------------- |\n  | Europe (Ireland)           | eu-west-1      |\n  | Europe (London)            | eu-west-2      |\n  | Europe (Frankfurt)         | eu-central-1   |\n  | Europe (Milan)             | eu-south-1     |\n  | Europe (Paris)             | eu-west-3      |\n  | Europe (Stockholm)         | eu-north-1     |\n  | US East (Ohio)             | us-east-2      |\n  | US East (N. Virginia)      | us-east-1      |\n  | US West (N. California)    | us-west-1      |\n  | US West (Oregon)           | us-west-2      |\n  | Canada (Central)           | ca-central-1   |\n  | Africa (Cape Town)         | af-south-1     |\n  | Asia Pacific (Hong Kong)   | ap-east-1      |\n  | Asia Pacific (Mumbai)      | ap-south-1     |\n  | Asia Pacific (Osaka-Local) | ap-northeast-3 |\n  | Asia Pacific (Seoul)       | ap-northeast-2 |\n  | Asia Pacific (Singapore)   | ap-southeast-1 |\n  | Asia Pacific (Sydney)      | ap-southeast-2 |\n  | Asia Pacific (Tokyo)       | ap-northeast-1 |\n  | China (Beijing)            | cn-north-1     |\n  | China (Ningxia)            | cn-northwest-1 |\n  | Middle East (Bahrain)      | me-south-1     |\n  | South America (São Paulo)  | sa-east-1      |\n\n\u003c/details\u003e\n\n## 4. Test your application\n\nAfter a successful deployment, some information about the stack will be printed to the terminal (**URLs** of the deployed services, links to **logs**, **metrics**, etc.).\n\n1. Make multiple requests to https://YOUR_MAIN_GATEWAY_URL/produce/YOUR_MESSAGE\n2. Go to kafkaConsumer -\u003e logs (AWS link will be printed to the console) and see how the messages are handled in\n   batches.\n\n\n## 5. Run the application in development mode\nTo run functions in the development mode (remotely on AWS), you can use the\n[dev command](https://docs.stacktape.com/cli/commands/dev/). For example, to develop and debug lambda function `kafkaProducer`, you can use\n\n```bash\nstacktape dev --region \u003c\u003cyour-region\u003e\u003e --stage \u003c\u003cstage\u003e\u003e --resourceName kafkaProducer\n```\n\nThe command will:\n- quickly re-build and re-deploy your new function code\n- watch for the function logs and pretty-print them to the terminal\n\nThe function is rebuilt and redeployed, when you either:\n- type `rs + enter` to the terminal\n- use the `--watch` option and one of your source code files changes\n\n## 6. Hotswap deploys\n- Stacktape deployments use [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) under the hood. It\n  brings a lot of guarantees and convenience, but can be slow for certain use-cases.\n\n- To speed up the deployment, you can use the `--hotSwap` flag which avoids using Cloudformation.\n- Hotswap deployments work only for source code changes (for lambda function, containers and batch jobs) and for content uploads to buckets.\n- If the update deployment is not hot-swappable, Stacktape will automatically fall back to using a Cloudformation deployment.\n```bash\nstacktape deploy --hotSwap --stage \u003c\u003cstage\u003e\u003e --region \u003c\u003cregion\u003e\u003e --projectName \u003c\u003cproject-name\u003e\u003e\n```\n\n## 7. Delete your stack\n\n- If you no longer want to use your stack, you can delete it.\n- Stacktape will automatically delete every infrastructure resource and deployment artifact associated with your stack.\n\n```bash\nstacktape delete --stage \u003c\u003cstage\u003e\u003e --region \u003c\u003cregion\u003e\u003e\n```\n\n# Stack description\n\n  Stacktape uses a simple `stacktape.yml` configuration file to describe infrastructure resources, packaging, deployment\n  pipeline and other aspects of your project.\n\n  You can deploy your project to multiple environments (stages) - for\n  example `production`, `staging` or `dev-john`. A stack is a running instance of an project. It consists of your application\n  code (if any) and the infrastructure resources required to run it.\n\n  The configuration for this project is described below.\n\n  ## 1. Resources\n\n  - Every resource must have an arbitrary, alphanumeric name (A-z0-9).\n  - Stacktape resources consist of multiple underlying AWS or 3rd party resources.\n### 1.1 HTTP API Gateway\n\nAPI Gateway receives requests and invokes our kafkaProducer function.\n\nYou can specify [more properties](https://docs.stacktape.com/resources/http-api-gateways/) on the gateway, in our case\nwe are going with the default setup.\n\n```yml\nresources:\n  mainApiGateway:\n    type: http-api-gateway\n```\n\n### 1.2 Upstash Kafka topic\n\nUpstash Kafka topic is a message hub within our application. Both consumer and producer functions are using it.\n\nYou can specify [more properties](https://docs.stacktape.com/resources/upstash-kafka-topics/) on the topic, i.e specify\ncustom Upstash Kafka cluster or set the number of topic partitions. In this case, we are keeping the defaults.\n\n```yml\nkafkaTopic:\n  type: upstash-kafka-topic\n```\n\n### 1.3 Functions\n\nCore of our application consists of two serverless functions:\n\n- **producer function** - produces messages into kafka topic (triggered by HTTP API gateway)\n- **consumer function** - consumes messages from kafka topic (triggered when there are messages in the topic)\n\n#### 1.3.1 Producer function\n\nProducer function is configured as follows:\n\n- **Packaging** - determines how the lambda artifact is built. The easiest and most optimized way to build the lambda\n  from Typescript/Javascript is using `stacktape-lambda-buildpack`. We only need to configure `entryfilePath`. Stacktape\n  automatically transpiles and builds the application code with all of its dependencies, creates the lambda zip\n  artifact, and uploads it to a pre-created S3 bucket on AWS. You can also use\n  [other types of packaging](https://docs.stacktape.com/configuration/packaging/#packaging-lambda-functions).\n- **ConnectTo list** - we are adding the topic `kafkaTopic` into `connectTo` list. By doing this, Stacktape will\n  automatically inject relevant environment variables into the container runtime (such as its address, rest token, etc.)\n- **Events** - Events determine how is function triggered. In this case, we are triggering the function when an event\n  (HTTP request) is delivered to the HTTP API gateway to URL path `/produce/{message}`, where `{message}` is a path\n  parameter(message can be arbitrary value). The event(request) including the path parameter is passed to the function\n  handler as an argument.\n\n```yml\nkafkaProducer:\n  type: function\n  properties:\n    packaging:\n      type: stacktape-lambda-buildpack\n      properties:\n        entryfilePath: ./src/kafka-producer.ts\n    events:\n      - type: http-api-gateway\n        properties:\n          httpApiGatewayName: mainApiGateway\n          path: /produce/{message}\n          method: GET\n    connectTo:\n      - kafkaTopic\n```\n\n#### 1.3.2 Consumer function\n\nConsumer function is configured as follows:\n\n- **Packaging** - determines how the lambda artifact is built. The easiest and most optimized way to build the lambda\n  from Typescript/Javascript is using `stacktape-lambda-buildpack`. We only need to configure `entryfilePath`. Stacktape\n  automatically transpiles and builds the application code with all of its dependencies, creates the lambda zip\n  artifact, and uploads it to a pre-created S3 bucket on AWS. You can also use\n  [other types of packaging](https://docs.stacktape.com/configuration/packaging/#packaging-lambda-functions).\n- **Events** - Events determine how is function triggered(invoked). In this case, we are triggering the function when\n  there are messages in the upstash kafka topic. We are also configuring `batchSize`(amount of messages that can be\n  included in one invocation) and `maxBatchWindowSeconds`(amount of seconds, messages are collected, before the function\n  is invoked).\n\n```yml\nkafkaConsumer:\n  type: function\n  properties:\n    packaging:\n      type: stacktape-lambda-buildpack\n      properties:\n        entryfilePath: ./src/kafka-consumer.ts\n    events:\n      - type: kafka-topic\n        properties:\n          upstashKafkaTopic: kafkaTopic\n          batchSize: 5\n          maxBatchWindowSeconds: 10\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstacktape%2Fstarter-kafka-stream-processing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstacktape%2Fstarter-kafka-stream-processing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstacktape%2Fstarter-kafka-stream-processing/lists"}