https://github.com/jorgermduarte/poc-s3-kinesis-opensearch
https://github.com/jorgermduarte/poc-s3-kinesis-opensearch
aws consumer docker kinesis localstack opensearch s3 s3-bucket streaming ui
Last synced: 6 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/jorgermduarte/poc-s3-kinesis-opensearch
- Owner: jorgermduarte
- License: mit
- Created: 2025-04-01T13:21:33.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2025-04-09T09:31:22.000Z (7 months ago)
- Last Synced: 2025-04-12T20:54:03.666Z (6 months ago)
- Topics: aws, consumer, docker, kinesis, localstack, opensearch, s3, s3-bucket, streaming, ui
- Language: JavaScript
- Homepage:
- Size: 231 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
- License: LICENSE
Awesome Lists containing this project
README
# S3 File Upload with LocalStack and Kinesis Integration for OpenSearch
A local development environment demonstrating file uploads to S3 with mock trigger to Kinesis for event streaming using LocalStack.
## Features
- 📁 File upload to LocalStack S3 bucket
- 📨 Real-time event streaming with Kinesis
- 🐳 Dockerized LocalStack setup
- 📊 Web interface for file management
- 🔍 Kinesis record inspection
- 📦 Pre-configured AWS resources initialization
- 🧪 Sample JSON data handling
## Prerequisites
- Docker & Docker Compose
- Node.js 16+
- AWS CLI (optional)
## UI
# OpenSearch Integration
The Kinesis consumer service:
1. Continuously polls the Kinesis stream
2. Processes new file upload events
3. Retrieves files from S3
4. Indexes file content in OpenSearch
**Access OpenSearch:**
- Search API for files index: http://localhost:9200/files/_search
- Search API for products index: http://localhost:9200/products/_search
# Getting Started
## Executing the necessary services
```
docker compose up localstack --build -d
docker compose up localstack-init --build -d
docker compose up opensearch --build -d
```
## Executing the Kinesis Consumer
```
npm install
npm start
```
## Executing the s3-upload-app
```
npm install
npm start
```
# S3 File submission
Before the submission and having all the services executed we can see that we don't have any data for the products index.
After the .json file submission to S3 and given the mock trigger to kinesis, we can verify that the product is added correctly.
