https://github.com/adudko/streaming-data-loader
Tool for efficient loading and processing of streaming data from Kafka to Elasticsearch
https://github.com/adudko/streaming-data-loader
asynchronous bulk-api docker-compose-infra elasticsearch kafka-consumer streaming-data-processing
Last synced: 2 days ago
JSON representation
Tool for efficient loading and processing of streaming data from Kafka to Elasticsearch
- Host: GitHub
- URL: https://github.com/adudko/streaming-data-loader
- Owner: aDudko
- License: mit
- Created: 2025-03-22T22:13:15.000Z (about 1 year ago)
- Default Branch: master
- Last Pushed: 2025-03-23T04:43:05.000Z (about 1 year ago)
- Last Synced: 2025-03-23T05:24:17.526Z (about 1 year ago)
- Topics: asynchronous, bulk-api, docker-compose-infra, elasticsearch, kafka-consumer, streaming-data-processing
- Language: Python
- Homepage:
- Size: 2.93 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Streaming Data Loader
[](https://www.python.org)
[](https://kafka.apache.org/)
[](https://www.elastic.co/elasticsearch/)
[](https://prometheus.io/)
[](https://grafana.com/)
[](https://docs.python.org/3/library/asyncio.html)
[](https://docs.pytest.org)
[](https://www.docker.com/)
[](https://kubernetes.io/)
[](LICENSE)
---
## Overview
**Streaming Data Loader** is a high-performance, async-powered microservice for processing data streams from **Kafka**
and bulk-loading them into **Elasticsearch**. It combines modern Python tooling with observability best practices to
provide **reliable, scalable, and debuggable data pipelines**.
It’s fast, resilient, and production-ready — ideal for those who need lightweight alternatives to complex ETL systems.
---
## Key Features
- **Asynchronous processing** with `asyncio`, `aiokafka`, and `aiohttp`
- **Batch insertions** for throughput efficiency
- **Retry & fault-tolerant** logic for Kafka and Elasticsearch
- **Configurable** via `.env` and `pydantic-settings`
- **Docker & Kubernetes ready**
- **Prometheus + Grafana** monitoring included
- **Tested** with `pytest`, including integration scenarios
---
## Quick Start
### 🐳 Docker Compose
```bash
docker-compose up --build
```
- http://localhost:9090 → Prometheus
- http://localhost:3000 → Grafana (admin / admin)
- http://localhost:8080 → Kafka UI
### ☸️ Kubernetes
#### Step 1 — Deploy
```bash
./k8s-deploy.sh
```
#### Step 2 — Cleanup
```bash
./k8s-clean.sh
```
---
## Architecture
This project uses **Hexagonal Architecture** (Ports and Adapters), ensuring modularity, extensibility, and clean separation of concerns.
```text
Kafka -→ KafkaConsumerService -→ EventService -→ ElasticsearchClientService -→ Elasticsearch
│ ↓
└-→ Metrics + Logging (Prometheus + JSON logs)
```
### Layers
- **Input Ports**: Kafka Consumer (aiokafka), deserialization, batching
- **Application Core**: Event transformation, validation, retry logic
- **Output Ports**: Async Elasticsearch client, bulk insert, failure handling
- **Infrastructure**: Docker, Kubernetes, logging, metrics, monitoring
---
## 🔍 Why Choose This Over Logstash, Flume, etc.?
- ✅ **True async** data pipeline — lower latency, better throughput
- ✅ **No heavyweight config DSL** — Python code, `pyproject.toml`, `.env`
- ✅ **Built-in retries & fault handling** — robust out of the box
- ✅ **JSON logging** and metric labeling for full observability
- ✅ **Open-source & customizable** — perfect for modern data teams
---
## Observability
Prometheus scrapes metrics on `/metrics` (port `8000`). Dashboards are automatically provisioned in Grafana.
| Metric | Description |
|-------------------------------------|------------------------------------|
| `messages_processed_total` | Total number of processed messages |
| `errors_total` | Total errors during processing |
| `consume_duration_seconds` | Time spent reading from Kafka |
| `response_duration_seconds` | Time to insert into Elasticsearch |
| `transform_duration_seconds` | Time spent transforming messages |
| `batch_processing_duration_seconds` | Full batch processing time |
---
## Testing
```bash
pytest -v
```
Includes:
- ✅ Unit tests
- ✅ Integration tests (Kafka → ES)
- ✅ Metrics verification
- ✅ Config validation
---
## Technologies
- `Python 3.12` + `asyncio`
- `Kafka + aiokafka`
- `Elasticsearch` `Bulk API`
- `Pydantic` `dotenv` `poetry`
- `Prometheus` `Grafana`
- `Docker` `docker-compose`
- `Kubernetes-ready`
- JSON logging (`python-json-logger`)
---
## Project Structure
```text
streaming-data-loader/
├── configs/ # Prometheus / Grafana
├── src/ # Main source code
│ ├── domain/
│ ├── ports/
│ ├── services/ # Event processing logic
│ ├── config.py # Settings & env config
│ ├── logger.py # JSON logger setup
│ └── metrics.py # Prometheus metrics
├── tests/ # Unit & integration tests
├── k8s/ # Kubernetes manifests
├── docker-compose.yml
├── Dockerfile
├── deploy.sh / clean.sh
└── pyproject.toml
```
---
## If you find this useful...
...give it a star, fork it, or mention it in your next data project!
## Author
**Anatoly Dudko**
[GitHub @aDudko](https://github.com/aDudko) • [LinkedIn](https://www.linkedin.com/in/dudko-anatol/)