https://github.com/dheerajcl/distributed_logging_system
https://github.com/dheerajcl/distributed_logging_system
apache-kafka distributed-computing elasticsearch elasticsearch-plugin elasticsearch-queries fluentd fluentd-logger fluentd-plugin kafka-plugins kafka-python kibana-visualization logging pub-sub-system
Last synced: 10 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/dheerajcl/distributed_logging_system
- Owner: dheerajcl
- License: apache-2.0
- Created: 2024-10-29T14:09:01.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-22T07:28:26.000Z (about 1 year ago)
- Last Synced: 2025-03-25T09:12:26.562Z (11 months ago)
- Topics: apache-kafka, distributed-computing, elasticsearch, elasticsearch-plugin, elasticsearch-queries, fluentd, fluentd-logger, fluentd-plugin, kafka-plugins, kafka-python, kibana-visualization, logging, pub-sub-system
- Language: Python
- Homepage:
- Size: 1.89 MB
- Stars: 5
- Watchers: 1
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Distributed Logging System(BIG DATA)
## Project Overview
Effective log management is crucial for operational excellence in a microservices architecture. This project aims to streamline the collection, storage, and analysis of logs generated by various services, enhancing the ability to quickly track application behavior and identify errors. By capturing relevant metadata alongside each log entry and enabling real-time ingestion and querying, the system improves operational visibility and facilitates proactive responses to potential issues. Ultimately, this distributed logging framework enhances resilience and maintainability in a dynamic application landscape.
## System Architecture & Flow
### Components
1. **Microservices(Process):** Represent distributed nodes that independently generate logs and send heartbeat signals to monitor their status.
2. **Log Accumulator:** Collects log data from each node, structures it, and forwards it to the Pub-Sub model for centralized log management.
3. **Pub-Sub Model:** Acts as a communication layer, facilitating reliable, asynchronous distribution of logs.
4. **Log Storage:** A system for indexing and storing logs in a searchable format for easy access and monitoring.
5. **Alerting System:** Listens for specific log levels (e.g., ERROR, FATAL, etc) in real-time, generating alerts to ensure prompt responses to critical events.
6. **Heartbeat Mechanism:** Provides failure detection by alerting when a node stops sending heartbeats, signaling that the node may have failed.
---
## Tools & Technologies
- **Programming Language:** Python
- **Log Accumulator:** Fluentd
- **Pub-Sub Model:** Apache Kafka
- **Log Storage:** Elasticsearch
- **Visualization:** Kibana
---
## Log Levels
- **INFO:** General system operations information.
- **WARN:** Indications of potential issues.
- **ERROR:** Errors that allow continued operation.
- **FATAL:** Critical issues requiring immediate attention.
---
## Metadata for Logs
**Example Structures:**
- **Microservice Registration Message:**
```json
{
"node_id": "",
"message_type": "REGISTRATION",
"service_name": "PaymentService",
"timestamp": ""
}
```
- **INFO Log:**
```json
{
"log_id": "",
"node_id": "",
"log_level": "INFO",
"message_type": "LOG",
"message": "",
"service_name": "",
"timestamp": ""
}
```
- **WARN Log:**
```json
{
"log_id": "",
"node_id": "",
"log_level": "WARN",
"message_type": "LOG",
"message": "",
"service_name": "",
"response_time_ms": "",
"threshold_limit_ms": "",
"timestamp": ""
}
```
- **ERROR Log:**
```json
{
"log_id": "",
"node_id": "",
"log_level": "ERROR",
"message_type": "LOG",
"message": "",
"service_name": "",
"error_details": {
"error_code": "",
"error_message": ""
},
"timestamp": ""
}
```
- **Heartbeat Message:**
```json
{
"node_id": "",
"message_type": "HEARTBEAT",
"status": "UP/DOWN",
"timestamp": ""
}
```
- **Microservice Registry:**
```json
{
"message_type":"REGISTRATION",
"node_id": "",
"service_name": "",
"status": "UP/DOWN",
"timestamp": ""
}
```
## References
- [Distributed Logging System Design on Medium]()
- [Distributed Logging Architecture on DZone]()
- [Remote Kafka Connection Guide by Bitnami]()
- [Fluentd Output to Kafka Documentation]()
- [Python Client for Elasticsearch - Getting Started Guide]()