https://github.com/pixelcaliber/log-monitor

Real-time log monitoring and broadcasting app leveraging asyncio, WebSockets, docker and load balancing through NGINX for monitoring vast size log files and broadcasting updates to multiple clients in real-time.
https://github.com/pixelcaliber/log-monitor

asyncio docker docker-compose nginx-proxy tail-f websocket

Last synced: about 6 hours ago
JSON representation

Real-time log monitoring and broadcasting app leveraging asyncio, WebSockets, docker and load balancing through NGINX for monitoring vast size log files and broadcasting updates to multiple clients in real-time.

• Host: GitHub
• URL: https://github.com/pixelcaliber/log-monitor
• Owner: pixelcaliber
• Created: 2024-07-08T16:48:53.000Z (4 months ago)
• Default Branch: master
• Last Pushed: 2024-07-08T17:13:07.000Z (4 months ago)
• Last Synced: 2024-07-08T21:17:41.499Z (4 months ago)
• Topics: asyncio, docker, docker-compose, nginx-proxy, tail-f, websocket
• Language: Python
• Homepage:
• Size: 12.7 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Real-Time Log Monitoring and Broadcasting System

## Project Overview

This application is a robust, real-time log monitoring and broadcasting system. It leverages the power of asyncio, WebSockets, docker and load balancing through NGINX to create a scalable solution for monitoring vast size log files and broadcasting updates to multiple clients in real-time.

## Key Features

1. Real-time log file monitoring

2. WebSocket-based live updates to connected clients without them to refresh the page

3. Multithreaded architecture for improved performance and handling multiple clients

4. Load balancing to handle to distribute the load among the servers achieving scalability

5. System leverages containerization and load balancing to ensure high availability and scalability

## System Architecture

#### LogMonitor

The `LogMonitor` class is designed to continuously read from a specified log file present in remote server through docker bind mount, detecting new entries as they are appended. It implements the Observer pattern, allowing multiple listeners (such as the WebSocket server) to be notified of new log entries in real-time.

Key features:

- Efficient file reading using binary mode and seek operations

- Support for retrieving the last N lines of the log file

- Thread-safe observer registration and notification

#### Socket (WebSocket Server)

The `Socket` class manages WebSocket connections with clients, providing real-time updates of log changes. It's built on top of the `websockets` library, leveraging Python's asynchronous capabilities for high-performance communication through asyncio.

Key features:

- Asynchronous client handling for improved scalability

- Periodic client pinging to maintain connection integrity and freeing up space 

#### Flask Application

The Flask application serves as the main entry point and provides a RESTful API for interacting with the system. It's designed using the application factory pattern, allowing for easy configuration and extension.

## Performance and Scalability

The system has been designed and tested for high performance and scalability, check `load_test.py`:

### Load Testing Results

Load testing has been performed to ensure the system can handle a large number of concurrent connections:

- Concurrent Clients: Successfully tested with 1000 simultaneous client connections.

- Server Instances: The test was conducted with 30 application instances.

- Performance: The system maintained responsiveness and data integrity under this high load.

#### These results demonstrate the system's capability to handle a significant number of concurrent users, making it suitable for large-scale deployments.

## Approach and Design Principles

1. **Asynchronous Programming**: Leveraging async/await syntax for efficient I/O operations and client handling.

2. **Observer Pattern**: Implemented in the LogMonitor to allow flexible and decoupled event notification.

3. **Factory Pattern**: Used in the Flask application setup for improved configurability and testing.

### Commands

```

cd app

    python -m venv venv

    source venv/bin/active

    

    python test/load_test.py 

cd ..

```

### minikube commands:

```

brew install minikube

brew install docker

brew install docker-compose 

brew install qemu

    -> once qemu installed follow this page: 

        https://minikube.sigs.k8s.io/docs/drivers/qemu/ 

    and do the essentials to prevent the errors

minikube ip

minikube status

minikube start --driver qemu --network socket_vmnet --extra-config=kubelet.authorization-mode=AlwaysAllow

minikube delete

```

### docker commands

```

docker compose up --build -d --scale app=3 (any number of servers which you want)

docker ps --format '{{.Names}}' 

docker system prune -a

if getting docker-desktop config error, run -> 

    vi ~/.docker/config.json 

then, remove desktop credentials, file should be like this:

    {

            "auths": {},

            "experimental": "disabled"

    }

    

    

docker exec -it nginx /bin/bash

docker system prune -a --volume 

-> prune the entire docker system

docker images prune

docker volumes prune

```

### How to start?

```

 minikube start --driver qemu --network socket_vmnet --extra-config=kubelet.authorization-mode=AlwaysAllow

 -> start the minikube

 

minikube mount /Users/abhinavpandey/Downloads/practice/first/shared:/minikube-host

-> mount the file path to minikube-host 

docker compose up --build -d --scale app=10

-> start the container with number of pods of your choice

minikube ip

-> get the minikube

ws:///ws (eg: "ws://192.168.105.2/ws")

-> this the route to which we can connect using postman and get the updates

```