https://github.com/amj2001/live-logs
An implementation of log processing and real time analytics
https://github.com/amj2001/live-logs
bullmq nextjs nodejs supabase
Last synced: 2 months ago
JSON representation
An implementation of log processing and real time analytics
- Host: GitHub
- URL: https://github.com/amj2001/live-logs
- Owner: AMJ2001
- Created: 2025-03-12T19:37:05.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-17T12:01:48.000Z (over 1 year ago)
- Last Synced: 2025-03-17T13:25:27.419Z (over 1 year ago)
- Topics: bullmq, nextjs, nodejs, supabase
- Language: TypeScript
- Homepage:
- Size: 1.83 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Real-Time Log Processing 🚀
An application that handles large log files, processes them for errors, tracks specific keywords, and analyzes IPs.
It comes with a dashboard that provides real-time updates on queue status and job details, specifc to each user.
---
## **🚀 Technologies Used**
| **Technology** | |
|----------------------|--|
|  | Server Logic |
|  | Frontend Framework |
|  | Job Queue Management |
|  | Containerization |
| **Supabase** | Database & Auth |
| **BullMQ** | Job Queueing |
| **Framer Motion** | UI Animations |
---
## **⚙️ Setup Instructions**
### **With Docker**
1. Clone the repository:
```bash
git clone
cd
```
2. Set up the environment variables in a .env file:
```bash
SUPABASE_URL=
SUPABASE_KEY=
NEXT_PUBLIC_WEBSOCKET_URL=
NEXT_PUBLIC_API_BASE_URL=
```
3. Build and start the project with Docker Compose:
```bash
docker-compose up --build
```
4. Navigate to http://localhost:3000
### **With Docker**
You will have to setup both frontend and backend folders separately and run the following commands:
```bash
npm install
npm run build
npm start
```
Here is a sample log file to test with: https://drive.google.com/file/d/1aj6SxL2P9ONPQXbGv7GNqhbc9xOzRxal/view?usp=sharing
## **📊 Benchmarks**
Here are some performance benchmarks for the Real-Time Log Processing system:
- **File Processing Speed**:
- Successfully processes a 10MB log file in under **75 seconds**.
- Optimized to handle large files efficiently using Node.js and BullMQ.
- **Concurrent Job Handling**:
- Processes up to **50 concurrent jobs** with a robust retry mechanism in place for fault tolerance.
- **Queue Throughput**:
- Capable of processing over **1,000+ log jobs per hour** using Redis and BullMQ.
- **Real-Time Updates**:
- Maintains a **~1-second delay** for real-time queue and job updates via WebSocket depending on the file size.
- **Scalability**:
- Scales seamlessly to handle large workloads by leveraging Docker and Redis.
---
## **🚀 Key Features**
- **Real-Time Log Monitoring**:
Analyze logs for keywords, errors, and IPs in real time, with updates directly on the dashboard.
- **Error and Keyword Tracking**:
Automatically tracks and counts errors, keywords, and IP matches from the logs.
- **BullMQ Job Queue**:
Efficient job queueing with support for retries, concurrency control, and fault tolerance.
- **Supabase Integration**:
Logs metadata, such as processing stats, are stored securely in the Supabase database.
- **WebSocket-Based Updates**:
Real-time queue and job status updates ensure a seamless user experience on the dashboard.
- **Scalable Microservices Architecture**:
Built using Docker and Docker Compose for easy deployment and scaling.
- **Authentication**:
Fully integrated user authentication using Supabase for secure access to the platform.
- **Dashboard**:
Sleek and responsive React dashboard built with Next.js, providing powerful data visualization tools.
---