{"id":26410435,"url":"https://github.com/amj2001/live-logs","last_synced_at":"2026-04-11T21:47:01.196Z","repository":{"id":282879939,"uuid":"947500994","full_name":"AMJ2001/Live-Logs","owner":"AMJ2001","description":"An implementation of log processing and real time analytics","archived":false,"fork":false,"pushed_at":"2025-03-17T12:01:48.000Z","size":1924,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-17T13:25:27.419Z","etag":null,"topics":["bullmq","nextjs","nodejs","supabase"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AMJ2001.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-12T19:37:05.000Z","updated_at":"2025-03-17T12:01:51.000Z","dependencies_parsed_at":"2025-03-17T13:25:30.139Z","dependency_job_id":"9c58dfba-3ec7-415b-976c-9ba1f20cea92","html_url":"https://github.com/AMJ2001/Live-Logs","commit_stats":null,"previous_names":["amj2001/live-logs"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AMJ2001%2FLive-Logs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AMJ2001%2FLive-Logs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AMJ2001%2FLive-Logs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AMJ2001%2FLive-Logs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AMJ2001","download_url":"https://codeload.github.com/AMJ2001/Live-Logs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244102849,"owners_count":20398386,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bullmq","nextjs","nodejs","supabase"],"created_at":"2025-03-17T20:17:50.584Z","updated_at":"2026-04-11T21:47:01.110Z","avatar_url":"https://github.com/AMJ2001.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Real-Time Log Processing 🚀\n\nAn application that handles large log files, processes them for errors, tracks specific keywords, and analyzes IPs.\u003cbr\u003e It comes with a dashboard that provides real-time updates on queue status and job details, specifc to each user.\n\n---\n\n## **🚀 Technologies Used**\n\n| **Technology**      |  |\n|----------------------|--|\n| ![Node.js](https://img.icons8.com/color/48/000000/nodejs.png)       | Server Logic |\n| ![Next.js](https://img.icons8.com/color/48/000000/nextjs.png)      | Frontend Framework |\n| ![Redis](https://img.icons8.com/color/48/000000/redis.png)          | Job Queue Management |\n| ![Docker](https://img.icons8.com/fluency/48/000000/docker.png)      | Containerization |\n| **Supabase** | Database \u0026 Auth |\n| **BullMQ** | Job Queueing |\n| **Framer Motion** | UI Animations |\n\n---\n\n## **⚙️ Setup Instructions**\n\n### **With Docker**\n1. Clone the repository:\n   ```bash\n     git clone \u003crepository-url\u003e\n     cd \u003crepository-folder\u003e\n   ```\n2. Set up the environment variables in a .env file:\n   ```bash\n    SUPABASE_URL=\u003cyour_supabase_url\u003e\n    SUPABASE_KEY=\u003cyour_supabase_key\u003e\n    NEXT_PUBLIC_WEBSOCKET_URL=\u003cyour_websocket_url\u003e\n    NEXT_PUBLIC_API_BASE_URL=\u003cyour_api_base_url\u003e\n   ```\n3. Build and start the project with Docker Compose:\n   ```bash\n   docker-compose up --build\n   ```\n4. Navigate to \u003cu\u003ehttp://localhost:3000\u003cu\u003e\n\n### **With Docker**\nYou will have to setup both frontend and backend folders separately and run the following commands:\n ```bash\n  npm install\n  npm run build\n  npm start\n```\n\nHere is a sample log file to test with: https://drive.google.com/file/d/1aj6SxL2P9ONPQXbGv7GNqhbc9xOzRxal/view?usp=sharing\n\u003cbr\u003e\n## **📊 Benchmarks**\n\nHere are some performance benchmarks for the Real-Time Log Processing system:\n\n- **File Processing Speed**:  \n  - Successfully processes a 10MB log file in under **75 seconds**.\n  - Optimized to handle large files efficiently using Node.js and BullMQ.\n\n- **Concurrent Job Handling**:  \n  - Processes up to **50 concurrent jobs** with a robust retry mechanism in place for fault tolerance.\n\n- **Queue Throughput**:  \n  - Capable of processing over **1,000+ log jobs per hour** using Redis and BullMQ.\n\n- **Real-Time Updates**:  \n  - Maintains a **~1-second delay** for real-time queue and job updates via WebSocket depending on the file size.\n\n- **Scalability**:  \n  - Scales seamlessly to handle large workloads by leveraging Docker and Redis.\n\n---\n\n## **🚀 Key Features**\n\n- **Real-Time Log Monitoring**:  \n  Analyze logs for keywords, errors, and IPs in real time, with updates directly on the dashboard.\n\n- **Error and Keyword Tracking**:  \n  Automatically tracks and counts errors, keywords, and IP matches from the logs.\n\n- **BullMQ Job Queue**:  \n  Efficient job queueing with support for retries, concurrency control, and fault tolerance.\n\n- **Supabase Integration**:  \n  Logs metadata, such as processing stats, are stored securely in the Supabase database.\n\n- **WebSocket-Based Updates**:  \n  Real-time queue and job status updates ensure a seamless user experience on the dashboard.\n\n- **Scalable Microservices Architecture**:  \n  Built using Docker and Docker Compose for easy deployment and scaling.\n\n- **Authentication**:  \n  Fully integrated user authentication using Supabase for secure access to the platform.\n\n- **Dashboard**:  \n  Sleek and responsive React dashboard built with Next.js, providing powerful data visualization tools.\n\n---\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famj2001%2Flive-logs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famj2001%2Flive-logs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famj2001%2Flive-logs/lists"}