https://github.com/daniel-sogbey/llm_log_pipeline
Containerized Go service that uses LLMs (e.g., LLaMA 3.1 Instruct) to analyze backend logs via RabbitMQ and store structured insights in PostgreSQL
https://github.com/daniel-sogbey/llm_log_pipeline
docker docker-compose go golang llm llms logs microservices postgresql rabbitmq togetherai
Last synced: 3 months ago
JSON representation
Containerized Go service that uses LLMs (e.g., LLaMA 3.1 Instruct) to analyze backend logs via RabbitMQ and store structured insights in PostgreSQL
- Host: GitHub
- URL: https://github.com/daniel-sogbey/llm_log_pipeline
- Owner: Daniel-Sogbey
- Created: 2025-06-14T16:22:29.000Z (about 1 year ago)
- Default Branch: master
- Last Pushed: 2025-06-15T01:48:34.000Z (about 1 year ago)
- Last Synced: 2025-06-15T02:35:59.377Z (about 1 year ago)
- Topics: docker, docker-compose, go, golang, llm, llms, logs, microservices, postgresql, rabbitmq, togetherai
- Language: Go
- Homepage:
- Size: 58.6 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# LLM Log Pipeline
**LLM Log Pipeline** is a containerized Go-based service that uses an LLM (e.g., Meta LLaMA 3.1 Instruct, 8B) to automatically analyze and summarize backend logs. It ingests logs through RabbitMQ, processes them with a large language model, and stores insights in a PostgreSQL database for future reference or visualization.
---
## Features
* Parses raw application logs with stack traces
* Uses LLM to extract:
* Root cause
* Severity
* Time of occurrence
* Stack trace analysis
* File and line number
* Suggested fix
* Summary
* Stores enriched logs in PostgreSQL
* Runs as a Dockerized microservice
* Easily extendable (e.g., dashboard, alerting)
---
## Architecture
---
## Getting Started
### 1. Clone the Repo
```bash
git clone https://github.com/Daniel-Sogbey/llm_log_pipeline.git
cd llm_log_pipeline
```
### 2. Environment Setup
Add the `.env` file:
Update it with your own values:
```env
AMQP_CONNECTION_URL=amqp://guest:guest@host.docker.internal:5672/
LLM_ENDPOINT=https://api.together.xyz/v1/chat/completions
LLM_AUTHORIZATION_KEY=your_llm_api_key
DB_DSN=postgres://admin:secret@host.docker.internal:5432/mydb?sslmode=disable
LLM_MODEL=meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo
EXCHANGE=logs
EXCHANGE_KIND=fanout
QUEUE=logs_queue
ROUTING_KEY=
```
### 3. Pull and Run from DockerHub
Pull and run the service:
```bash
docker pull sogbey/llm_log_pipeline:latest
docker run --env-file .env sogbey/llm_log_pipeline
```
### 4. Build and Run with Docker Locally
Build and run the service:
```bash
docker build -t llm_log_pipeline .
docker run --env-file .env llm_log_pipeline
```
### 5. (Optional) Docker Compose
Use the provided Docker Compose setup to launch RabbitMQ, PostgreSQL, and the pipeline:
```bash
docker compose up --build
#.env file should contain the following
AMQP_CONNECTION_URL=amqp://guest:guest@rabbitmq
LLM_ENDPOINT=https://api.together.xyz/v1/chat/completions
LLM_AUTHORIZATION_KEY=your_llm_api_key
DB_DSN=postgres://admin:secret@postgres/mydb?sslmode=disable
LLM_MODEL=meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo
EXCHANGE=logs
EXCHANGE_KIND=fanout
QUEUE=logs_queue
ROUTING_KEY=
```
* RabbitMQ Management UI: [http://localhost:15672](http://localhost:15672)
* Default credentials: `guest / guest`
---
## How It Works
1. Producer sends raw log entries into RabbitMQ.
2. Exchange fans out the message to queues.
3. Consumer (written in Go) listens to the queue.
4. Consumer calls the LLM API to analyze logs.
5. Stores structured output in PostgreSQL.
### Example Output from LLM:
```json
{
"analysis": "The log indicates an error occurred when trying to find a record, resulting in a server error.",
"cause": "The most likely root cause of the error is a missing or incorrect record in the database.",
"severity": "high",
"time_of_occurrence": "2025-04-06T17:32:01Z",
"stacktrace_insight": "The error occurred in the paymentWebHookHandler function in the checkout package.",
"file": "cmd/api/checkout.go",
"line_number": "237",
"summary": "Record not found error",
"comprehensive_detail": "This error affects the system by preventing the payment webhook handler from functioning correctly, resulting in failed payment processing. From a user's perspective, this may cause payment failures or errors when attempting to process payments. The system will need to handle this error and provide a suitable response to the user.",
"suggested_way_to_fix": "Check the database for missing or incorrect records, and ensure that the payment webhook handler is correctly configured to handle record not found errors."
}
```
---
## Tech Stack
* Go
* RabbitMQ
* PostgreSQL
* Together.ai (or any LLM with OpenAI-compatible API)
* Docker
---
## CI/CD Pipeline (Coming Soon)
Planned CI/CD capabilities:
* [x] GitHub Actions workflow for automated testing and builds
* [x] Docker image publishing to Docker Hub
Once implemented, this will allow:
* Automated testing on push/PRs
* Docker image builds triggered by changes
* Docker Hub deployments from GitHub
---
## Extending This Project
You can add:
* Web Dashboard UI (Next.js, Vue, etc.)
* Alert system for high-severity logs (Slack, Email)
* LLM provider support: OpenAI, OpenRouter, Groq, etc.
---
## Contributing
1. Fork the repo
2. Create your feature branch:
```bash
git checkout -b feature/foo
```
3. Commit changes:
```bash
git commit -am 'Add foo'
```
4. Push to branch:
```bash
git push origin feature/foo
```
5. Submit a pull request
---
## Credits
* LLM API: Together.ai
* Diagram made with draw\.io