{"id":28816764,"url":"https://github.com/daniel-sogbey/llm_log_pipeline","last_synced_at":"2026-05-08T05:05:56.394Z","repository":{"id":299157314,"uuid":"1002060964","full_name":"Daniel-Sogbey/llm_log_pipeline","owner":"Daniel-Sogbey","description":"Containerized Go service that uses LLMs (e.g., LLaMA 3.1 Instruct) to analyze backend logs via RabbitMQ and store structured insights in PostgreSQL","archived":false,"fork":false,"pushed_at":"2025-06-15T01:48:34.000Z","size":60,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-06-15T02:35:59.377Z","etag":null,"topics":["docker","docker-compose","go","golang","llm","llms","logs","microservices","postgresql","rabbitmq","togetherai"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Daniel-Sogbey.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-14T16:22:29.000Z","updated_at":"2025-06-15T02:28:17.000Z","dependencies_parsed_at":"2025-06-15T02:36:02.676Z","dependency_job_id":"5f7a833a-b7d6-4178-b24b-4be4c02da720","html_url":"https://github.com/Daniel-Sogbey/llm_log_pipeline","commit_stats":null,"previous_names":["daniel-sogbey/llm_log_pipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Daniel-Sogbey/llm_log_pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Sogbey%2Fllm_log_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Sogbey%2Fllm_log_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Sogbey%2Fllm_log_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Sogbey%2Fllm_log_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Daniel-Sogbey","download_url":"https://codeload.github.com/Daniel-Sogbey/llm_log_pipeline/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Sogbey%2Fllm_log_pipeline/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260595742,"owners_count":23033789,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","go","golang","llm","llms","logs","microservices","postgresql","rabbitmq","togetherai"],"created_at":"2025-06-18T17:06:49.312Z","updated_at":"2026-05-08T05:05:56.364Z","avatar_url":"https://github.com/Daniel-Sogbey.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM Log Pipeline\n\n**LLM Log Pipeline** is a containerized Go-based service that uses an LLM (e.g., Meta LLaMA 3.1 Instruct, 8B) to automatically analyze and summarize backend logs. It ingests logs through RabbitMQ, processes them with a large language model, and stores insights in a PostgreSQL database for future reference or visualization.\n\n---\n\n##  Features\n\n* Parses raw application logs with stack traces\n* Uses LLM to extract:\n\n    * Root cause\n    * Severity\n    * Time of occurrence\n    * Stack trace analysis\n    * File and line number\n    * Suggested fix\n    * Summary\n* Stores enriched logs in PostgreSQL\n* Runs as a Dockerized microservice\n* Easily extendable (e.g., dashboard, alerting)\n\n---\n\n##  Architecture\n\n![Architecture](./docs/llm_log_pipeline_architecture_diagram.png)\n\n---\n\n##  Getting Started\n\n### 1. Clone the Repo\n\n```bash\ngit clone https://github.com/Daniel-Sogbey/llm_log_pipeline.git\ncd llm_log_pipeline\n```\n\n### 2. Environment Setup\n\nAdd the `.env` file:\n\nUpdate it with your own values:\n\n```env\nAMQP_CONNECTION_URL=amqp://guest:guest@host.docker.internal:5672/\nLLM_ENDPOINT=https://api.together.xyz/v1/chat/completions\nLLM_AUTHORIZATION_KEY=your_llm_api_key\nDB_DSN=postgres://admin:secret@host.docker.internal:5432/mydb?sslmode=disable\nLLM_MODEL=meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo\nEXCHANGE=logs\nEXCHANGE_KIND=fanout\nQUEUE=logs_queue\nROUTING_KEY=\n```\n\n### 3.  Pull and Run from DockerHub\n\nPull and run the service:\n\n```bash\ndocker pull sogbey/llm_log_pipeline:latest\ndocker run --env-file .env sogbey/llm_log_pipeline\n```\n\n\n### 4.  Build and Run with Docker Locally\n\nBuild and run the service:\n\n```bash\ndocker build -t llm_log_pipeline .\ndocker run --env-file .env llm_log_pipeline\n```\n\n### 5. (Optional) Docker Compose\n\nUse the provided Docker Compose setup to launch RabbitMQ, PostgreSQL, and the pipeline:\n\n```bash\ndocker compose up --build\n\n#.env file should contain the following\n\nAMQP_CONNECTION_URL=amqp://guest:guest@rabbitmq\nLLM_ENDPOINT=https://api.together.xyz/v1/chat/completions\nLLM_AUTHORIZATION_KEY=your_llm_api_key\nDB_DSN=postgres://admin:secret@postgres/mydb?sslmode=disable\nLLM_MODEL=meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo\nEXCHANGE=logs\nEXCHANGE_KIND=fanout\nQUEUE=logs_queue\nROUTING_KEY=\n```\n\n* RabbitMQ Management UI: [http://localhost:15672](http://localhost:15672)\n* Default credentials: `guest / guest`\n\n---\n\n## How It Works\n\n1. Producer sends raw log entries into RabbitMQ.\n2. Exchange fans out the message to queues.\n3. Consumer (written in Go) listens to the queue.\n4. Consumer calls the LLM API to analyze logs.\n5. Stores structured output in PostgreSQL.\n\n\n### Example Output from LLM:\n\n```json\n{\n  \"analysis\": \"The log indicates an error occurred when trying to find a record, resulting in a server error.\",\n  \"cause\": \"The most likely root cause of the error is a missing or incorrect record in the database.\",\n  \"severity\": \"high\",\n  \"time_of_occurrence\": \"2025-04-06T17:32:01Z\",\n  \"stacktrace_insight\": \"The error occurred in the paymentWebHookHandler function in the checkout package.\",\n  \"file\": \"cmd/api/checkout.go\",\n  \"line_number\": \"237\",\n  \"summary\": \"Record not found error\",\n  \"comprehensive_detail\": \"This error affects the system by preventing the payment webhook handler from functioning correctly, resulting in failed payment processing. From a user's perspective, this may cause payment failures or errors when attempting to process payments. The system will need to handle this error and provide a suitable response to the user.\",\n  \"suggested_way_to_fix\": \"Check the database for missing or incorrect records, and ensure that the payment webhook handler is correctly configured to handle record not found errors.\"\n}\n```\n\n\n---\n\n## Tech Stack\n\n* Go\n* RabbitMQ\n* PostgreSQL\n* Together.ai (or any LLM with OpenAI-compatible API)\n* Docker\n\n---\n\n## CI/CD Pipeline (Coming Soon)\n\nPlanned CI/CD capabilities:\n\n* [x] GitHub Actions workflow for automated testing and builds\n* [x] Docker image publishing to Docker Hub\n\nOnce implemented, this will allow:\n\n* Automated testing on push/PRs\n* Docker image builds triggered by changes\n* Docker Hub deployments from GitHub\n\n---\n\n##  Extending This Project\n\nYou can add:\n\n*  Web Dashboard UI (Next.js, Vue, etc.)\n* Alert system for high-severity logs (Slack, Email)\n* LLM provider support: OpenAI, OpenRouter, Groq, etc.\n\n---\n\n## Contributing\n\n1. Fork the repo\n\n2. Create your feature branch:\n\n```bash\ngit checkout -b feature/foo\n```\n\n3. Commit changes:\n\n```bash\ngit commit -am 'Add foo'\n```\n\n4. Push to branch:\n\n```bash\ngit push origin feature/foo\n```\n\n5. Submit a pull request\n\n---\n\n##  Credits\n\n* LLM API: Together.ai\n* Diagram made with draw\\.io\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaniel-sogbey%2Fllm_log_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdaniel-sogbey%2Fllm_log_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaniel-sogbey%2Fllm_log_pipeline/lists"}