{"id":29554733,"url":"https://github.com/rohittcodes/flashio","last_synced_at":"2025-07-18T07:33:13.971Z","repository":{"id":299025215,"uuid":"1001826715","full_name":"rohittcodes/flashio","owner":"rohittcodes","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-15T10:44:53.000Z","size":1667,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-04T21:07:13.841Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rohittcodes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-14T05:36:54.000Z","updated_at":"2025-06-15T10:44:56.000Z","dependencies_parsed_at":"2025-06-14T08:19:41.339Z","dependency_job_id":"4a386430-5687-435a-9497-638cb80aa68d","html_url":"https://github.com/rohittcodes/flashio","commit_stats":null,"previous_names":["rohittcodes/flashio"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rohittcodes/flashio","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohittcodes%2Fflashio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohittcodes%2Fflashio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohittcodes%2Fflashio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohittcodes%2Fflashio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rohittcodes","download_url":"https://codeload.github.com/rohittcodes/flashio/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohittcodes%2Fflashio/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265720604,"owners_count":23817269,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-18T07:31:43.502Z","updated_at":"2025-07-18T07:33:13.954Z","avatar_url":"https://github.com/rohittcodes.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ⚡ Flash.io\n### Intelligent Real-Time Log Analysis with AI\n\n\u003cdiv align=\"center\"\u003e\n  \n[![FastAPI](https://img.shields.io/badge/FastAPI-0.115.9-009688?style=for-the-badge\u0026logo=fastapi)](https://fastapi.tiangolo.com/)\n[![Python](https://img.shields.io/badge/Python-3.13+-3776AB?style=for-the-badge\u0026logo=python)](https://python.org)\n[![ChromaDB](https://img.shields.io/badge/ChromaDB-Vector_DB-FF6B6B?style=for-the-badge)](https://www.trychroma.com/)\n[![AI](https://img.shields.io/badge/Multi_LLM-AI_Powered-FFD43B?style=for-the-badge\u0026logo=openai)](https://openai.com)\n\n*Transform your log chaos into actionable insights with the power of AI*\n\n[🚀 Quick Start](#-quick-start) • [✨ Features](#-features) • [📖 Documentation](#-documentation) • [🎯 Use Cases](#-use-cases)\n\n\u003c/div\u003e\n\n---\n\n## 🌟 What is Flash.io?\n\n**Flash.io** is a cutting-edge, AI-powered log analysis platform that revolutionizes how DevOps teams, system administrators, and developers monitor, analyze, and troubleshoot their applications and infrastructure. By combining advanced vector embeddings, multiple Large Language Models (LLMs), and real-time streaming capabilities, Flash.io transforms raw log data into intelligent, actionable insights.\n\n### 🎯 The Problem We Solve\n\n- **Log Overload**: Modern systems generate millions of log entries daily\n- **Manual Analysis**: Traditional log analysis is time-consuming and error-prone\n- **Pattern Recognition**: Identifying critical issues buried in noise\n- **Real-time Monitoring**: Need for instant alerts and insights\n- **Context Loss**: Difficulty correlating events across different services\n\n### Demo Video\nhttps://github.com/user-attachments/assets/dfe8d475-a244-48e9-aabe-198100a36a01\n\n### 💡 Our Solution\n\nFlash.io leverages state-of-the-art AI to provide:\n- **Natural Language Queries**: Ask questions in plain English\n- **Intelligent Pattern Detection**: AI identifies anomalies and correlations\n- **Real-time Analysis**: Live log streaming with instant insights\n- **Multi-Provider LLM Support**: Choose from Google Gemini, OpenAI GPT, Anthropic Claude, or Groq\n- **Vector-Based Search**: Semantic understanding of log content\n- **Historical Context**: Maintains conversation history and analysis state\n\n---\n\n## ✨ Features\n\n### 🤖 **AI-Powered Analysis**\n- **Multi-LLM Support**: Google Gemini, OpenAI GPT-4, Anthropic Claude, Groq Llama\n- **Natural Language Queries**: \"Show me database errors from the last hour\"\n- **Intelligent Summarization**: AI-generated insights and recommendations\n- **Context-Aware Analysis**: Maintains conversation history and system state\n\n### 🔍 **Advanced Search \u0026 Discovery**\n- **Vector Embeddings**: Semantic search using Google's embedding models\n- **Smart Filtering**: Time-based, service-based, and severity-based filters\n- **Pattern Recognition**: Automatically detect recurring issues and anomalies\n- **Correlation Analysis**: Link related events across different services\n\n### 📊 **Real-Time Monitoring**\n- **Live Log Streaming**: WebSocket-based real-time log ingestion\n- **Instant Alerts**: Immediate notification of critical issues\n- **Agent State Management**: Persistent analysis state across sessions\n- **Automated Cleanup**: Configurable log retention policies\n\n### 🏗️ **Flexible Architecture**\n- **Multiple Data Sources**: Support for various log formats and sources\n- **RESTful API**: Complete programmatic access to all features\n- **Extensible Design**: Plugin architecture for custom log sources\n- **Scalable Storage**: ChromaDB vector database for efficient storage\n\n### 🎨 **Modern User Interface**\n- **Responsive Design**: Beautiful, mobile-first web interface\n- **Real-Time Updates**: Live data streaming with WebSocket connections\n- **Interactive Dashboards**: Intuitive navigation and data visualization\n- **Progressive Enhancement**: Works across all modern browsers\n\n---\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- **Python 3.13+**\n- **API Keys** for at least one LLM provider:\n  - Google AI (Gemini) - Recommended\n  - OpenAI (GPT-4)\n  - Anthropic (Claude)\n  - Groq (Llama)\n\n### 🛠️ Installation\n\n1. **Clone the Repository**\n   ```bash\n   git clone https://github.com/yourusername/flashio.git\n   cd flashio\n   ```\n\n2. **Install Dependencies**\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n3. **Generate Security Keys**\n   ```bash\n   python generate_keys.py\n   ```\n\n4. **Configure Environment**\n   ```bash\n   cp .env_sample .env\n   # Edit .env with your API keys and configuration\n   ```\n\n5. **Set Up Database** (Optional)\n   ```bash\n   python setup_database.py\n   ```\n\n6. **Start the Application**\n   ```bash\n   python main.py\n   ```\n\n7. **Access Flash.io**\n   ```\n   🌐 Web Interface: http://localhost:8000\n   📚 API Documentation: http://localhost:8000/docs\n   ```\n\n---\n\n## ⚙️ Configuration\n\n### Environment Variables\n\nCreate a `.env` file in the root directory:\n\n```env\n# Database Configuration\nCHROMA_DB_PATH=./chroma_db\nLOG_RETENTION_DAYS=30\nMAX_BATCH_SIZE=100\nPROCESSING_INTERVAL=5\n\n# AI Provider Configuration\nDEFAULT_PROVIDER=google\nDEFAULT_MODEL=gemini-1.5-pro\n\n# API Keys (at least one required)\nGOOGLE_API_KEY=your_google_api_key_here\nOPENAI_API_KEY=your_openai_api_key_here\nGROQ_API_KEY=your_groq_api_key_here\nANTHROPIC_API_KEY=your_anthropic_api_key_here\n\n# Security Configuration\nENCRYPTION_KEY=your_generated_encryption_key\nAPI_KEY_HASH=your_generated_api_key_hash\n```\n\n### Supported LLM Providers\n\n| Provider | Models | Use Case |\n|----------|--------|----------|\n| **Google** | `gemini-1.5-pro`, `gemini-1.5-flash` | Recommended for general use |\n| **OpenAI** | `gpt-4`, `gpt-3.5-turbo` | Excellent reasoning capabilities |\n| **Anthropic** | `claude-3-opus`, `claude-3-sonnet` | Superior safety and analysis |\n| **Groq** | `llama3-70b-instruct`, `llama3-8b-8192` | High-speed inference |\n\n---\n\n## 🎯 Use Cases\n\n### 🔧 **DevOps \u0026 SRE**\n- **Incident Response**: Quickly identify root causes during outages\n- **Performance Monitoring**: Detect degradation patterns before they impact users\n- **Capacity Planning**: Analyze usage trends and resource consumption\n- **Automated Reporting**: Generate intelligent summaries for stakeholders\n\n### 🏢 **Enterprise Operations**\n- **Compliance Monitoring**: Track security events and access patterns\n- **Business Intelligence**: Extract insights from application logs\n- **Cost Optimization**: Identify resource waste and optimization opportunities\n- **Predictive Maintenance**: Anticipate failures before they occur\n\n### 👨‍💻 **Development Teams**\n- **Bug Investigation**: Trace issues across microservices\n- **Performance Optimization**: Identify bottlenecks and slow queries\n- **Feature Monitoring**: Track new feature adoption and issues\n- **Code Quality**: Monitor error rates and exception patterns\n\n---\n\n## 📖 Documentation\n\n### 🌐 **Web Interface Guide**\n\n#### Navigation Tabs\n\n- **📥 Ingest**: Upload and import log files\n- **🔍 Query**: Natural language log analysis\n- **🤖 Agent**: AI agent state and conversation history\n- **🗃️ Database**: Log storage management and statistics\n- **📡 Live**: Real-time log streaming and monitoring\n\n#### Using Natural Language Queries\n\n```\nExamples:\n• \"Show me all database errors from the last 2 hours\"\n• \"Find memory usage spikes in the payment service\"\n• \"What caused the 500 errors this morning?\"\n• \"Compare performance between today and yesterday\"\n• \"Show me unusual patterns in the authentication logs\"\n```\n\n### 🔌 **API Reference**\n\n#### Core Endpoints\n\n- **`POST /queries/natural`** - Submit natural language queries\n- **`GET /queries/insights`** - Retrieve system insights and agent state\n- **`POST /queries/configure-model`** - Switch between LLM providers\n- **`POST /ingest/logs`** - Ingest log data\n- **`GET /logs/search`** - Search logs with filters\n- **`WebSocket /ws`** - Real-time log streaming\n\n#### Example API Usage\n\n```python\nimport requests\n\n# Natural language query\nresponse = requests.post(\"http://localhost:8000/queries/natural\", json={\n    \"query\": \"Show me database connection errors from the last hour\",\n    \"max_logs\": 50,\n    \"include_context\": True\n})\n\n# Process results\nanalysis = response.json()\nprint(f\"Analysis: {analysis['analysis']}\")\nfor log in analysis['logs']:\n    print(f\"[{log['timestamp']}] {log['message']}\")\n```\n\n---\n\n## 🏗️ Architecture\n\n### System Components\n\n```mermaid\ngraph TB\n    A[Web UI] --\u003e B[FastAPI Server]\n    B --\u003e C[Query Router]\n    B --\u003e D[Ingest Router]\n    B --\u003e E[WebSocket Manager]\n    \n    C --\u003e F[Log Analysis Agent]\n    F --\u003e G[Multi-LLM Provider]\n    F --\u003e H[ChromaDB Vector Store]\n    \n    D --\u003e H\n    E --\u003e I[Real-time Streaming]\n    \n    G --\u003e J[Google Gemini]\n    G --\u003e K[OpenAI GPT]\n    G --\u003e L[Anthropic Claude]\n    G --\u003e M[Groq Llama]\n    \n    H --\u003e N[Vector Embeddings]\n    H --\u003e O[Log Storage]\n```\n\n### Data Flow\n\n1. **Log Ingestion**: Logs enter through API endpoints or real-time streams\n2. **Vector Embedding**: Content is processed using Google's embedding models\n3. **Storage**: Logs are stored in ChromaDB with vector representations\n4. **Query Processing**: Natural language queries are translated to search parameters\n5. **AI Analysis**: Selected LLM analyzes logs and generates insights\n6. **Response**: Results are returned with analysis and recommendations\n\n---\n\n## 🔧 Advanced Usage\n\n### Custom Log Sources\n\nExtend Flash.io by implementing custom log sources:\n\n```python\nfrom log_sources import LogSource\n\nclass CustomLogSource(LogSource):\n    async def stream_logs(self):\n        # Implement your custom log streaming logic\n        pass\n    \n    async def get_logs(self, start_time, end_time, filters):\n        # Implement log retrieval logic\n        pass\n```\n\n### Database Integration\n\nFlash.io supports multiple database backends for log storage:\n\n- **PostgreSQL**: Enterprise-grade relational storage\n- **MySQL**: Popular open-source database\n- **SQLite**: Lightweight local storage\n\n### Cloud Deployments\n\nDeploy Flash.io to your preferred cloud platform:\n\n- **AWS**: Use CloudWatch integration for log ingestion\n- **Google Cloud**: Leverage Stackdriver logs\n- **Azure**: Connect to Azure Monitor logs\n- **Docker**: Container-ready deployment\n\n---\n\n## 🛡️ Security\n\n### Data Protection\n- **Encryption at Rest**: All sensitive data is encrypted using AES-256\n- **API Key Management**: Secure credential storage and rotation\n- **Access Control**: Token-based authentication for API access\n- **Data Retention**: Configurable automatic cleanup policies\n\n### Privacy\n- **Local Processing**: Logs can be processed entirely on-premises\n- **No Data Transmission**: Optional cloud-free operation\n- **Audit Trails**: Complete logging of all analysis activities\n\n---\n\n## 🤝 Contributing\n\nWe welcome contributions! Here's how to get started:\n\n1. **Fork the Repository**\n2. **Create a Feature Branch**: `git checkout -b feature/amazing-feature`\n3. **Commit Your Changes**: `git commit -m 'Add amazing feature'`\n4. **Push to Branch**: `git push origin feature/amazing-feature`\n5. **Open a Pull Request**\n\n### Development Setup\n\n```bash\n# Install development dependencies\npip install -r requirements.txt\n\n# Run tests\npython -m pytest\n\n# Start development server\npython main.py\n```\n\n---\n\n## 📋 Requirements\n\n### Core Dependencies\n- **FastAPI 0.115.9**: Modern, fast web framework\n- **ChromaDB 1.0.5**: Vector database for embeddings\n- **LangChain**: LLM integration framework\n- **Google AI**: Generative AI and embeddings\n- **Uvicorn**: ASGI server for deployment\n\n### AI Provider SDKs\n- **google-generativeai**: Google Gemini integration\n- **openai**: OpenAI GPT models\n- **anthropic**: Anthropic Claude models\n- **groq**: Groq inference platform\n\n---\n\n## 🔍 Troubleshooting\n\n### Common Issues\n\n**API Key Errors**\n```bash\n# Verify your API keys are correctly set\npython -c \"from config import GOOGLE_API_KEY; print('✓ API key configured' if GOOGLE_API_KEY else '✗ API key missing')\"\n```\n\n**Database Connection Issues**\n```bash\n# Check ChromaDB path and permissions\nls -la ./chroma_db/\n```\n\n**Memory Issues**\n```bash\n# Reduce batch size in configuration\nexport MAX_BATCH_SIZE=50\n```\n\n### Getting Help\n\n- **Documentation**: Check our comprehensive guides\n- **Issues**: Report bugs on GitHub Issues\n- **Discussions**: Join our community discussions\n- **Support**: Contact our support team\n\n---\n\n## 🚀 Roadmap\n\n### Upcoming Features\n\n- **🎨 Enhanced Visualizations**: Interactive charts and graphs\n- **📱 Mobile App**: Native iOS and Android applications\n- **🔌 More Integrations**: Slack, Teams, PagerDuty notifications\n- **🤖 Advanced AI**: Custom model fine-tuning capabilities\n- **☁️ Cloud Services**: Managed Flash.io cloud offering\n\n### Version History\n\n- **v0.1.0**: Initial release with core features\n- **v0.2.0**: Multi-LLM support and enhanced UI\n- **v0.3.0**: Real-time streaming and WebSocket integration\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n### 🌟 Star us on GitHub if Flash.io helps your team!\n\n\n*Transform your logs into insights with the power of AI*\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frohittcodes%2Fflashio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frohittcodes%2Fflashio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frohittcodes%2Fflashio/lists"}