{"id":29100608,"url":"https://github.com/ibrahimghali/dynamic-kpi-dashboard","last_synced_at":"2026-04-27T17:33:47.597Z","repository":{"id":301529154,"uuid":"1009048662","full_name":"Ibrahimghali/Dynamic-kpi-dashboard","owner":"Ibrahimghali","description":"Dynamic KPI Dashboards using Superset, Trino \u0026 MongoDB — A scalable, no-code architecture to visualize KPIs without frontend development. Connect Superset to MongoDB via Trino and define KPIs as simple config entries.","archived":false,"fork":false,"pushed_at":"2025-07-07T13:38:58.000Z","size":1447,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-07T14:24:43.311Z","etag":null,"topics":["mongodb","scalability","sql","supertest","trino"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Ibrahimghali.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-26T13:49:10.000Z","updated_at":"2025-07-07T13:39:02.000Z","dependencies_parsed_at":"2025-06-28T10:20:14.736Z","dependency_job_id":null,"html_url":"https://github.com/Ibrahimghali/Dynamic-kpi-dashboard","commit_stats":null,"previous_names":["ibrahimghali/mongo-superset-using-trino","ibrahimghali/dynamic-kpi-dashboard"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Ibrahimghali/Dynamic-kpi-dashboard","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ibrahimghali%2FDynamic-kpi-dashboard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ibrahimghali%2FDynamic-kpi-dashboard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ibrahimghali%2FDynamic-kpi-dashboard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ibrahimghali%2FDynamic-kpi-dashboard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Ibrahimghali","download_url":"https://codeload.github.com/Ibrahimghali/Dynamic-kpi-dashboard/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ibrahimghali%2FDynamic-kpi-dashboard/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32348048,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-27T17:12:42.749Z","status":"ssl_error","status_checked_at":"2026-04-27T17:12:41.658Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mongodb","scalability","sql","supertest","trino"],"created_at":"2025-06-28T18:37:13.273Z","updated_at":"2026-04-27T17:33:47.591Z","avatar_url":"https://github.com/Ibrahimghali.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dynamic KPI Dashboard 📊\n\n\u003e **Transform your data analytics workflow from manual frontend development to dynamic, configuration-driven dashboards**\n\n[![SonarQube Quality](https://img.shields.io/badge/SonarQube-Integrated-green.svg)](http://localhost:9000)\n[![Docker](https://img.shields.io/badge/Docker-Compose-blue.svg)](docker-compose.yml)\n[![Python](https://img.shields.io/badge/Python-3.11-blue.svg)](requirements.txt)\n[![GitLab CI](https://img.shields.io/badge/GitLab-CI%2FCD-orange.svg)](.gitlab-ci.yml)\n\nThis project provides a **scalable microservices architecture** to connect **Apache Superset to MongoDB via Trino**, enabling **dynamic dashboards** without manual frontend coding.\n\n---\n\n## 🎯 Goals \u0026 Value Proposition\n\n### **Business Problem Solved**\n- **Before**: Each KPI requires ~1 day of Angular development\n- **After**: KPIs created instantly through configuration\n- **Result**: 10x faster dashboard creation, zero frontend dependencies\n\n### **Technical Goals**\n- ✅ **Zero-code KPI creation** for non-technical users\n- ✅ **Scalable architecture** supporting unlimited metrics  \n- ✅ **Real-time dashboards** with automatic data refresh\n- ✅ **Enterprise-grade** monitoring and code quality\n\n---\n\n## 🏗️ Architecture Overview\n\n![Architecture Diagram](assets/architecture.png)\n\n### **Microservices Stack**\n```\n┌─────────────┐    ┌─────────────┐    ┌─────────────┐\n│   Superset  │◄───┤    Trino    │◄───┤   MongoDB   │\n│ (Frontend)  │    │ (Query Eng) │    │ (Data Lake) │\n└─────────────┘    └─────────────┘    └─────────────┘\n       │                   │                   │\n       ▼                   ▼                   ▼\n  Port 8088           Port 8080           Port 27017\n```\n\n### **Data Flow**\n1. **MongoDB**: Stores KPI definitions + business data\n2. **Trino**: Distributed query engine for MongoDB\n3. **Superset**: Web-based visualization platform\n4. **SonarQube**: Code quality monitoring\n5. **Docker**: Container orchestration\n\n---\n\n## 🚀 Quick Start\n\n### **1. Start All Services**\n```bash\n# Launch the entire stack\ndocker-compose up -d\n\n# Check service status\ndocker-compose ps\n```\n\n### **2. Generate Sample Data**\n```bash\n# Generate financial data for testing\ndocker-compose run --rm data-generator\n```\n\n### **3. Access Applications**\n| Service | URL | Credentials |\n|---------|-----|-------------|\n| **Superset** | http://localhost:8088 | `admin` / `admin` |\n| **Trino UI** | http://localhost:8080 | No auth required |\n| **SonarQube** | http://localhost:9000 | `admin` / `admin` |\n| **MongoDB** | localhost:27017 | `admin` / `admin` |\n\n---\n\n## � CI/CD Pipeline\n\n### **GitLab CI/CD Stages**\n```\n┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐\n│  Validate   │───▶│    Test     │───▶│   Quality   │───▶│    Build    │───▶│   Deploy    │\n│ • Lint Code │    │ • Unit Tests│    │ • SonarQube │    │ • Docker    │    │ • Staging   │\n│ • Dockerfile│    │ • Integration│    │ • Coverage  │    │ • Registry  │    │ • Production│\n└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘\n```\n\n### **Pipeline Features**\n- ✅ **Automated Code Quality** - Linting, formatting, SonarQube analysis\n- ✅ **Comprehensive Testing** - Unit tests, integration tests, coverage reports\n- ✅ **Docker Image Building** - Automated image builds and registry push\n- ✅ **Multi-Environment Deployment** - Staging and production environments\n- ✅ **Performance Testing** - Locust-based load testing\n- ✅ **Manual Gates** - Production deployments require manual approval\n\n### **Required GitLab Variables**\nSet these in GitLab → Settings → CI/CD → Variables:\n```bash\nSONAR_HOST_URL=https://your-sonarqube-instance.com\nSONAR_TOKEN=your-sonarqube-token\nCI_REGISTRY_USER=your-registry-username\nCI_REGISTRY_PASSWORD=your-registry-password\n```\n\n---\n\n## �🔧 Configuration \u0026 Setup\n\n### **Connect Superset to MongoDB**\n1. Go to Superset → Settings → Database Connections\n2. Add new connection:\n   ```\n   URI: trino://trino@trino:8080/mongodb\n   ```\n\n### **Sample KPI Definition**\nStore dynamic KPI configs in MongoDB:\n```json\n{\n  \"kpi_id\": \"revenue_growth\",\n  \"title\": \"Monthly Revenue Growth\",\n  \"type\": \"line_chart\",\n  \"query\": \"SELECT date, SUM(close * volume) as revenue FROM mongodb.financial_data.stock_prices GROUP BY date ORDER BY date\",\n  \"visualization\": {\n    \"format\": \"currency\",\n    \"refresh_rate\": \"5min\",\n    \"chart_type\": \"line\"\n  },\n  \"permissions\": [\"analyst\", \"manager\", \"admin\"]\n}\n```\n\n---\n\n## 📈 Code Quality \u0026 Testing\n\n### **SonarQube Analysis**\n```bash\n# Start SonarQube\ndocker-compose up -d sonarqube sonar_postgres\n\n# Run code analysis\ndocker-compose run --rm sonar-scanner sonar-scanner\n\n# View results at http://localhost:9000\n```\n\n### **Load Testing with Locust**\n```bash\n# Install Locust\npip install locust\n\n# Run performance tests\nlocust -f src/test/test_superset.py --host=http://localhost:8088\n```\n\n### **Current Quality Metrics**\n- ✅ **0 Bugs** detected\n- ✅ **0 Vulnerabilities** found  \n- ✅ **0 Code Smells** identified\n- ⚠️ **1 Encoding Warning** (minor)\n\n---\n\n## 📊 Sample Queries \u0026 Use Cases\n\n### **Financial Analytics**\n```sql\n-- Stock performance by sector\nSELECT \n    sector,\n    AVG(close) as avg_price,\n    SUM(volume) as total_volume\nFROM mongodb.financial_data.stock_prices \nWHERE date \u003e= current_date - interval '30' day\nGROUP BY sector;\n```\n\n### **Real-time Monitoring**\n```sql\n-- Recent trading activity\nSELECT \n    symbol,\n    company_name,\n    close,\n    volume,\n    date\nFROM mongodb.financial_data.stock_prices \nORDER BY date DESC \nLIMIT 100;\n```\n\n---\n\n## 🔧 Development \u0026 Deployment\n\n### **Project Structure**\n```\n📁 Dynamic-kpi-dashboard/\n├── 📁 src/\n│   ├── 📁 main/           # Business logic\n│   └── 📁 test/           # Test suites\n├── 📁 config/\n│   ├── 📁 superset/       # Superset configs\n│   └── 📁 trino/          # Trino catalogs\n├── 📁 docs/               # Documentation\n├── 📁 assets/             # Architecture diagrams\n├── 🐳 docker-compose.yml  # Service orchestration\n├── 🐳 Dockerfile          # Data generator image\n├── 📊 sonar-project.properties  # Code quality config\n└── 📋 requirements.txt    # Python dependencies\n```\n\n### **Technology Stack**\n- **Backend**: Python 3.11, PyMongo\n- **Database**: MongoDB, PostgreSQL  \n- **Analytics**: Apache Superset, Trino\n- **Quality**: SonarQube\n- **DevOps**: Docker, Docker Compose\n\n---\n\n## 🚦 Monitoring \u0026 Observability\n\n### **Health Checks**\n```bash\n# Check all services\ndocker-compose ps\n\n# View logs\ndocker-compose logs -f superset\ndocker-compose logs -f trino\ndocker-compose logs -f mongodb\n```\n\n### **Performance Metrics**\n- **Superset Response Time**: \u003c 2s for dashboards\n- **Trino Query Performance**: \u003c 5s for complex aggregations\n- **MongoDB Throughput**: 1M+ documents/second\n\n---\n\n## 🔮 Roadmap \u0026 Future Enhancements\n\n### **Phase 1** ✅ (Current)\n- [x] Basic Superset-Trino-MongoDB integration\n- [x] Sample data generation\n- [x] Docker containerization\n- [x] Code quality monitoring\n- [x] GitLab CI/CD pipeline\n\n### **Phase 2** 🚧 (In Progress)\n- [ ] KPI Definition Admin Panel\n- [ ] Automated dashboard creation via Superset API\n- [ ] Role-based access control\n- [ ] Real-time data streaming\n\n### **Phase 3** 📋 (Planned)\n- [ ] Advanced visualization templates\n- [ ] Machine learning integration\n- [ ] Multi-tenant support\n- [ ] Enterprise SSO integration\n---\n\n### **Development Setup**\n```bash\n# Clone repository\ngit clone \u003crepository-url\u003e\ncd Dynamic-kpi-dashboard\n\n# Install dependencies\npip install -r requirements.txt\n\n# Run tests\npython -m pytest src/test/\n\n# Run code quality checks\ndocker-compose run --rm sonar-scanner sonar-scanner\n```\n\n### **Supported Versions**\n- **Superset**: 3.0.0\n- **Trino**: 443\n- **MongoDB**: Latest\n- **Python**: 3.11+\n\n---\n\n## 📝 License \u0026 Support\n\n**License**: MIT License  \n**Maintainer**: Smart Conseil Team  \n**Support**: Create an issue for bugs or feature requests\n\n---\n\n## 🏆 Success Metrics\n\n| Metric | Before | After | Improvement |\n|--------|--------|-------|-------------|\n| **KPI Creation Time** | 1 day | 5 minutes | **99% faster** |\n| **Developer Dependency** | High | Zero | **100% reduction** |\n| **Maintenance Overhead** | Manual | Automated | **Eliminated** |\n| **Scalability** | Limited | Unlimited | **∞ growth** |\n\n---\n\n## 🔒 Security Notice\n\nThis repository contains only development configurations with default credentials. \n\n**For production deployments:**\n- Change all default passwords in `docker-compose.yml`\n- Set secure `SUPERSET_SECRET_KEY` \n- Use proper database credentials\n- Configure SonarQube token via `SONAR_TOKEN` environment variable\n- Review and update all security settings\n\n**Never commit:**\n- Production credentials\n- API keys or tokens\n- Personal access tokens\n- SSL certificates or private keys\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibrahimghali%2Fdynamic-kpi-dashboard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fibrahimghali%2Fdynamic-kpi-dashboard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibrahimghali%2Fdynamic-kpi-dashboard/lists"}