https://github.com/ibrahimghali/dynamic-kpi-dashboard

Dynamic KPI Dashboards using Superset, Trino & MongoDB — A scalable, no-code architecture to visualize KPIs without frontend development. Connect Superset to MongoDB via Trino and define KPIs as simple config entries.
https://github.com/ibrahimghali/dynamic-kpi-dashboard

mongodb scalability sql supertest trino

Last synced: 2 months ago
JSON representation

Dynamic KPI Dashboards using Superset, Trino & MongoDB — A scalable, no-code architecture to visualize KPIs without frontend development. Connect Superset to MongoDB via Trino and define KPIs as simple config entries.

• Host: GitHub
• URL: https://github.com/ibrahimghali/dynamic-kpi-dashboard
• Owner: Ibrahimghali
• Created: 2025-06-26T13:49:10.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2025-07-07T13:38:58.000Z (12 months ago)
• Last Synced: 2025-07-07T14:24:43.311Z (12 months ago)
• Topics: mongodb, scalability, sql, supertest, trino
• Language: Python
• Homepage:
• Size: 1.38 MB
• Stars: 0
• Watchers: 0
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Dynamic KPI Dashboard 📊

> **Transform your data analytics workflow from manual frontend development to dynamic, configuration-driven dashboards**

[![SonarQube Quality](https://img.shields.io/badge/SonarQube-Integrated-green.svg)](http://localhost:9000)

[![Docker](https://img.shields.io/badge/Docker-Compose-blue.svg)](docker-compose.yml)

[![Python](https://img.shields.io/badge/Python-3.11-blue.svg)](requirements.txt)

[![GitLab CI](https://img.shields.io/badge/GitLab-CI%2FCD-orange.svg)](.gitlab-ci.yml)

This project provides a **scalable microservices architecture** to connect **Apache Superset to MongoDB via Trino**, enabling **dynamic dashboards** without manual frontend coding.

---

## 🎯 Goals & Value Proposition

### **Business Problem Solved**

- **Before**: Each KPI requires ~1 day of Angular development

- **After**: KPIs created instantly through configuration

- **Result**: 10x faster dashboard creation, zero frontend dependencies

### **Technical Goals**

- ✅ **Zero-code KPI creation** for non-technical users

- ✅ **Scalable architecture** supporting unlimited metrics  

- ✅ **Real-time dashboards** with automatic data refresh

- ✅ **Enterprise-grade** monitoring and code quality

---

## 🏗️ Architecture Overview

![Architecture Diagram](assets/architecture.png)

### **Microservices Stack**

```

┌─────────────┐    ┌─────────────┐    ┌─────────────┐

│   Superset  │◄───┤    Trino    │◄───┤   MongoDB   │

│ (Frontend)  │    │ (Query Eng) │    │ (Data Lake) │

└─────────────┘    └─────────────┘    └─────────────┘

       │                   │                   │

       ▼                   ▼                   ▼

  Port 8088           Port 8080           Port 27017

```

### **Data Flow**

1. **MongoDB**: Stores KPI definitions + business data

2. **Trino**: Distributed query engine for MongoDB

3. **Superset**: Web-based visualization platform

4. **SonarQube**: Code quality monitoring

5. **Docker**: Container orchestration

---

## 🚀 Quick Start

### **1. Start All Services**

```bash

# Launch the entire stack

docker-compose up -d

# Check service status

docker-compose ps

```

### **2. Generate Sample Data**

```bash

# Generate financial data for testing

docker-compose run --rm data-generator

```

### **3. Access Applications**

| Service | URL | Credentials |

|---------|-----|-------------|

| **Superset** | http://localhost:8088 | `admin` / `admin` |

| **Trino UI** | http://localhost:8080 | No auth required |

| **SonarQube** | http://localhost:9000 | `admin` / `admin` |

| **MongoDB** | localhost:27017 | `admin` / `admin` |

---

## � CI/CD Pipeline

### **GitLab CI/CD Stages**

```

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐

│  Validate   │───▶│    Test     │───▶│   Quality   │───▶│    Build    │───▶│   Deploy    │

│ • Lint Code │    │ • Unit Tests│    │ • SonarQube │    │ • Docker    │    │ • Staging   │

│ • Dockerfile│    │ • Integration│    │ • Coverage  │    │ • Registry  │    │ • Production│

└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘

```

### **Pipeline Features**

- ✅ **Automated Code Quality** - Linting, formatting, SonarQube analysis

- ✅ **Comprehensive Testing** - Unit tests, integration tests, coverage reports

- ✅ **Docker Image Building** - Automated image builds and registry push

- ✅ **Multi-Environment Deployment** - Staging and production environments

- ✅ **Performance Testing** - Locust-based load testing

- ✅ **Manual Gates** - Production deployments require manual approval

### **Required GitLab Variables**

Set these in GitLab → Settings → CI/CD → Variables:

```bash

SONAR_HOST_URL=https://your-sonarqube-instance.com

SONAR_TOKEN=your-sonarqube-token

CI_REGISTRY_USER=your-registry-username

CI_REGISTRY_PASSWORD=your-registry-password

```

---

## �🔧 Configuration & Setup

### **Connect Superset to MongoDB**

1. Go to Superset → Settings → Database Connections

2. Add new connection:

   ```

   URI: trino://trino@trino:8080/mongodb

   ```

### **Sample KPI Definition**

Store dynamic KPI configs in MongoDB:

```json

{

  "kpi_id": "revenue_growth",

  "title": "Monthly Revenue Growth",

  "type": "line_chart",

  "query": "SELECT date, SUM(close * volume) as revenue FROM mongodb.financial_data.stock_prices GROUP BY date ORDER BY date",

  "visualization": {

    "format": "currency",

    "refresh_rate": "5min",

    "chart_type": "line"

  },

  "permissions": ["analyst", "manager", "admin"]

}

```

---

## 📈 Code Quality & Testing

### **SonarQube Analysis**

```bash

# Start SonarQube

docker-compose up -d sonarqube sonar_postgres

# Run code analysis

docker-compose run --rm sonar-scanner sonar-scanner

# View results at http://localhost:9000

```

### **Load Testing with Locust**

```bash

# Install Locust

pip install locust

# Run performance tests

locust -f src/test/test_superset.py --host=http://localhost:8088

```

### **Current Quality Metrics**

- ✅ **0 Bugs** detected

- ✅ **0 Vulnerabilities** found  

- ✅ **0 Code Smells** identified

- ⚠️ **1 Encoding Warning** (minor)

---

## 📊 Sample Queries & Use Cases

### **Financial Analytics**

```sql

-- Stock performance by sector

SELECT 

    sector,

    AVG(close) as avg_price,

    SUM(volume) as total_volume

FROM mongodb.financial_data.stock_prices 

WHERE date >= current_date - interval '30' day

GROUP BY sector;

```

### **Real-time Monitoring**

```sql

-- Recent trading activity

SELECT 

    symbol,

    company_name,

    close,

    volume,

    date

FROM mongodb.financial_data.stock_prices 

ORDER BY date DESC 

LIMIT 100;

```

---

## 🔧 Development & Deployment

### **Project Structure**

```

📁 Dynamic-kpi-dashboard/

├── 📁 src/

│   ├── 📁 main/           # Business logic

│   └── 📁 test/           # Test suites

├── 📁 config/

│   ├── 📁 superset/       # Superset configs

│   └── 📁 trino/          # Trino catalogs

├── 📁 docs/               # Documentation

├── 📁 assets/             # Architecture diagrams

├── 🐳 docker-compose.yml  # Service orchestration

├── 🐳 Dockerfile          # Data generator image

├── 📊 sonar-project.properties  # Code quality config

└── 📋 requirements.txt    # Python dependencies

```

### **Technology Stack**

- **Backend**: Python 3.11, PyMongo

- **Database**: MongoDB, PostgreSQL  

- **Analytics**: Apache Superset, Trino

- **Quality**: SonarQube

- **DevOps**: Docker, Docker Compose

---

## 🚦 Monitoring & Observability

### **Health Checks**

```bash

# Check all services

docker-compose ps

# View logs

docker-compose logs -f superset

docker-compose logs -f trino

docker-compose logs -f mongodb

```

### **Performance Metrics**

- **Superset Response Time**: < 2s for dashboards

- **Trino Query Performance**: < 5s for complex aggregations

- **MongoDB Throughput**: 1M+ documents/second

---

## 🔮 Roadmap & Future Enhancements

### **Phase 1** ✅ (Current)

- [x] Basic Superset-Trino-MongoDB integration

- [x] Sample data generation

- [x] Docker containerization

- [x] Code quality monitoring

- [x] GitLab CI/CD pipeline

### **Phase 2** 🚧 (In Progress)

- [ ] KPI Definition Admin Panel

- [ ] Automated dashboard creation via Superset API

- [ ] Role-based access control

- [ ] Real-time data streaming

### **Phase 3** 📋 (Planned)

- [ ] Advanced visualization templates

- [ ] Machine learning integration

- [ ] Multi-tenant support

- [ ] Enterprise SSO integration

---

### **Development Setup**

```bash

# Clone repository

git clone 

cd Dynamic-kpi-dashboard

# Install dependencies

pip install -r requirements.txt

# Run tests

python -m pytest src/test/

# Run code quality checks

docker-compose run --rm sonar-scanner sonar-scanner

```

### **Supported Versions**

- **Superset**: 3.0.0

- **Trino**: 443

- **MongoDB**: Latest

- **Python**: 3.11+

---

## 📝 License & Support

**License**: MIT License  

**Maintainer**: Smart Conseil Team  

**Support**: Create an issue for bugs or feature requests

---

## 🏆 Success Metrics

| Metric | Before | After | Improvement |

|--------|--------|-------|-------------|

| **KPI Creation Time** | 1 day | 5 minutes | **99% faster** |

| **Developer Dependency** | High | Zero | **100% reduction** |

| **Maintenance Overhead** | Manual | Automated | **Eliminated** |

| **Scalability** | Limited | Unlimited | **∞ growth** |

---

## 🔒 Security Notice

This repository contains only development configurations with default credentials. 

**For production deployments:**

- Change all default passwords in `docker-compose.yml`

- Set secure `SUPERSET_SECRET_KEY` 

- Use proper database credentials

- Configure SonarQube token via `SONAR_TOKEN` environment variable

- Review and update all security settings

**Never commit:**

- Production credentials

- API keys or tokens

- Personal access tokens

- SSL certificates or private keys