https://github.com/hermya/cs511_breolap
Realtime-OLAPs can be used to analyze of high-velocity data. We aim to benchmark ReOLAPs with respect to critical financial use-cases.
https://github.com/hermya/cs511_breolap
benchmarking clickhouse docker druid fintech grafana kafka olap pinot prometheus python sql starrocks
Last synced: about 1 month ago
JSON representation
Realtime-OLAPs can be used to analyze of high-velocity data. We aim to benchmark ReOLAPs with respect to critical financial use-cases.
- Host: GitHub
- URL: https://github.com/hermya/cs511_breolap
- Owner: hermya
- Created: 2024-03-28T04:36:02.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2025-02-10T22:16:10.000Z (about 1 year ago)
- Last Synced: 2025-04-03T10:51:26.600Z (about 1 year ago)
- Topics: benchmarking, clickhouse, docker, druid, fintech, grafana, kafka, olap, pinot, prometheus, python, sql, starrocks
- Language: Jupyter Notebook
- Homepage:
- Size: 9.72 MB
- Stars: 2
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# BReOLAP: Your Fintech Realtime-OLAP Solution 🚀
Fintech organizations require an OLAP solution that can resolve complex analytical queries in milliseconds, enabling them to make informed, split-second decisions. Choosing the right Realtime-OLAP is crucial, and **BReOLAP** is here to guide you through selecting the best option for your specific needs! Check out **[this](https://lnkd.in/gQDAbzTF)** formal report for our end-to-end scientific study
----
## 🎯 Purpose
Our goal at BReOLAP is multi-fold:
- **Model Definition:** We define a model where specific requirements dictate the choice of database.
- **Benchmarking Insights:** Demonstrate our methodology for benchmark comparisons.
- **System Replication:** Provide insights into replicating contemporary systems for your use.
## 🏅 Candidate Realtime-OLAP
We've rigorously tested the following Realtime-OLAPs to ensure you get reliable, actionable insights:
| 
**ClickHouse** | 
**Pinot** |
|---------------------------------------------|------------------------------------------|
| 
**Druid** | 
**StarRocks** |
### 🚧 In Progress
We are also in the process of evaluating these promising databases:
* **Doris**
* **Kylin**
# Flow of data
```mermaid
graph TD;
subgraph ClickHouse & StarRocks
A[data-source] -->|Kafka| B[sink-connector]
subgraph DB1[Database]
F[node] <--> C[controller] <--> D[node]
end
B --> DB1
DB1 -->|Processed Data| E[Query Engine & Ingestion Verifier]
end
subgraph Druid & Pinot
G[data-source] -->|Kafka| H[internal-stream-ingestor]
subgraph DB2[Database]
I[controller] <--> J[node] <--> Q[historical-nodes]
end
H --> DB2
DB2 -->|Processed Data| K[Query Engine & Ingestion Lag Tracer]
end
```
---
## 🛠️ Deep Dive into Our Technology
We're ready to answer your questions immediately! For those interested in a deeper understanding, here's how we equip you with the best tools:
- **Distributed Systems Setup:** Learn how we configure each candidate database to function efficiently in a distributed environment.
- **Table Definitions:** Discover how we define tables to run benchmark tests effectively.
- **Data Stream Integration:** See how we set up input streams to simulate real-time data flow.
- **Query Execution:** Understand our configurations for running queries and retrieving results smoothly.
- **Monitoring and Metrics:** Explore our setup with Prometheus and Grafana for real-time metrics collection.
Feel free to explore our [Documentation](#) for detailed guides and setups.
## 📢 Stay Tuned
For updates, further details, and live results, keep an eye on this space! We're constantly working to expand our insights and tools to help you stay ahead in the fast-paced world of Fintech.
---
We're excited to help you navigate the complex landscape of Realtime-OLAP solutions. With BReOLAP, you're not just choosing a tool; you're setting the stage for future-proof financial analytics!