https://github.com/vcerqueira/modelradar
Aspect-based Forecasting Accuracy
https://github.com/vcerqueira/modelradar
deep-learning evaluation-framework forecasting machine-learning time-series
Last synced: 3 months ago
JSON representation
Aspect-based Forecasting Accuracy
- Host: GitHub
- URL: https://github.com/vcerqueira/modelradar
- Owner: vcerqueira
- License: other
- Created: 2024-04-18T14:44:11.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2025-07-15T20:51:52.000Z (9 months ago)
- Last Synced: 2025-10-25T13:55:32.993Z (6 months ago)
- Topics: deep-learning, evaluation-framework, forecasting, machine-learning, time-series
- Language: Jupyter Notebook
- Homepage:
- Size: 9.56 MB
- Stars: 5
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Model Radar 🎯
A framework for aspect-based evaluation of time series forecasting models based on Nixtla's ecosystem.
[](https://pypi.org/project/modelradar/)
[](https://github.com/vcerqueira/modelradar)
[](https://pepy.tech/project/modelradar)
## Overview
Model Radar introduces a novel aspect-based forecasting evaluation approach that goes beyond traditional aggregate metrics. Our framework enables:
- Fine-grained performance analysis across different forecasting aspects
- Better understanding of model behavior in varying conditions
- More informed model selection based on specific use case requirements
## 🚀 Getting Started
Check the `notebooks` folder for usage examples and tutorials.
Check **[ModelRadar-Experiments](https://github.com/vcerqueira/experiments-modelradar)** repository for a thorough
application of ModelRadar.
## Installation
You can install modelradar using pip:
```bash
pip install modelradar
```
### [Optional] Installation from source
To install modelradar from source, clone the repository and run the following command:
```bash
git clone https://github.com/vcerqueira/modelradar
pip install -e modelradar
```
### Prerequisites
Required dependencies:
```
utilsforecast==0.2.11
numpy==1.26.4
plotnine==0.14.5
```
⚠️ I've noticed some issues when running with more recent versions of numpy and utilsforecast.
Try to use the versions above.
### Examples
Besides the examples in the `notebooks` folder, here's some outputs you can get from *modelradar*:
- Spider chart with overview on several dimensions:
- Parallel coordinates chart with overview on several dimensions:
- Barplot chart controlling for a given variable (in this case, anomaly status):
- Grouped bar plot showing win/draw/loss ratios wrt different models:
## 📑 References
> Cerqueira, V., Roque, L., & Soares, C. "Forecasting with Deep Learning: Beyond Average of Average of Average Performance." Discovery Science: 27th International Conference, DS 2024, Pisa, Italy, 2024, Proceedings 27. Springer International Publishing, 2024.
Check DS24 folder to reproduce the experiments published on this paper.
The main repository and package contains an updated framework.
### **⚠️ WARNING**
> modelradar is in the early stages of development.
> The codebase may undergo significant changes.
> If you encounter any issues, please report
> them in [GitHub Issues](https://github.com/vcerqueira/modelradar/issues)