https://github.com/kaito47802/condped-anova
[KDD '26] Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces
https://github.com/kaito47802/condped-anova
ai artificial-intelligence automatic-machine-learning automl hyperparameter-optimization hyperparameter-tuning machine-learning
Last synced: 11 days ago
JSON representation
[KDD '26] Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces
- Host: GitHub
- URL: https://github.com/kaito47802/condped-anova
- Owner: kAIto47802
- Created: 2026-02-09T17:06:53.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2026-05-16T07:59:54.000Z (11 days ago)
- Last Synced: 2026-05-16T09:37:53.452Z (11 days ago)
- Topics: ai, artificial-intelligence, automatic-machine-learning, automl, hyperparameter-optimization, hyperparameter-tuning, machine-learning
- Language: Python
- Homepage:
- Size: 236 KB
- Stars: 3
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces
✨️ Accepted to the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining ✨
(KDD 2026)
Kaito Baba Yoshihiko Ozaki Shuhei Watanabe
We propose conditional PED-ANOVA (condPED-ANOVA), a principled framework for estimating hyperparameter importance (HPI) in conditional search spaces, where the presence or domain of a hyperparameter can depend on other hyperparameters.
Although the original PED-ANOVA provides a fast and efficient way to estimate HPI within the top-performing regions of the search space, it assumes a fixed, unconditional search space and therefore cannot properly handle conditional hyperparameters.
To address this, we introduce a conditional HPI for top-performing regions and derive a closed-form estimator that accurately reflects conditional activation and domain changes.
Experiments show that naive adaptations of existing HPI estimators yield misleading or uninterpretable importances in conditional settings, whereas condPED-ANOVA consistently provides meaningful importances that reflect the underlying conditional structure.
🚀 Quick Start 🚀
`cond_ped_anova` is fully compatible with [Optuna](https://github.com/optuna/optuna)’s built-in hyperparameter importance API.
## 1. Installation
```bash
uv add git+https://github.com/kAIto47802/condPED-ANOVA.git
# If you're using pip:
# pip install git+https://github.com/kAIto47802/condPED-ANOVA.git
```
## 2. Usage
```python
import optuna
from cond_ped_anova import CondPedAnovaImportanceEvaluator
def objective(trial: optuna.trial.Trial) -> float:
c = trial.suggest_float("c", 0.0, 1.0)
if c < 0.5:
x = trial.suggest_float("x", -5.0, -2.0)
return x
else:
y = trial.suggest_float("y", 2.0, 5.0)
return y
sampler = optuna.samplers.RandomSampler(seed=42)
study = optuna.create_study(direction="minimize", sampler=sampler)
study.optimize(objective, n_trials=1000)
evaluator = CondPedAnovaImportanceEvaluator()
importance = optuna.importance.get_param_importances(study, evaluator=evaluator)
```
📊 Reproducing the Results 📊
All experiments in [our paper](https://arxiv.org/abs/2601.20800) are fully reproducible.
## 1. Clone this Repository & Install Dependencies
```bash
git clone https://github.com/kAIto47802/condPED-ANOVA.git
```
```bash
uv sync --python 3.13 --extra experiments
# If you're using pip:
# pip install --upgrade pip # enable PEP 660 support
# pip install -e '.[experiments]'
```
## 2. Run Experiments
Running the commands below reproduces the corresponding experiments and regenerates the exact figures:
- condPED-ANOVA on synthetic problems (Figures 1, 7, and 11):
```bash
./run_cond_ped_anova.sh
```
- Baseline methods on synthetic problems (Figures 2, 9, and 10):
```bash
./run_baselines.sh
```
- Ablation study (Figures 3 and 4):
```bash
./run_ablations.sh
```
- condPED-ANOVA on real-world benchmarks (Figures 5 and 14 and Tables 1 and 2):
```bash
# prepare YAHPO Gym data
git clone https://github.com/slds-lmu/yahpo_data.git
./run_yahpo_gym.sh
```
- Runtime comparison (Figure 6):
```bash
./run_runtime_comparison.sh
```
- condPED-ANOVA Results with Different $N$ (Figure 8):
```bash
./run_cond_ped_anova_different_n.sh
```
- Additional Experiments on Synthetic
Problems (Figures 12 and 13):
```bash
./run_cond_ped_anova_additional.sh
```
## 3. Check the Results
The generated figures will be saved in the `figures/` directory.
The raw experiment outputs (pickled results) and the values used in the tables are saved under `results/`.
> [!NOTE]
> If a LaTeX environment is not available, the figures will be rendered without LaTeX, which may slightly change font rendering and layout.
🔖 Citation 🔖
If you find condPED-ANOVA useful in your research, please consider citing the following paper:
```bibtex
@article{baba2026condpedanova,
title={Conditional {PED-ANOVA}: Hyperparameter Importance in Hierarchical \& Dynamic Search Spaces},
author={Baba, Kaito and Ozaki, Yoshihiko and Watanabe, Shuhei},
journal={arXiv preprint arXiv:2601.20800},
year={2026},
}
```