https://github.com/udao-moo/udao-spark-optimizer

A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning
https://github.com/udao-moo/udao-spark-optimizer

knobs-tuning modeling multi-objective-optimization optimization spark sparksql

Last synced: 4 months ago
JSON representation

A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning

• Host: GitHub
• URL: https://github.com/udao-moo/udao-spark-optimizer
• Owner: udao-moo
• License: apache-2.0
• Created: 2024-07-13T21:33:16.000Z (12 months ago)
• Default Branch: main
• Last Pushed: 2024-12-12T15:51:44.000Z (7 months ago)
• Last Synced: 2025-02-02T01:11:22.341Z (5 months ago)
• Topics: knobs-tuning, modeling, multi-objective-optimization, optimization, spark, sparksql
• Language: Python
• Homepage:
• Size: 4.19 MB
• Stars: 6
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning

This repo is the implementation of the paper [A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning][1].

## Preparation

- Spark Setup: [spark-setup/README.md][2]

- Python Setup (python 3.9+), `pip install -r requirements.txt`

- Trace Collection Scripts [udao_trace_examples/README.md][3]

## Compile Time/Runtime Optimization

See details in [playground/README.md][4]

[1]: https://chenghao.pages.dev/papers/vldb24-lyu-tr.pdf

[2]: ./spark-setup/README.md

[3]: ./udao_trace_examples/README.md

[4]: ./playground/README.md