{"id":35190219,"url":"https://github.com/mgarralda/spark-self-tuning-framework","last_synced_at":"2026-05-20T15:34:47.491Z","repository":{"id":319792651,"uuid":"1078651502","full_name":"mgarralda/spark-self-tuning-framework","owner":"mgarralda","description":"A Self-Tuning Framework for Cost-Aware Apache Spark Configuration","archived":false,"fork":false,"pushed_at":"2025-10-28T08:08:01.000Z","size":6025,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-28T10:08:29.168Z","etag":null,"topics":["bayesian-optimization","big-data","metaheuristic-optimisation","spark-tuning","transfer-learning"],"latest_commit_sha":null,"homepage":"https://doi.org/10.1016/j.future.2025.107730","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mgarralda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-18T05:58:58.000Z","updated_at":"2025-10-28T08:08:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"b0dfdfe7-07e0-4f04-a8ca-5c5917e68e04","html_url":"https://github.com/mgarralda/spark-self-tuning-framework","commit_stats":null,"previous_names":["mgarralda/spark-self-tuning-framework"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mgarralda/spark-self-tuning-framework","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgarralda%2Fspark-self-tuning-framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgarralda%2Fspark-self-tuning-framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgarralda%2Fspark-self-tuning-framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgarralda%2Fspark-self-tuning-framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mgarralda","download_url":"https://codeload.github.com/mgarralda/spark-self-tuning-framework/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgarralda%2Fspark-self-tuning-framework/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28111183,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-29T02:00:07.021Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bayesian-optimization","big-data","metaheuristic-optimisation","spark-tuning","transfer-learning"],"created_at":"2025-12-29T05:38:00.408Z","updated_at":"2026-05-20T15:34:47.484Z","avatar_url":"https://github.com/mgarralda.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark Self-Tuning Framework (STL–ILS–TS–BO)\n\nImplementation of the framework proposed in  \n**“A hybrid metaheuristics–Bayesian Optimization framework with safe transfer learning for continuous Spark tuning”**  \n(*Future Generation Computer Systems*, 2025).  \nDOI: https://doi.org/10.1016/j.future.2025.108325\n\n---\n\n## 🧠 Overview\n\nThe **Spark Self-Tuning Framework** provides continuous and adaptive optimization of Apache Spark configurations by combining:\n\n- **Bayesian Optimization (BO)** with a custom acquisition function (`LCB`)\n- **Compositional surrogate models** for performance and uncertainty estimation\n- **Iterated Local Search + Tabu Search (ILS–TS)** for guided exploration and local refinement\n- **Safe Transfer Learning (STL-PARN)** to reuse historical workload executions\n- **Baseline implementations**: *Garralda*, *TurBO*, *YORO*, and *Naïve BO*\n\nThis framework enables cost-aware, knowledge-driven configuration tuning for complex Spark workloads.\n\n---\n\n## 📁 Project Structure\n\n```\nproject-root/\n├── src/                       # Core framework\n│   └── framework/\n│       ├── proposed/          # Main optimization method\n│       ├── metaheuristics/    # Tabu + ILS modules\n│       ├── bayesian_optimization/\n│       └── safe_transfer_learning/\n├── src_resources/             # Experiment runners\n├── resources/                 # Datasets \u0026 results\n```\n\n---\n\n## 📊 Data\n\nExperimental data and benchmarks are provided under:\n\n```\nresources/\n├── dataset/\n│   ├── historical_dataset.json\n│   ├── lhs_initialization.json\n├── experiment_results/\n│   ├── performance_model/\n│   ├── optimization_model/\n```\n\n---\n\n## 📜 License\n\nThis project is dual-licensed under:\n\n- **CC BY-NC 4.0** for academic and research use  \n  \u003chttps://creativecommons.org/licenses/by-nc/4.0/\u003e\n- **Commercial use is not allowed.**  \n  Any use of this software or its derivatives for commercial purposes is strictly prohibited.\n\nDistributed on an “AS IS” basis, without warranties or conditions of any kind.  \nSee the [LICENSE](LICENSE) file for details.\n\n---\n\n## 📚 Citation\n\nIf you use this framework, its methodology, infrastructure, datasets, or derived components in research, benchmarking studies, technical documentation, or industrial reports, please cite the associated article and/or doctoral thesis.\n\n### Article\n\n```bibtex\n@article{GarraldaBarrio2025,\n    title     = {A hybrid metaheuristics–Bayesian optimization framework with safe transfer learning for continuous spark tuning},\n    author    = {Mariano Garralda-Barrio and Carlos Eiras-Franco and Verónica Bolón-Canedo},\n    journal   = {Future Generation Computer Systems},\n    pages     = {108325},\n    year      = {2025},\n    issn      = {0167-739X},\n    doi       = {https://doi.org/10.1016/j.future.2025.108325},\n    publisher = {Elsevier},\n    note      = {Code available at \\url{https://github.com/mgarralda/spark-self-tuning-framework}},\n    keywords  = {Performance modeling, Big data, Machine learning, Apache Spark, Distributed computing}\n}\n```\n\n### Doctoral Thesis\n\n```bibtex\n@phdthesis{GarraldaBarrio2026,\n    author    = {Mariano Garralda Barrio},\n    title     = {AI-Driven Optimization in Distributed Computing Systems: A Self-Tuning Framework},\n    school    = {University of Coruña},\n    year      = {2026},\n    type      = {Doctoral Thesis},\n    url       = {https://hdl.handle.net/2183/48114}\n}\n```\n\n### References\n\n- Garralda-Barrio, M., Eiras-Franco, C., \u0026 Bolón-Canedo, V. (2025).  \n  *A hybrid metaheuristics–Bayesian optimization framework with safe transfer learning for continuous Spark tuning*.  \n  Future Generation Computer Systems.  \n  https://doi.org/10.1016/j.future.2025.108325\n\n- Garralda Barrio, M. (2026).  \n  *AI-Driven Optimization in Distributed Computing Systems: A Self-Tuning Framework*.  \n  Doctoral Thesis, University of Coruña.  \n  https://hdl.handle.net/2183/48114\n\n---\n\n## 📬 Contact\n\nFor questions, collaborations, or feedback, please contact:  \n**Mariano Garralda**  \n[mariano.garralda@udc.es](mailto:mariano.garralda@udc.es)  \nUniversidade da Coruña (UDC)\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmgarralda%2Fspark-self-tuning-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmgarralda%2Fspark-self-tuning-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmgarralda%2Fspark-self-tuning-framework/lists"}