{"id":47755733,"url":"https://github.com/marcelovca90/auto-ml-evaluation","last_synced_at":"2026-04-03T04:17:30.626Z","repository":{"id":121008489,"uuid":"568866288","full_name":"marcelovca90/auto-ml-evaluation","owner":"marcelovca90","description":"Code of the article \"A practical evaluation of AutoML tools for binary, multiclass, and multilabel classification\".","archived":false,"fork":false,"pushed_at":"2025-05-15T18:25:19.000Z","size":70560,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-15T19:34:54.867Z","etag":null,"topics":["automl","classification","hyperparameter-optimization","machine-learning","neural-architecture-search"],"latest_commit_sha":null,"homepage":"https://www.researchsquare.com/article/rs-4172933/latest","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marcelovca90.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-11-21T15:21:07.000Z","updated_at":"2025-05-15T18:25:23.000Z","dependencies_parsed_at":null,"dependency_job_id":"f743b38c-0a68-4d20-a79b-cbb463356fb1","html_url":"https://github.com/marcelovca90/auto-ml-evaluation","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/marcelovca90/auto-ml-evaluation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelovca90%2Fauto-ml-evaluation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelovca90%2Fauto-ml-evaluation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelovca90%2Fauto-ml-evaluation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelovca90%2Fauto-ml-evaluation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marcelovca90","download_url":"https://codeload.github.com/marcelovca90/auto-ml-evaluation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelovca90%2Fauto-ml-evaluation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31333234,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-03T03:20:36.090Z","status":"ssl_error","status_checked_at":"2026-04-03T03:20:35.133Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automl","classification","hyperparameter-optimization","machine-learning","neural-architecture-search"],"created_at":"2026-04-03T04:17:29.925Z","updated_at":"2026-04-03T04:17:30.611Z","avatar_url":"https://github.com/marcelovca90.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A Practical Evaluation of AutoML Tools for Binary, Multiclass, and Multilabel Classification\n\nAuthors: *Marcelo V. C. Aragão, Augusto G. Afonso, Rafaela C. Ferraz, Rairon G. Ferreira, Sávio G. Leite, Felipe A. P. de Figueiredo, and Samuel B. Mafra.*\n\n## Abstract:\n    Selecting the most suitable Automated Machine Learning (AutoML) tool is pivotal for\n    achieving optimal performance in diverse classification tasks, including binary,\n    multiclass, and multilabel scenarios. The wide range of frameworks with distinct\n    features and capabilities complicates this decision, necessitating a systematic\n    evaluation. This study benchmarks sixteen AutoML tools, including AutoGluon,\n    AutoSklearn, TPOT, PyCaret, and Lightwood, across all three classification types\n    using twenty-one real-world datasets. Unlike prior studies focusing on a subset of\n    classification tasks or a limited number of tools, we provide a unified evaluation\n    of sixteen frameworks, incorporating feature-based comparisons, time-constrained\n    experiments, and multi-tier statistical validation. A key contribution of our study\n    is the in-depth assessment of multilabel classification, exploring both native and\n    label-powerset representations and revealing that several tools lack robust\n    multilabel capabilities. Our findings demonstrate that AutoSklearn excels in\n    predictive performance for binary and multiclass settings, albeit at longer training\n    times, while Lightwood and AutoKeras offer faster training at the cost of predictive\n    performance on complex datasets. AutoGluon emerges as the best overall solution,\n    balancing predictive accuracy with computational efficiency. Our statistical\n    analysis – at per-dataset, across-datasets, and all-datasets levels – confirms\n    significant performance differences among tools, highlighting accuracy-speed\n    trade-offs in AutoML. These insights underscore the importance of aligning tool\n    selection with specific problem characteristics and resource constraints. The\n    open-source code and reproducible experimental protocols further ensure the study’s\n    value as a robust resource for researchers and practitioners.\n\n## Setup and Execution:\nThe tests require a need a Linux installation (bare-metal or virtualized).\n\n```\ngit clone https://github.com/marcelovca90/auto-ml-evaluation.git\ncd auto-ml-evaluation\nconda create -n auto-ml-evaluation python=3.8\nconda activate auto-ml-evaluation\nchmod +x run.sh\n./run.sh\n```\n\nNote: if you want to use Label Powerset, make sure to set `LABEL_POWERSET = True` in `common.py`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarcelovca90%2Fauto-ml-evaluation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarcelovca90%2Fauto-ml-evaluation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarcelovca90%2Fauto-ml-evaluation/lists"}