{"id":28471627,"url":"https://github.com/openml/openml-aslib","last_synced_at":"2025-07-01T22:31:55.362Z","repository":{"id":66771321,"uuid":"127681597","full_name":"openml/openml-aslib","owner":"openml","description":"Turns an OpenML study into an ASLib scenario","archived":false,"fork":false,"pushed_at":"2018-05-10T20:54:28.000Z","size":21,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-06-07T11:08:04.618Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-04-02T00:07:02.000Z","updated_at":"2018-05-10T18:55:03.000Z","dependencies_parsed_at":"2023-02-23T17:45:10.634Z","dependency_job_id":null,"html_url":"https://github.com/openml/openml-aslib","commit_stats":{"total_commits":13,"total_committers":1,"mean_commits":13.0,"dds":0.0,"last_synced_commit":"5b98f2e4658f17555c2d01a9b88571fe9dfa0027"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/openml/openml-aslib","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openml%2Fopenml-aslib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openml%2Fopenml-aslib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openml%2Fopenml-aslib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openml%2Fopenml-aslib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openml","download_url":"https://codeload.github.com/openml/openml-aslib/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openml%2Fopenml-aslib/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263046427,"owners_count":23405186,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-07T11:08:08.117Z","updated_at":"2025-07-01T22:31:55.342Z","avatar_url":"https://github.com/openml.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenML-ASLib\n\nLibrary that converts a set of experiments from OpenML into an ASLib scenario. The experiments are defined by either\n* A study id\n* A set of tasks and setup ids\n* A benchmark suite and a set of scikit-learn classifiers\n\nThe remainder of this tutorial requires knowledge about basic OpenML components, such as datasets, tasks, flows and setups.\nFor an introduction to these concepts in the context of OpenML, please read this [blog post](https://medium.com/open-machine-learning/basic-components-of-openml-a5745634c664). \n\n## How to generate an ASLib scenario\nThere are several ways of using OpenML to generate new ASLib scenario's.\n\n### Use an existing study\nAn OpenML study is a collection of datasets, tasks, flows, setups and runs.\nStudies bundle these components in a single URL, convenient for attaching to a paper.\nFor example, [this study](https://www.openml.org/s/34) contains all algorithm runs that were used for the [OpenML-WEKA-2017 scenario](https://github.com/coseal/aslib_data/tree/master/OPENML-WEKA-2017). \n\nThe OpenML-ASLib extension will take all registered tasks and setups from this study, and will try to find all corresponding runs on this cross-product on OpenML. \nNote that it is suggested to have the complete grid of algorithm performances (i.e., all setups should have been ran on all tasks).\n\nRun example:\n`python examples/generate_from_study.py --measure predictive_accuracy --study_id 34`\nThis generates a copy of the OpenML-WEKA-2017 scenario.\n\nAlternatively, if the task ids and setup ids are already known (which requires some moderate OpenML skills) it is also possible to generate a scenario using these:\n`python examples/generate.py --measure predictive_accuracy --task_ids 1701 1702 1705 --setup_ids 2361 2362`\n\n\n### Generate runs on the fly\n\nThe example script `generate_from_sklearn.py` shows how to run several scikit-learn on an benchmark suite (= all tasks in an OpenML study) and create a ASLib scenario from this. \nIn the [current example](https://github.com/openml/openml-aslib/blob/master/examples/generate_by_sklearn.py), a [Decision Stump](http://scikit-learn.org/stable/modules/tree.html) and [Naive Bayes](http://scikit-learn.org/stable/modules/naive_bayes.html) are ran on all tasks in the [OpenML-CC18](https://arxiv.org/abs/1708.03731), a curated benchmark suite. \nHowever, the script can be easily adapted to run on any benchmark suite or combination of scikit-learn classifiers. \nWhen the result of a given setup/task is already present on OpenML, the script will skip this combination and reuse the earlier result. \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenml%2Fopenml-aslib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenml%2Fopenml-aslib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenml%2Fopenml-aslib/lists"}