{"id":15973702,"url":"https://github.com/c-bata/signate-978-aiaccel-optimizer","last_synced_at":"2025-04-04T16:21:17.421Z","repository":{"id":187340743,"uuid":"621272876","full_name":"c-bata/signate-978-aiaccel-optimizer","owner":"c-bata","description":"A solution (aiaccel's optimizer) for the SIGNATE HPO competition","archived":false,"fork":false,"pushed_at":"2023-04-03T03:32:39.000Z","size":806,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-10T01:46:17.329Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/c-bata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-03-30T10:33:06.000Z","updated_at":"2023-04-03T03:22:25.000Z","dependencies_parsed_at":"2023-08-09T23:46:30.225Z","dependency_job_id":null,"html_url":"https://github.com/c-bata/signate-978-aiaccel-optimizer","commit_stats":null,"previous_names":["c-bata/signate-978-aiaccel-optimizer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-bata%2Fsignate-978-aiaccel-optimizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-bata%2Fsignate-978-aiaccel-optimizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-bata%2Fsignate-978-aiaccel-optimizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-bata%2Fsignate-978-aiaccel-optimizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/c-bata","download_url":"https://codeload.github.com/c-bata/signate-978-aiaccel-optimizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247208404,"owners_count":20901570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-07T21:06:31.875Z","updated_at":"2025-04-04T16:21:17.398Z","avatar_url":"https://github.com/c-bata.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# signate-978-aiaccel-optimizer\n\nA solution ([aiaccel](https://github.com/aistairc/aiaccel) optimizer) for the SIGNATE HPO competition.\n\n## Solution\n\n### `optimizer_final_selection`\n\nThis folder contains the source code that I selected as the final submission.\nIt performed worse on the private leaderboard though.\nThe main idea of this method is as follows:\n\n1. Select a trust region, which is expected to contain promising solutions based on past trial results, and sample from this subspace.\n   The trust region is centered around the parameters that produced the best evaluation value and is calculated with a width of `trust_region_width = (upper_bound - lower_bound) * max((50 - trial_number) / 100, 0.01)`.\n   The basic idea comes from TuRBO, which samples several parameters from within the trust region and expands (explores) it if the best evaluation value improves, and shrinks (exploits) it if it does not improve.\n   We designed this algorithm to focus more on exploitation (shrinking) towards the end of the search when the number of evaluations is fixed (in this competition, it was 100).\n2. Use Sobol quasi-random sequences instead of random sampling for initial sampling. The low-discrepancy property of Sobol sequences reduces the bias of initial search points and improves search performance stability.\n3. Increase robustness to unknown objective functions by alternating between multiple optimization methods with different strengths for solving different types of problems.\n\n### `optimizer_fanova_trust_region`\n\nThis folder contains the source code for an algorithm that did not perform well on the public leaderboard, but showed promising results on local benchmarking.\n\nThis method narrows down the trust region of high-scoring hyperparameters according to the fANOVA score, which intuitively results in an effect similar to step-wise tuning.\nStep-wise tuning, as explained in the following blog on Optuna's LightGBMTuner, tunes the best hyperparameters in order of importance.\nThis allows for an early and significant reduction of the search space, enabling the discovery of better hyperparameters with fewer evaluations.\n\n\u003e LightGBM Tuner: New Optuna Integration for Hyperparameter Optimization\n\u003e https://medium.com/optuna/lightgbm-tuner-new-optuna-integration-for-hyperparameter-optimization-8b7095e99258\n\nAs further improvement of the algorithm, the fANOVA score is calculated in two stages to build a trust region.\n\n* In the first stage, the fANOVA score is calculated using all past trial results to determine the trust region.\n* In the second stage, the fANOVA score is calculated using only the top-performing trials to further narrow down the trust region.\n\nThis is because the fANOVA score only scores the contribution of each parameter to the objective value, so a parameter with a high fANOVA score, such as the learning rate, can unfairly lower the score of other hyperparameters.\nThus, in the second stage, the top-N trials are used to calculate the fANOVA score with these parameters fixed, and the trust region is determined accordingly.\n\n## Tools\n\nDuring the competition, I created following tools to smoothly evaluate the performance and analyse the algorithm's behavior.\n\n### `train_surrogate_evaluator.py`\n\nThis script trains a surrogate model of the objective function from multiple aiaccel execution results (`reports.csv`) using `RandomForest`.\nBy using the model to evaluate the algorithm, the evaluation of the algorithm can be done very quickly.\n\n### `tools/optuna_dashboard.py`\n\nA script to create an Optuna Study from the execution results of aiaccel (reports.csv) and visualize/analyze it using Optuna Dashboard.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc-bata%2Fsignate-978-aiaccel-optimizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fc-bata%2Fsignate-978-aiaccel-optimizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc-bata%2Fsignate-978-aiaccel-optimizer/lists"}