{"id":20106142,"url":"https://github.com/himkt/allennlp-optuna","last_synced_at":"2025-08-03T11:32:27.057Z","repository":{"id":45973038,"uuid":"298173720","full_name":"himkt/allennlp-optuna","owner":"himkt","description":"⚡️ AllenNLP plugin for adding subcommands to use Optuna, making hyperparameter optimization easy","archived":false,"fork":false,"pushed_at":"2021-11-23T12:30:02.000Z","size":347,"stargazers_count":33,"open_issues_count":2,"forks_count":5,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-11-23T16:51:39.560Z","etag":null,"topics":["allennlp","allennlp-subcommand","hyperparameter-optimization","hyperparameters","jsonnet","optuna"],"latest_commit_sha":null,"homepage":"https://allennlp-optuna.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/himkt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-24T05:01:38.000Z","updated_at":"2024-11-02T11:47:54.000Z","dependencies_parsed_at":"2022-09-02T18:41:30.719Z","dependency_job_id":null,"html_url":"https://github.com/himkt/allennlp-optuna","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/himkt%2Fallennlp-optuna","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/himkt%2Fallennlp-optuna/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/himkt%2Fallennlp-optuna/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/himkt%2Fallennlp-optuna/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/himkt","download_url":"https://codeload.github.com/himkt/allennlp-optuna/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228540832,"owners_count":17934030,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["allennlp","allennlp-subcommand","hyperparameter-optimization","hyperparameters","jsonnet","optuna"],"created_at":"2024-11-13T17:49:12.886Z","updated_at":"2024-12-07T00:08:54.413Z","avatar_url":"https://github.com/himkt.png","language":"Python","readme":"# allennlp-optuna: Hyperparameter Optimization Library for AllenNLP\n\n`allennlp-optuna` is [AllenNLP](https://github.com/allenai/allennlp) plugin for\nhyperparameter optimization using [Optuna](https://github.com/optuna/optuna).\n\n\n#### Supported environments\n\nMachine \\ Device | Single GPU             | Multi GPUs\n---------------- | ---------------------- | ---------------\nSingle Node      | :white_check_mark:     | Partial\nMulti Nodes      | :white_check_mark:     | Partial\n\nAllenNLP provides a way of distributed training (https://medium.com/ai2-blog/c4d7c17eb6d6).\nUnfortunately, `allennlp-optuna` doesn't fully support this feature.\nWith multiple GPUs, you can run hyperparameter optimization.\nBut you cannot enable a pruning feature.\n(For more detail, please see [himkt/allennlp-optuna#20](https://github.com/himkt/allennlp-optuna/issues/20)\nand [optuna/optuna#1990](https://github.com/optuna/optuna/issues/1990))\n\nAlternatively, `allennlp-optuna` supports distributed optimization with multiple machines.\nPlease read the [tutorial](https://allennlp-optuna.readthedocs.io/en/latest/tutorial/hyperparameter_optimization_at_scale.html) about\ndistributed optimization in `allennlp-optuna`.\nYou can also learn about a mechanism of Optuna in the [paper](https://arxiv.org/pdf/1907.10902.pdf)\nor [documentation](https://optuna.readthedocs.io/en/stable/).\n\n\n#### Documentation\n\nYou can read the documentation on [readthedocs](https://allennlp-optuna.readthedocs.io/).\n\n\n## 1. Installation\n\n```sh\npip install allennlp_optuna\n\n# Create .allennlp_plugins at the top of your repository or $HOME/.allennlp/plugins\n# For more information, please see https://github.com/allenai/allennlp#plugins\necho 'allennlp_optuna' \u003e\u003e .allennlp_plugins\n```\n\n\n## 2. Optimization\n\n\n### 2.1. AllenNLP config\n\nModel configuration written in Jsonnet.\n\nYou have to replace values of hyperparameters with jsonnet function `std.extVar`.\nRemember casting external variables to desired types by `std.parseInt`, `std.parseJson`.\n\n```jsonnet\nlocal lr = 0.1;  // before\n↓↓↓\nlocal lr = std.parseJson(std.extVar('lr'));  // after\n```\n\nFor more information, please refer to [AllenNLP Guide](https://guide.allennlp.org/hyperparameter-optimization).\n\n\n### 2.2. Define hyperparameter search speaces\n\nYou can define search space in Json.\n\nEach hyperparameter config must have `type` and `keyword`.\nYou can see what parameters are available for each hyperparameter in\n[Optuna API reference](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.trial.Trial.html#optuna.trial.Trial).\n\n```json\n[\n  {\n    \"type\": \"int\",\n    \"attributes\": {\n      \"name\": \"embedding_dim\",\n      \"low\": 64,\n      \"high\": 128\n    }\n  },\n  {\n    \"type\": \"int\",\n    \"attributes\": {\n      \"name\": \"max_filter_size\",\n      \"low\": 2,\n      \"high\": 5\n    }\n  },\n  {\n    \"type\": \"int\",\n    \"attributes\": {\n      \"name\": \"num_filters\",\n      \"low\": 64,\n      \"high\": 256\n    }\n  },\n  {\n    \"type\": \"int\",\n    \"attributes\": {\n      \"name\": \"output_dim\",\n      \"low\": 64,\n      \"high\": 256\n    }\n  },\n  {\n    \"type\": \"float\",\n    \"attributes\": {\n      \"name\": \"dropout\",\n      \"low\": 0.0,\n      \"high\": 0.5\n    }\n  },\n  {\n    \"type\": \"float\",\n    \"attributes\": {\n      \"name\": \"lr\",\n      \"low\": 5e-3,\n      \"high\": 5e-1,\n      \"log\": true\n    }\n  }\n]\n```\n\nParameters for `suggest_#{type}` are available for config of `type=#{type}`. (e.g. when `type=float`,\nyou can see the available parameters in [suggest\\_float](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.trial.Trial.html#optuna.trial.Trial.suggest_float)\n\nPlease see the [example](./config/hparams.json) in detail.\n\n\n### 2.3. Optimize hyperparameters by allennlp cli\n\n\n```shell\nallennlp tune \\\n    config/imdb_optuna.jsonnet \\\n    config/hparams.json \\\n    --serialization-dir result/hpo \\\n    --study-name test\n```\n\nOptionally, you can specify the metrics and direction you are optimizing for:\n\n```shell\nallennlp tune \\\n    config/imdb_optuna.jsonnet \\\n    config/hparams.json \\\n    --serialization-dir result/hpo \\\n    --study-name test \\\n    --metrics best_validation_accuracy \\\n    --direction maximize\n```\n\n### 2.4. [Optional] Specify Optuna configurations\n\nYou can choose a pruner/sample implemented in Optuna.\nTo specify a pruner/sampler, create a JSON config file\n\nThe example of [optuna.json](./config/optuna.json) looks like:\n\n```json\n{\n  \"pruner\": {\n    \"type\": \"HyperbandPruner\",\n    \"attributes\": {\n      \"min_resource\": 1,\n      \"reduction_factor\": 5\n    }\n  },\n  \"sampler\": {\n    \"type\": \"TPESampler\",\n    \"attributes\": {\n      \"n_startup_trials\": 5\n    }\n  }\n}\n```\n\nAnd add a epoch callback to your configuration.\n(https://guide.allennlp.org/hyperparameter-optimization#6)\n\n```\n  callbacks: [\n    {\n      type: 'optuna_pruner',\n    }\n  ],\n```\n\n- [`config/imdb_optuna.jsonnet`](./config/imdb_optuna.jsonnet) is a simple configuration for allennlp-optuna\n- [`config/imdb_optuna_with_pruning.jsonnet`](./config/imdb_optuna_with_pruning.jsonnet) is a configuration using Optuna pruner (and TPEsampler)\n\n```sh\n$ diff config/imdb_optuna.jsonnet config/imdb_optuna_with_pruning.jsonnet\n32d31\n\u003c   datasets_for_vocab_creation: ['train'],\n58a58,62\n\u003e     callbacks: [\n\u003e       {\n\u003e         type: 'optuna_pruner',\n\u003e       }\n\u003e     ],\n```\n\nThen, you can use a pruning callback by running following:\n\n```shell\nallennlp tune \\\n    config/imdb_optuna_with_pruning.jsonnet \\\n    config/hparams.json \\\n    --optuna-param-path config/optuna.json \\\n    --serialization-dir result/hpo_with_optuna_config \\\n    --study-name test_with_pruning\n```\n\n\n\n## 3. Get best hyperparameters\n\n```shell\nallennlp best-params \\\n    --study-name test\n```\n\n\n## 4. Retrain a model with optimized hyperparameters\n\n```shell\nallennlp retrain \\\n    config/imdb_optuna.jsonnet \\\n    --serialization-dir retrain_result \\\n    --study-name test\n```\n\n\n## 5. Hyperparameter optimization at scale!\n\nyou can run optimizations in parallel.\nYou can easily run distributed optimization by adding an option\n`--skip-if-exists` to `allennlp tune` command.\n\n```\nallennlp tune \\\n    config/imdb_optuna.jsonnet \\\n    config/hparams.json \\\n    --optuna-param-path config/optuna.json \\\n    --serialization-dir result \\\n    --study-name test \\\n    --skip-if-exists\n```\n\nallennlp-optuna uses SQLite as a default storage for storing results.\nYou can easily run distributed optimization **over machines**\nby using MySQL or PostgreSQL as a storage.\n\nFor example, if you want to use MySQL as a storage,\nthe command should be like following:\n\n```\nallennlp tune \\\n    config/imdb_optuna.jsonnet \\\n    config/hparams.json \\\n    --optuna-param-path config/optuna.json \\\n    --serialization-dir result \\\n    --study-name test \\\n    --storage mysql://\u003cuser_name\u003e:\u003cpasswd\u003e@\u003cdb_host\u003e/\u003cdb_name\u003e \\\n    --skip-if-exists\n```\n\nYou can run the above command on each machine to\nrun multi-node distributed optimization.\n\nIf you want to know about a mechanism of Optuna distributed optimization,\nplease see the official documentation: https://optuna.readthedocs.io/en/latest/tutorial/10_key_features/004_distributed.html\n\n\n#### Reference\n\n- Cookpad Techlife (in Japanese): https://techlife.cookpad.com/entry/2020/11/06/110000\n  - `allennlp-optuna` is used for optimizing hyperparameter of NER model\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhimkt%2Fallennlp-optuna","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhimkt%2Fallennlp-optuna","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhimkt%2Fallennlp-optuna/lists"}