{"id":18398893,"url":"https://github.com/lyeoni/korquad","last_synced_at":"2025-08-04T23:17:07.562Z","repository":{"id":196956105,"uuid":"190349058","full_name":"lyeoni/KorQuAD","owner":"lyeoni","description":"KorQuAD (Korean Question Answering Dataset) submission guide using PyTorch pretrained BERT","archived":false,"fork":false,"pushed_at":"2019-06-18T01:56:05.000Z","size":1414,"stargazers_count":31,"open_issues_count":0,"forks_count":9,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-04-07T05:35:45.284Z","etag":null,"topics":["bert","korquad","machine-reading-comprehension","pytorch","question-answering"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lyeoni.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-06-05T07:42:22.000Z","updated_at":"2024-04-14T07:18:12.000Z","dependencies_parsed_at":"2023-09-28T11:47:29.383Z","dependency_job_id":null,"html_url":"https://github.com/lyeoni/KorQuAD","commit_stats":null,"previous_names":["lyeoni/korquad"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lyeoni/KorQuAD","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyeoni%2FKorQuAD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyeoni%2FKorQuAD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyeoni%2FKorQuAD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyeoni%2FKorQuAD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lyeoni","download_url":"https://codeload.github.com/lyeoni/KorQuAD/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyeoni%2FKorQuAD/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268801923,"owners_count":24309642,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-04T02:00:09.867Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","korquad","machine-reading-comprehension","pytorch","question-answering"],"created_at":"2024-11-06T02:24:53.400Z","updated_at":"2025-08-04T23:17:07.535Z","avatar_url":"https://github.com/lyeoni.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KorQuAD\n[huggingface/pytorch-pretrained-BERT](https://github.com/huggingface/pytorch-pretrained-BERT) repository contains op-for-op PyTorch reimplementations, pre-trained models and fine-tuning examples for Google's BERT model.\nAnd as a result of submission using the run_squad.py code provided by [huggingface/pytorch-pretrained-BERT](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py) repository, it ranked 30th in the test set with EM= 71.47, F1= 89.71, respectively, as shown below. (2019.06.17)\n\n\u003cp align=\"center\"\u003e\n\u003cimg height=50 src=\"https://github.com/lyeoni/KorQuAD/blob/master/images/submission-3.png\" /\u003e\n\u003c/p\u003e\n\nA trained BERT model is publicly available. So, I'm going to cover **the process of submitting** a model and result for official evaluation on KorQuAD. Once your model has been evaluated officially, your scores will be added to the [leaderboard](https://korquad.github.io/). Thus I would assume you already completed model training in KorQuAD, and have a trained model archive.\n\n## Overview\nA pre-trained language model, **BERT**, is publicly available. For KorQuAD submission, what you have to do is to **fine-tune** the pre-trained BERT model on KorQuAD. And fine-tuning can be done simply by running `run_squad.py` (in [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py)) on the KorQuAD dataset.\n\nAnd, even if you fine-tune BERT with the default hyper-parameters in [`run_squad.py`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py), we can get the following results (It's a score that can be ranked in the 30th grade, based on June 2019):\n```\n$ python evaluate-v1.0.py KorQuAD_v1.0_dev.json predictions.json\n{\"exact_match\": 70.29788708001385, \"f1\": 90.08062112089534}\n```\n\n## BERT Fine-tuning\nBelow table shows the results of the BERT models fine-tuned with various hyper-parameters.\n\n- Model : model name with detailed description\n- #steps : number of optimization steps in training\n- EM (Exact Match) : ratio of accurate prediction to actualanswer text\n- F1 : score on how the actual answer text overlaps with the prediction\n\n|Model|#stpes|EM(dev)|F1(dev)|\n|---|---:|---:|---:|\n|**_change maximum sequence length_**||||\n|BERT-Multilingual - _baseline_ ([description](https://github.com/lyeoni/KorQuAD/blob/master/logs/bert-base-multilingual-cased-default.txt))|7,587|70.298|90.081|\n|BERT-Multilingual - _baseline_ (+ max_seq_length=512) ([description](https://github.com/lyeoni/KorQuAD/blob/master/logs/bert-base-multilingual-cased-1.txt))|6,336|70.80|90.104|\n|**_change train batch size_**||||\n|BERT-Multilingual - _baseline_ (+ train_batch_size=16) ([description](https://github.com/lyeoni/KorQuAD/blob/master/logs/bert-base-multilingual-cased-2.txt))|15,174|70.159|89.818|\n|BERT-Multilingual - _baseline_ (+ max_seq_length=512, train_batch_size=16) ([description](https://github.com/lyeoni/KorQuAD/blob/master/logs/bert-base-multilingual-cased-3.txt))|12,669|69.830|89.407|\n|**_change learning-rate_**||||\n|BERT-Multilingual - _baseline_ (+ learning_rate=3e-5) ([description](https://github.com/lyeoni/KorQuAD/blob/master/logs/bert-base-multilingual-cased-4.txt))|7,587|70.419|90.241|\n|BERT-Multilingual - _baseline_ (+ learning_rate=3e-5, train_batch_size=16) ([description](https://github.com/lyeoni/KorQuAD/blob/master/logs/bert-base-multilingual-cased-5.txt))|15,174|70.229|90.114|\n|BERT-Multilingual - _baseline_ (+ max_seq_length=512, learning_rate=3e-5)|6,336|**70.644**|**90.246**|\n|BERT-Multilingual - _baseline_ (+ max_seq_length=512, learning_rate=3e-5, train_batch_size=16)|12,669|70.419|90.179|\n\n\nTips for hyper-parameter tuning\n- `pre-trained model` : `bert-large-cased`, pre-trained model, is not recommended\n- `num_train_epochs` : set to default value\n- `learning rate` : the smaller, the better\n\n## Submission Process\nTo get official scores on the KorQuAD test set, we must submit(or upload) our model to the [CodaLab](https://worksheets.codalab.org/). This is because the integrity of test results should be preserved.\n\nHere's a detailed process that guides you through the official evaluation of our model.\n\n### 0. Create a CodaLab account\nYou can create a CodaLab account [here](https://worksheets.codalab.org/).\nClick `sign up` in the top-right corner of the CodaLab homepage.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/lyeoni/KorQuAD/blob/master/images/codalab_signup.png\" /\u003e\n\u003c/p\u003e\n\n\u003cbr\u003e\n\n### 1. Create a CodaLab worksheet\nClick the `My dashboard` in the top-right corner.\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/lyeoni/KorQuAD/blob/master/images/codalab_dashboard.png\" /\u003e\n\u003c/p\u003e\n\nClick the `New Worksheet` in the upper-right corner and name your worksheet.\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/lyeoni/KorQuAD/blob/master/images/codalab_worksheet.png\" /\u003e\n\u003c/p\u003e\n\n\u003cbr\u003e\n\n### 2. Upload trained model with source code\nBegin by uploading archive for the trained model onto Codalab.\n\nFor example, I trained (BERT) model using [`run_squad.py`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py) code from [huggingface/pytorch-pretrained-BERT repository](https://github.com/huggingface/pytorch-pretrained-BERT), and the archive of trained model consists of the following:\n- run_squad.py : python script to generate the predictions\n- vocab.txt : vocabulary file\n- config.json : a configuration file for the model\n- pytorch_model.bin : a PyTorch dump of a trained BERT model (saved with the usual torch.save())\n\nCodaLab requires us that the prediction python script should run with the following arguments. Therefore, if your python script does not run with the following arguments, the script needs to be modified.\n\n```\nCodaLab\u003e python \u003cpath-to-prediction-python-script\u003e \u003cinput-data-json-file\u003e \u003coutput-prediction-json-file\u003e\n```\n\nIf you use [`run_squad.py`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py) code from [huggingface/pytorch-pretrained-BERT repository](https://github.com/huggingface/pytorch-pretrained-BERT) for training, and have difficulty modifying this to the above format, just use `run_squad_for_submission.py` in this repository. It runs with the arguments required by CodaLab, and generates predictions. And it should be in the same place as the archive of a trained model (vocab.txt, config.json, pytorch_model.bin).\n\n\u003cbr\u003e\n\n### 3. Run your trained model on the dev set\nCopy the dev data to the worksheet by using the following command into the CodaLab terminal. Do not upload the dev data directly!\n\n```\nCodaLab\u003e cl add bundle korquad-data//KorQuAD_v1.0_dev.json .\n```\n\nThen run the command for the python script generates the predictions on the dev data. Be sure to replace `\u003cpath-to-prediction-python-script\u003e` with your actual program path.\n```\nCodaLab\u003e cl run :KorQuAD_v1.0_dev.json :\u003cpath-to-prediction-python-script\u003e \"python \u003cpath-to-prediction-python-script\u003e KorQuAD_v1.0_dev.json predictions.json\" -n run-predictions\n```\n#### Running jobs that use GPUs\nIf you run the following command on the default Docker image(codalab/ubuntu:1.9), the GPU is not available because the image does not support CUDA. This can be checked as follows.\n```\nCodaLab\u003e cl run :check_cuda.py \"python check_cuda.py\"\n\n# stdout\u003e\n# ('Python Version: ', '2.7.12')\n# ('PyTorch Version: ', '0.4.1')\n# ('CUDA Available: ', False)\n# ('Device Count: ', 0)\n```\n\nTo enable GPU acceleration, we need to include the `--request-gpus` flag like belows.\n```\nCodaLab\u003e cl run :check_cuda.py \"python check_cuda.py\" --request-gpus 1\n\n# stdout\u003e\n# ('Python Version: ', '2.7.12')\n# ('PyTorch Version: ', '1.0.1.post2')\n# ('CUDA Available: ', True)\n# ('Device Count: ', 1L)\n# ('Current Device: ', 0L)\n# ('Device Name', u'Tesla M60')\n```\n\ncf. You can also increase the GPU memory by using `--request-memory` flag, because default GPU memory size (2GB) is too small.\n```\nCodaLab\u003e cl run :check_cuda.py \"python check_cuda.py\" --request-gpus 1 --request-memory 11g\n```\n\n\nNow, all jobs (e.g., prediction) can be conducted on a GPU.\n\n#### Specifying Environments with Docker\nCodaLab uses Docker containers to define the environment of a run bundle. Each Docker container is based on a Docker image, which specifies the full environment, including which Linux kernel version, which libraries, etc. The default Docekr image is `codalab/ubuntu:1.9`, which consists of Ubuntu 14.04 plus some standard packages (e.g., Python).\n\nIn CodaLab, when you create a run, you can specify which Docker container you want to use like below.\n```\nCodaLab\u003e cl run \u003ccommand\u003e --request-docker-image \u003cdocker-image-to-use\u003e\n```\n\n**Here we use [lyeoni/pytorch_pretrained_bert](https://cloud.docker.com/repository/docker/lyeoni/pytorch_pretrained_bert) docker image that contains PyTorch 1.0.0 (with CUDA 9.0) and pre-trained BERT models re-implemented by PyTorch.**\n\n[lyeoni/pytorch_pretrained_bert](https://cloud.docker.com/repository/docker/lyeoni/pytorch_pretrained_bert) image is based on 2 repository, [anibali's PyTorch docker image](https://github.com/anibali/docker-pytorch) that contains various CUDA-enabled version of the PyTorch image, and [PyTorch Pretrained BERT](https://github.com/huggingface/pytorch-pretrained-BERT) that contains PyTorch reimplementations, pre-trained models for Google's BERT model.\n\nlyeoni/pytorch_pretrained_bert docker image makes both CUDA-enabled PyTorch and pre-trained BERT model easy to use in CodaLab.\n\n```\nCodaLab\u003e cl run :check_cuda.py \"python check_cuda.py\" --request-docker-image lyeoni/pytorch_pretrained_bert --request-gpus 1\n\n# Python Version:  3.6.5\n# PyTorch Version:  1.0.0\n# CUDA Available:  True\n# Device Count:  1\n# Current Device:  0\n# Device Name Tesla K80\n```\n\n**The final command to generate the predictions on the dev set is like belows.**\n```\nCodaLab\u003e cl run :config.json :vocab.txt :pytorch_model.bin :KorQuAD_v1.0_dev.json :run_squad.py \"python run_squad.py KorQuAD_v1.0_dev.json predictions.json\" -n run-predictions --request-docker-image lyeoni/pytorch_pretrained_bert --request-gpus 1 --request-memory 11g\n```\n\n\u003cbr\u003e\n\n### 4. Evaluate the predictions on the dev set\nIf the bundle state is **ready** not failed, we extract out the predictions file into a bundle of its own. Let's do this as follows:\n```\n# MODELNAME should not contain spaces (avoid using special characters too).\nCodaLab\u003e cl make run-predictions/predictions.json -n predictions-{MODELNAME}\n```\n\nNow, let's verify that we can evaluate the predictions on the dev set.\n```\nCodaLab\u003e cl macro korquad-utils/dev-evaluate-v1.0 predictions-{MODELNAME}\n```\n\nOnce this succeeds, you should see the scores for your model appended to the worksheet.\n\n\u003cbr\u003e\n\n### 5. Submission\nFollow the submission guide provided in the official site. This step requires only a formatted descriptions(e.g., model name, your name, institution, etc.) to add an official leaderboard.\n\n- [KorQuAD Submission Guide (English Ver.)](https://worksheets.codalab.org/worksheets/0xee93409b823f436d88741ab115403559/)\n- [KorQuAD Submission Guide (Korean Ver.)](https://worksheets.codalab.org/worksheets/0x7b06f2ebd0584748a3a281018e7d19b0/)\n\n## Appendix\n- In CodaLab, each run has a state, which evolves through the following values:\n  - `created` : initial state\n  - `staged` : for run bundles, meaning dependencies are ready\n  - `preparing` : launch a worker just for this run, waiting for it\n  - `running` : a worker is running the command\n  - `ready/failed` : terminal states corresponding to a successful or unsuccessful run\n\n## References\n- [LG CNS AI Research Team] [KorQuAD, The Korean Question Answering Dataset](https://korquad.github.io/)\n- [LG CNS AI Research Team] [KorQuAD Submission Guide (English Ver.)](https://worksheets.codalab.org/worksheets/0xee93409b823f436d88741ab115403559/)\n- [LG CNS AI Research Team] [KorQuAD Submission Guide (Korean Ver.)](https://worksheets.codalab.org/worksheets/0x7b06f2ebd0584748a3a281018e7d19b0/)\n- [CodaLab] [CodaLab, A collaborative platform for reproducible research](https://worksheets.codalab.org/)\n- [codalab/codalab-worksheets] [Execution](https://github.com/codalab/codalab-worksheets/wiki/Execution)\n- [codalab/codalab-worksheets] [CLI Reference](https://github.com/codalab/codalab-worksheets/wiki/CLI-Reference)\n- [anibali/docker-pytorch] [PyTorch Docker image](https://github.com/anibali/docker-pytorch)\n- [huggingface/pytorch-pretrained-BERT] [PyTorch Pretrained BERT: The Big \u0026 Extending Repository of pretrained Transformers](https://github.com/huggingface/pytorch-pretrained-BERT)\n- [huggingface/pytorch-pretrained-BERT] [Fixing issue \"Training beyond specified 't_total' steps with schedule 'warmup_linear'\" reported in #556 #604](https://github.com/huggingface/pytorch-pretrained-BERT/pull/604)\n- [lyeoni/pytorch_pretrained_bert] [Docker image with pyotrch and pretrained_bert](https://cloud.docker.com/repository/docker/lyeoni/pytorch_pretrained_bert)\n- [pbaumgartner/pytorch-bert] [pytorch-bert-image](https://hub.docker.com/r/pbaumgartner/pytorch-bert/dockerfile)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flyeoni%2Fkorquad","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flyeoni%2Fkorquad","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flyeoni%2Fkorquad/lists"}