{"id":23046353,"url":"https://github.com/pfecht/bert-exploration","last_synced_at":"2026-01-12T07:26:52.291Z","repository":{"id":93992998,"uuid":"161790970","full_name":"pfecht/bert-exploration","owner":"pfecht","description":null,"archived":false,"fork":false,"pushed_at":"2018-12-16T12:02:14.000Z","size":1,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-03T02:22:25.930Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pfecht.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-14T13:51:40.000Z","updated_at":"2018-12-16T12:02:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"ffd99ce3-2605-4a43-82ee-9e2b63aeccaa","html_url":"https://github.com/pfecht/bert-exploration","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pfecht/bert-exploration","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pfecht%2Fbert-exploration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pfecht%2Fbert-exploration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pfecht%2Fbert-exploration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pfecht%2Fbert-exploration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pfecht","download_url":"https://codeload.github.com/pfecht/bert-exploration/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pfecht%2Fbert-exploration/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28336510,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T06:09:07.588Z","status":"ssl_error","status_checked_at":"2026-01-12T06:05:18.301Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-15T21:31:40.729Z","updated_at":"2026-01-12T07:26:52.275Z","avatar_url":"https://github.com/pfecht.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# BERT exploration\n\nWorkspace to explore classification and other tasks based on the [pytorch implementation](https://github.com/huggingface/pytorch-pretrained-BERT) of the original bert paper (https://arxiv.org/abs/1810.04805)\n\n\n# GLUE \nSee [this notebook](https://colab.research.google.com/drive/1Qc0JOJ3x4vUU3nNTtBoD5GONrrrOtNdr) for an implementation of the GLUE tasks.\n\n\n# SWAG\nThe Situations With Adversarial Generations (SWAG) dataset contains 113k sentence-pair com- pletion examples that evaluate grounded common- sense inference (Zellers et al., 2018). Given a sentence from a video captioning\ndataset, the task is to decide among four choices the most plausible continuation. \n\nRunning\n\n```bash \nexport SWAG_DIR=/home/pfecht/thesis/swagaf\n\npython run_swag.py \\\n  --bert_model bert-base-uncased \\\n  --do_train \\\n  --do_eval \\\n  --data_dir $SWAG_DIR/data \\\n  --train_batch_size 16 \\\n  --learning_rate 2e-5 \\\n  --num_train_epochs 3.0 \\\n  --max_seq_length 80 \\\n  --output_dir /home/pfecht/tmp/swag_output/  \\\n  --gradient_accumulation_steps 4\n```\n\nresults in\n\n\n* **Accuracy**: **78.58** (BERT paper **81.6**)\n\n```\n12/14/2018 18:42:18 - INFO - __main__ -     eval_accuracy = 0.7858642407277817\n12/14/2018 18:42:18 - INFO - __main__ -     eval_loss = 0.6655298910721517\n12/14/2018 18:42:18 - INFO - __main__ -     global_step = 13788\n12/14/2018 18:42:18 - INFO - __main__ -     loss = 0.07108418613090857\n```\n\nwith fine-tuning time on a single GPU (GeForce GTX TITAN X): **around 4 hours**.\n\n\n# SQuAD\n\u003e see https://github.com/huggingface/pytorch-pretrained-BERT#fine-tuning-with-bert-running-the-examples\n\n\nRunning\n\n```bash\n$ python run_squad.py \\\n  --bert_model bert-base-uncased \\\n  --do_train \\\n  --do_predict \\\n  --train_file $SQUAD_DIR/train-v1.1.json \\\n  --predict_file $SQUAD_DIR/dev-v1.1.json \\\n  --train_batch_size 12 \\\n  --learning_rate 3e-5 \\\n  --num_train_epochs 2.0 \\\n  --max_seq_length 384 \\\n  --doc_stride 128 \\\n  --output_dir $OUT_DIR \\\n  --optimize_on_cpu\n```\n\n* `optimize_on_CPU` is important to obtain enough space on the GPU for training. BertOptimiezer stores 2 moving averages of the weights of the model wich means We have to store 3-times the size of the model in the GPU if we don't move it to CPU.\n* OOM errors are proportional to `train_batch_size` and `max_seq_length`.\n\nresults in \n\n* **F1 score**: 88.28 (BERT paper 88.5)\n* **EM (Exact match)**: 81.05 (BERT paper = 80.8)\n\nwith fine-tuning time on a single GPU (GeForce GTX TITAN X): **around 8 hours**.\n\nrunning evaluation based on\n\n```json\n$ python evaluate-v1.1.py /home/pfecht/res/SQUAD/dev-v1.1.json predictions.json\n{\"f1\": 88.28409344840951, \"exact_match\": 81.05014191106906}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpfecht%2Fbert-exploration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpfecht%2Fbert-exploration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpfecht%2Fbert-exploration/lists"}