{"id":27966088,"url":"https://github.com/yuanxiaosc/deep_dynamic_contextualized_word_representation","last_synced_at":"2025-05-07T20:16:35.230Z","repository":{"id":97827351,"uuid":"157370657","full_name":"yuanxiaosc/Deep_dynamic_contextualized_word_representation","owner":"yuanxiaosc","description":"TensorFlow code and pre-trained models for A Dynamic Word Representation Model Based on Deep Context. It combines the idea of  BERT model and ELMo's deep context word representation.","archived":false,"fork":false,"pushed_at":"2018-12-27T13:08:35.000Z","size":74,"stargazers_count":15,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-07T20:16:30.519Z","etag":null,"topics":["bert","elmo","nlp","transformer"],"latest_commit_sha":null,"homepage":"https://yuanxiaosc.github.io/2018/11/27/Bidirectional_Encoder_Representations_Transformers/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yuanxiaosc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-11-13T11:37:46.000Z","updated_at":"2025-01-17T12:53:40.000Z","dependencies_parsed_at":"2024-01-14T02:37:23.273Z","dependency_job_id":"46fed977-4f98-4f3f-9a24-584ae634b2ea","html_url":"https://github.com/yuanxiaosc/Deep_dynamic_contextualized_word_representation","commit_stats":null,"previous_names":["yuanxiaosc/deep_dynamic_word_representation"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanxiaosc%2FDeep_dynamic_contextualized_word_representation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanxiaosc%2FDeep_dynamic_contextualized_word_representation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanxiaosc%2FDeep_dynamic_contextualized_word_representation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanxiaosc%2FDeep_dynamic_contextualized_word_representation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yuanxiaosc","download_url":"https://codeload.github.com/yuanxiaosc/Deep_dynamic_contextualized_word_representation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252949253,"owners_count":21830154,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","elmo","nlp","transformer"],"created_at":"2025-05-07T20:16:34.580Z","updated_at":"2025-05-07T20:16:35.210Z","avatar_url":"https://github.com/yuanxiaosc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deep dynamic Contextualized word representation (DDCWR)\nTensorFlow code and pre-trained models for DDCWR\n\n# Important explanation\n1. The method of the model is simple, only using the feed forward neural network with attention mechanism.\n2. Model training is fast, and only a few cycles can be used to train the model. The value of the initialization parameter comes from the BERT model of Google.\n3. The effect of the model is very good. In most cases, it is consistent with the current (2018-11-13) optimal model. Sometimes the effect is better. The optimal effect can be seen in [gluebenchmark](https://gluebenchmark.com/leaderboard).\n\n# Thought of article\n\nThis model Deep_dynamic_word_representation(DDWR) combines the BERT model and ELMo's deep context word representation.\n\nThe BERT comes from [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805)\nThe ELMo comes from [Deep contextualized word representations](https://arxiv.org/abs/1802.05365v2)\n\n# Basic usage method\n\n## Download Pre-trained models\n\n[BERT-Base, Uncased](https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip)\n\n## Doenload [GLUE data](https://gluebenchmark.com/tasks)DATA\n\nusing this [script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e)\n\n## Sentence (and sentence-pair) classification tasks\n\ndifference\n```\nexport BERT_BASE_DIR=/path/to/bert/uncased_L-12_H-768_A-12\nexport GLUE_DIR=/path/to/glue\n\npython run_classifier_elmo.py \\\n  --task_name=MRPC \\\n  --do_train=true \\\n  --do_eval=true \\\n  --data_dir=$GLUE_DIR/MRPC \\\n  --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n  --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n  --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \\\n  --max_seq_length=128 \\\n  --train_batch_size=32 \\\n  --learning_rate=2e-5 \\\n  --num_train_epochs=3.0 \\\n  --output_dir=/tmp/mrpc_output/\n```\n\n### Prediction from classifier\n\u003e the same as https://github.com/google-research/bert\n\n```\nexport BERT_BASE_DIR=/path/to/bert/uncased_L-12_H-768_A-12\nexport GLUE_DIR=/path/to/glue\nexport TRAINED_CLASSIFIER=/path/to/fine/tuned/classifier\n\npython run_classifier_elmo.py \\\n  --task_name=MRPC \\\n  --do_predict=true \\\n  --data_dir=$GLUE_DIR/MRPC \\\n  --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n  --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n  --init_checkpoint=$TRAINED_CLASSIFIER \\\n  --max_seq_length=128 \\\n  --output_dir=/tmp/mrpc_output/\n```\nmore methods to [google-research/bert](https://github.com/google-research/bert)\n\n\n## Solve [SQUAD1.1](https://rajpurkar.github.io/SQuAD-explorer/) problem\n\n\u003e the same as https://github.com/google-research/bert\n\ndifference\n```\npython run_squad_elmo.py --vocab_file=$BERT_BASE_DIR/vocab.txt --bert_config_file=$BERT_BASE_DIR/bert_config.json --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt --do_train=True --train_file=$SQUAD_DIR/train-v1.1.json --do_predict=True --predict_file=$SQUAD_DIR/dev-v1.1.json --train_batch_size=12 --learning_rate=3e-5 --num_train_epochs=2.0 --max_seq_length=384 --doc_stride=128 --output_dir=./tmp/elmo_squad_base/\n```\n\n## Experimental Result\n\n```\npython run_squad_elmo.py\n{“exact_match”: 81.20151371807, “f1”: 88.56178500169332}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuanxiaosc%2Fdeep_dynamic_contextualized_word_representation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyuanxiaosc%2Fdeep_dynamic_contextualized_word_representation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuanxiaosc%2Fdeep_dynamic_contextualized_word_representation/lists"}