{"id":13534932,"url":"https://github.com/miroozyx/BERT_with_keras","last_synced_at":"2025-04-02T00:31:02.888Z","repository":{"id":217043818,"uuid":"161617353","full_name":"miroozyx/BERT_with_keras","owner":"miroozyx","description":"A Keras version of Google's BERT model","archived":false,"fork":false,"pushed_at":"2019-11-04T08:26:58.000Z","size":396,"stargazers_count":33,"open_issues_count":0,"forks_count":11,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-11-02T22:32:53.127Z","etag":null,"topics":["bert","deep-learning","nlp","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/miroozyx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-13T09:44:17.000Z","updated_at":"2024-02-29T04:51:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"298fc5f8-60d0-4bf0-86d4-8eda811ed4e4","html_url":"https://github.com/miroozyx/BERT_with_keras","commit_stats":null,"previous_names":["miroozyx/bert_with_keras"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miroozyx%2FBERT_with_keras","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miroozyx%2FBERT_with_keras/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miroozyx%2FBERT_with_keras/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miroozyx%2FBERT_with_keras/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/miroozyx","download_url":"https://codeload.github.com/miroozyx/BERT_with_keras/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246734975,"owners_count":20825211,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","deep-learning","nlp","tensorflow"],"created_at":"2024-08-01T08:00:47.040Z","updated_at":"2025-04-02T00:31:02.862Z","avatar_url":"https://github.com/miroozyx.png","language":"Python","funding_links":[],"categories":["implement of BERT besides tensorflow:"],"sub_categories":[],"readme":"# BERT_with_keras\nThis is a implementation of **BERT**(**B**idirectional **E**ncoder **R**epresentation of **T**ransformer) with **Keras**.\n\nThe backend of Keras must be **tensorflow**.\n\n## Usage\n\nHere is a quick-start example to preprocess raw data for pretraining and fine-tuning for text classification.\nFor more details, see [Predicting Movie Review Sentriment with BERT](https://github.com/miroozyx/BERT_with_keras/blob/master/movie_reviews_classification.ipynb)\n\n### Data\nLet's use Standord's Large Movie Review Dataset for **BERT** pretraining and fine-tuning, the code below, which downloads,extracts and imports the dateset, is \nborrowed from this [tensorflow tutorial](https://www.tensorflow.org/hub/tutorials/text_classification_with_tf_hub). The\ndataset consists of IMDB movie reviews labeled by positivity from 1 to 10.\n```python\nimport os\nimport re\nimport tensorflow as tf\nimport pandas as pd\n\n# Load all files from a directory in a DataFrame.\ndef load_directory_data(directory):\n    data = {}\n    data[\"sentence\"] = []\n    data[\"sentiment\"] = []\n    for file_path in os.listdir(directory):\n        with tf.gfile.GFile(os.path.join(directory, file_path), \"r\") as f:\n            data[\"sentence\"].append(f.read())\n            data[\"sentiment\"].append(re.match(\"\\d+_(\\d+)\\.txt\", file_path).group(1))\n    return pd.DataFrame.from_dict(data)\n\n# Merge positive and negative examples, add a polarity column and shuffle.\ndef load_dataset(directory):\n    pos_df = load_directory_data(os.path.join(directory, \"pos\"))\n    neg_df = load_directory_data(os.path.join(directory, \"neg\"))\n    pos_df[\"polarity\"] = 1\n    neg_df[\"polarity\"] = 0\n    return pd.concat([pos_df, neg_df]).sample(frac=1).reset_index(drop=True)\n\n# Download and process the dataset files.\ndef download_and_load_datasets(force_download=False):\n    dataset = tf.keras.utils.get_file(\n        fname=\"aclImdb.tar.gz\", \n        origin=\"http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\", \n        extract=True)\n  \n    train_df = load_dataset(os.path.join(os.path.dirname(dataset), \n                                         \"aclImdb\", \"train\"))\n    test_df = load_dataset(os.path.join(os.path.dirname(dataset), \n                                          \"aclImdb\", \"test\"))\n    return train_df, test_df\n \ntrain, test = download_and_load_datasets()\n```\n\n### pre-training\n\nlet's train a bert pre-training model.\n```python\nimport os\nimport spacy\nfrom const import bert_data_path,bert_model_path\nfrom preprocess import create_pretraining_data_from_docs\nfrom pretraining import bert_pretraining\nnlp = spacy.load('en')\n\n# use IMDB movie review as pretraining data\ntexts = train['sentence'].tolist() + test['sentence'].tolist()\n\nsentences_texts=[]\nfor text in texts:\n    doc = nlp(text)\n    sentences_texts.append([s.text for s in doc.sents])\n\nvocab_path = os.path.join(bert_data_path, 'vocab.txt')\n\ncreate_pretraining_data_from_docs(sentences_texts,\n                                  vocab_path=vocab_path,\n                                  save_path=os.path.join(bert_data_path,'pretraining_data.npz'),\n                                  token_method='wordpiece',\n                                  language='en',\n                                  dupe_factor=10)\n\nbert_pretraining(train_data_path=os.path.join(bert_data_path,'pretraining_data.npz'),\n                 bert_config_file=os.path.join(bert_data_path, 'bert_config.json'),\n                 save_path=bert_model_path,\n                 batch_size=32,\n                 seq_length=128,\n                 max_predictions_per_seq=20,\n                 val_batch_size=32,\n                 multi_gpu=0,\n                 num_warmup_steps=1000,\n                 checkpoints_interval_steps=1000,\n                 pretraining_model_name='bert_pretraining.h5',\n                 encoder_model_name='bert_encoder.h5')\n```\nThen, pertraining data would be found in save_dir. \n### Fine-tuning\nYou can use the pre-training model as the initial point for your NLP model. \nFor example, you can use the pre-training model to init a classfier model. \n```python\nimport os\nimport keras\nimport numpy as np\nfrom const import bert_data_path, bert_model_path\nfrom modeling import BertConfig\nfrom classifier import SingleSeqDataProcessor, convert_examples_to_features, Text_Classifier, save_features, TextSequence\nfrom tokenization import FullTokenizer\nfrom optimization import AdamWeightDecayOpt\nfrom checkpoint import StepModelCheckpoint\n\n# data preprossing\ntrain_examples = SingleSeqDataProcessor.get_train_examples(train_data=train['sentence'].tolist(),labels=train['polarity'].tolist())\ndev_exmaples = SingleSeqDataProcessor.get_dev_examples(dev_data=test['sentence'].tolist(), labels=test['polarity'].tolist())\n\n# `word piece tokenizer` need to a prepared vocabulary.\nvocab_path = os.path.join(bert_data_path, 'vocab.txt')\n\n# load vocab to tokenizer\ntokenizer = FullTokenizer(vocab_path, do_lower_case=True)\n\n# convert the train and dev examples to features\ntrain_features = convert_examples_to_features(train_examples, \n                                              label_list=[0,1], \n                                              max_seq_length=128, \n                                              tokenizer= tokenizer)\ndev_features = convert_examples_to_features(dev_exmaples, label_list=[0,1], max_seq_length=128, tokenizer=tokenizer)\n\n# convert features to a dictionary of numpy arrays.\ntrain_features_array_dict = save_features(features=train_features)\ndev_features_array_dict = save_features(features=dev_features)\n\n# get train and validation data\ntrain_x = [train_features_array_dict['input_ids'], train_features_array_dict['input_mask'], train_features_array_dict['segment_ids']]\ntrain_y = keras.utils.to_categorical(train_features_array_dict['label_ids'], 2)\nval_x = [dev_features_array_dict['input_ids'], dev_features_array_dict['input_mask'], dev_features_array_dict['segment_ids']]\nval_y = keras.utils.to_categorical(dev_features_array_dict['label_ids'],2)\n\n# load bert configuration file\nconfig = BertConfig.from_json_file(os.path.join(bert_data_path, 'bert_config.json'))\nepochs = 3\nnum_gpus = None\n# if you come across a OOM problem, reduce the batch size.\nbatch_size = 16\n\n# calculation the number of training steps by epoch size.\nnum_train_samples = len(train_features_array_dict['input_ids'])\nnum_train_steps = int(np.ceil(num_train_samples / batch_size)) * epochs\nprint(\"number of train steps: {}\".format(num_train_steps))\n\n# Use weight decay adam optimizer. this optimizer is sightly different with Keras's Standard Adam optimizer. \n# For more details, view source code of AdamWeightDecayOpt.\nadam = AdamWeightDecayOpt(\n        lr=5e-5,\n        num_train_steps=num_train_steps,\n        num_warmup_steps=100,\n        beta_1=0.9,\n        beta_2=0.999,\n        epsilon=1e-6,\n        weight_decay_rate=0.01,\n        exclude_from_weight_decay=[\"LayerNorm\", \"layer_norm\", \"bias\"]\n    )\n\n# This checkpoint evaluate the bert model performance on batch end.\ncheckpoint = StepModelCheckpoint(filepath=\"%s/%s\" % (bert_model_path, 'imdb_classifer_model.h5'),\n                                 verbose=1, monitor='val_acc',\n                                 save_best_only=True,\n                                 xlen=3,\n                                 period=100,\n                                 start_step=100,\n                                 val_batch_size=128)\n# create a model\nclassifier = Text_Classifier(bert_config=config,\n                             pretrain_model_path=os.path.join(bert_model_path, 'bert_movie_reviews_encoder.h5'),\n                             batch_size=batch_size,\n                             seq_length=128,\n                             optimizer=adam,\n                             num_classes=2,\n                             metrics=['acc'],\n                             multi_gpu= num_gpus\n                             )\n\n# when using multi-gpus, the parallel model of bert cann't be used to evaluate/predict.\n# You can only use the cpu_build model to evalate and predict.\nif num_gpus is not None:\n    checkpoint.single_gpu_model = classifier.model\n\n# train model\ngenerator= TextSequence(x=train_x,y=train_y,batch_size=batch_size)\nhistory = classifier.fit_generator(generator=generator,\n                                   epochs=epochs,\n                                   shuffle=True,\n                                   callbacks=[checkpoint],\n                                   validation_data=(val_x,val_y)\n                                   )                                \n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiroozyx%2FBERT_with_keras","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmiroozyx%2FBERT_with_keras","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiroozyx%2FBERT_with_keras/lists"}