https://github.com/peterding/albert

An Implementation for ALBERT with Tensorflow 2.0
https://github.com/peterding/albert

Last synced: 3 months ago
JSON representation

An Implementation for ALBERT with Tensorflow 2.0

• Host: GitHub
• URL: https://github.com/peterding/albert
• Owner: PeterDing
• Created: 2019-10-31T07:23:24.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2019-10-31T07:24:05.000Z (over 5 years ago)
• Last Synced: 2025-01-12T17:47:10.838Z (5 months ago)
• Language: Python
• Size: 19.5 KB
• Stars: 0
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# An Implementation for ALBERT with Tensorflow 2.0

An Implementation of [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942) with Tensorflow 2.0.

## Create Training Dataset

```

python3 create_pretraining_data.py \

    --file_patterns="data/*txt" \

    --max_seq_length=512 \

    --max_mask_length=20 \

    --tfrecord_file_num=8 \

    --whole_word_mask=False \

    --vocab_file='data/vocab.txt'

```

## Pre Training

```

python3 run_pretraining.py \

    --strategy=mirror \

    --model_type=base \

    --train_files='data-training/*' \

    --eval_files='data-training/*'

```