https://github.com/nishiwen1214/zero-few-shot-electra

Code of paper: "ELECTRA is a Zero-shot learner, Too" -- Prompt-based ELECTRA for zero-shot learning.
https://github.com/nishiwen1214/zero-few-shot-electra

bert4keras electra prompt-learning rtd zero-shot-learning

Last synced: about 9 hours ago
JSON representation

Code of paper: "ELECTRA is a Zero-shot learner, Too" -- Prompt-based ELECTRA for zero-shot learning.

• Host: GitHub
• URL: https://github.com/nishiwen1214/zero-few-shot-electra
• Owner: nishiwen1214
• License: mit
• Created: 2021-12-05T04:56:50.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2022-08-16T06:22:31.000Z (over 2 years ago)
• Last Synced: 2023-03-09T05:27:15.191Z (over 1 year ago)
• Topics: bert4keras, electra, prompt-learning, rtd, zero-shot-learning
• Language: Python
• Homepage:
• Size: 43 KB
• Stars: 6
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# 《ELECTRA is a Zero-Shot Learner, Too》 

## Overview

Recently, for few-shot or even zero-shot learning, the new paradigm “*pre-train, prompt, and predict*” has achieved remarkable achievements compared with the “*pre-train, fine-tune*” paradigm. A series of small language models (e.g., BERT, ALBERT, RoBERTa) based on **Masked Language Model (MLM)** pre-training tasks became popular and widely used. However, another efficient and powerful pre-trained language model, ELECTRA, has probably been neglected. This paper attempts to accomplish several NLP tasks in the zero-shot scenario using a sample-efficient ELECTRA original pre-training task—**Replaced Token Detection (RTD)**. Through extensive experiments on 15 various NLP datasets, we find that ELECTRA performs surprisingly well as a zero-shot learner, which proves the ELECTRA model has more potential to be stimulated.



## Use

```

python electra_classification.py

python electra_sentence_pair_classification.py

 

python electra_STS-B.py

```

## Main experimental results



## Environment

bert4keras>=0.10.8, tensorflow = 1.15.0, keras = 2.3.1；

## Acknowledgements

Our code is based on [Jianlin Su](https://github.com/bojone)'s [bert4keras](https://github.com/bojone/bert4keras) and [Sun Yi](https://github.com/sunyilgdx)'s [NSP-BERT](https://github.com/sunyilgdx/NSP-BERT/). Thank you for your open source spirit!

## Citation

```

@article{ni2022electra,

  title={ELECTRA is a Zero-Shot Learner, Too},

  author={Ni, Shiwen and Kao, Hung-Yu},

  journal={arXiv preprint arXiv:2207.08141},

  year={2022}

}

```