https://github.com/srm47/gpt2-paraphraser-comparisons
Finetune GPT-2 models for paraphrasing and compare its outputs with other language models
https://github.com/srm47/gpt2-paraphraser-comparisons
bert-model fine-tuning huggingface huggingface-transformers nlp paraphrase-generation summarization
Last synced: 8 months ago
JSON representation
Finetune GPT-2 models for paraphrasing and compare its outputs with other language models
- Host: GitHub
- URL: https://github.com/srm47/gpt2-paraphraser-comparisons
- Owner: SRM47
- License: apache-2.0
- Created: 2022-11-19T10:31:21.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-01-13T17:35:15.000Z (over 3 years ago)
- Last Synced: 2025-02-22T01:27:43.461Z (over 1 year ago)
- Topics: bert-model, fine-tuning, huggingface, huggingface-transformers, nlp, paraphrase-generation, summarization
- Language: Jupyter Notebook
- Homepage:
- Size: 8.63 MB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# gpt2-paraphraser-comparisons
Finetune GPT-2 Models for paraphrasing and compare them to PEGASUS and BART
## Create Datasets
Use the script ``create_dataset.ipynb`` to create the dataset in the file ``combined.txt``. Each line contains the following: ``S>>>>
P
``, where ``S`` and ``P`` are paraphrased sentences. Sentences pairs are gathered from three different datasets available on huggingface.co
- TaPaCo (en) https://huggingface.co/datasets/tapaco
- Google PAWS https://huggingface.co/datasets/paws
- Quora https://huggingface.co/datasets/quora
## Finetune GPT-2 Models
Finetuned three different sized GPT 2 models for sentence level paraphrasing using the ``Trainer()`` API.
Models available on huggingface:
- SRM47/gpt2-paraphraser
- SRM47/gpt2-medium-paraphraser
- SRM47/gpt2-large-paraphraser
## Evaluate Models
To evaluate the finetuned GPT-2 models and other models, use the ``eval_models.ipynb`` script
## Results Analysis
See the paper ``final.pdf`` to read about the results of this investigation.
As of recent, large language models, particularly a part of the Generative Pre-Trained series, have demonstrated themselves to be powerful text generation models. Models such as GPT-2 (Radford et al., 2018) reveal that large language models have strong zero-shot capabilities in a variety of downstream natural language pro- cessing tasks. Other models, built for sequence to sequence modeling, such as PEGASUS, and BART have profound text summarization capa- bilities which can be adapted to paraphrasing. In this paper, I present an effective method for adapting GPT-2 for paraphrasing, and compare its paraphrasing outputs to fine tuned BART and PEGASUS based models from huggingface. Results show that GPT-2 based models produce less diverse paraphrases than PEGASUS and BART; GPT-2 based paraphrases do not alter lexical form as much as PEGASUS does.