https://github.com/jzbjyb/LPAQA

Language model Prompt And Query Archive
https://github.com/jzbjyb/LPAQA

Last synced: 2 months ago
JSON representation

Language model Prompt And Query Archive

• Host: GitHub
• URL: https://github.com/jzbjyb/LPAQA
• Owner: jzbjyb
• Created: 2019-11-27T09:54:39.000Z (about 5 years ago)
• Default Branch: master
• Last Pushed: 2021-05-11T04:49:20.000Z (over 3 years ago)
• Last Synced: 2024-08-04T06:01:06.238Z (6 months ago)
• Language: Shell
• Homepage:
• Size: 869 KB
• Stars: 156
• Watchers: 5
• Forks: 13
• Open Issues: 5
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-sentiment-attitude-extraction - [code

README

        
# LPAQA: Language model Prompt And Query Archive

This repository contains data and code for the paper [How Can We Know What Language Models Know?](https://arxiv.org/abs/1911.12543)

## Install

Our repository is based on [LAMA](https://github.com/facebookresearch/LAMA). Please download our fork

 from [here](https://github.com/jzbjyb/LAMA) and follow the instructions to set up the environment and download pre-trained language models.

```bash

# clone LPAQA

git clone https://github.com/jzbjyb/LPAQA.git LPAQA

# clone LAMA

git clone https://github.com/jzbjyb/LAMA.git LAMA

pushd LAMA && git reset --hard b6b1885c64de5981f249a8b65de25cb0802b4bd5 && rm -rf .git && popd

mv LAMA/* LPAQA/ && rm -rf LAMA

# follow the instructions to install

cd LPAQA

./setup.sh

```

## Retrieve factual knowledge from LMs

For example, to query the owner of MSN (Microsoft is the answer), you can either use manually created prompts (*x is owned by y*):

```bash

python lama/eval_generation.py --lm bert --t "MSN is owned by [MASK]."

```



or use LPAQA that ensembles a diversity of prompts:

```bash

# mined prompts

python lama/eval_ensemble.py --lm bert --subject MSN --relation P127 --prompts prompt/mine

# paraphrased prompts

python lama/eval_ensemble.py --lm bert --subject MSN --relation P127 --prompts prompt/paraphrase

```