jzbjyb/lpaqa

Language model Prompt And Query Archive

LPAQA: Language model Prompt And Query Archive

This repository contains data and code for the paper How Can We Know What Language Models Know?

Install

Our repository is based on LAMA. Please download our fork from here and follow the instructions to set up the environment and download pre-trained language models.

# clone LPAQA
git clone https://github.com/jzbjyb/LPAQA.git LPAQA
# clone LAMA
git clone https://github.com/jzbjyb/LAMA.git LAMA
pushd LAMA && git reset --hard b6b1885c64de5981f249a8b65de25cb0802b4bd5 && rm -rf .git && popd
mv LAMA/* LPAQA/ && rm -rf LAMA
# follow the instructions to install
cd LPAQA
./setup.sh

Retrieve factual knowledge from LMs

For example, to query the owner of MSN (Microsoft is the answer), you can either use manually created prompts (x is owned by y):

python lama/eval_generation.py --lm bert --t "MSN is owned by [MASK]."

or use LPAQA that ensembles a diversity of prompts:

# mined prompts
python lama/eval_ensemble.py --lm bert --subject MSN --relation P127 --prompts prompt/mine
# paraphrased prompts
python lama/eval_ensemble.py --lm bert --subject MSN --relation P127 --prompts prompt/paraphrase

lpaqa's Issues

TypeError: Highway.forward: `input` must be present

if this error occur, just install the right overrides version:

pip install overrides==3.1.0

This works.

Trouble running experiments

I'm trying to run the experiments (run_exp.sh) and I'm running into a problem with the rel_file. I assume it's one of the mine/paraphrase prompt files so for the --rel_file parameter I put something like "prompts/mine/P19.jsonl" but then I get an error saying the key 'relation' is expected. Do I need to add a relations key like in get_test_phrase_parameters?
i.e. relations = [{"relation": "P108", "template": ["[X] works for [Y] .", "[Y] commentator [X] ."]}]

More specifically, how do I recreate the micro/macro averaged accuracies of Table 2 and 3 in the paper?

Manual templates

Hi, thanks for sharing the code!
I can see templates by manual_paraphrase, but I'm don't know which one is manual or paraphrased. Could you also share the manual templates only?
Thanks

ModuleNotFoundError: No module named 'knowledge_bert'

from knowledge_bert import BertTokenizer, BertForMaskedLM, BasicTokenizer
from knowledge_bert.tokenization import whitespace_tokenize_ent

"knowledge_bert" is not in the https://github.com/facebookresearch/LAMA/tree/main/lama/modules. How can I get them?

Thanks!

Can you share the code to mine these prompts?

I am interested in the code to mine prompts automatically, can you share them? Or recommend some others' code do this insteadly? thanks

The provided weights are for which model

Hi, neither in the paper nor in the readme, is it specified, corresponding to which LM were those weights trained on. By that I mean, do I have to use those weights with the BERT-base or BERT-large? Moreover are they cased on uncased models?