hkust-knowcomp/cskb-population

Codes for the EMNLP2021 paper: Benchmarking Commonsense Knowledge Base Population (https://aclanthology.org/2021.emnlp-main.705.pdf). An updated version CKBP v2 (https://arxiv.org/pdf/2304.10392.pdf)

CSKB-Population

Codes for the EMNLP2021 paper: Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset

We have an expert-annotated and adversarially constructed evaluation set CKBP v2.

For data preprocessing and the codes released upon EMNLP 2021, checkout the branch emnlp2021 (https://github.com/HKUST-KnowComp/CSKB-Population/tree/emnlp2021).

Data

CKBP v1: training data, evaluation set.

CKBP v2: training data (updated), evaluation data

Environment

We tested our codes on python 3.8.5. Here are some packages their corresponding versions.

networkx                 2.5
numpy                    1.19.2
pandas                   1.2.2
scikit-learn             0.24.1
scipy                    1.6.3
sklearn                  0.0
spacy                    3.2.0
stanfordnlp              0.2.0
torch                    1.7.1
torch-geometric          1.7.2
torch-scatter            2.0.7
torch-sparse             0.6.9
torchsummary             1.5.1
torchtext                0.8.1
tqdm                     4.56.0
transformers             3.4.0

Dataset Preprocess

See the branch emnlp2021

Model Training

Download the data

kaggle datasets download -d tianqingfang/ckbp-emnlp2021

Or download at https://www.kaggle.com/datasets/tianqingfang/ckbp-emnlp2021.

Put the train.csv under data/ckbp_csv/emnlp2021/. The dev.csv and test.csv are automatically constructed dev and test sets, sampled from ASER and CSKBs (CSKB ground truth triples are labeled 1, negative sampled examples are labeled 0.)

The annotated evaluation set is at data/evaluation_set.csv. Please use this data set to test.

Train w/ KG-BERT (BERT-base)

KG-BERT baseline is provided here.

CUDA_VISIBLE_DEVICES=0 python models/train_kgbert_baseline.py \
    --ptlm bert-base-uncased \
    --lr 1e-5 \
    --epochs 1 \
    --output_dir results \
    --train_csv_path data/ckbp_csv/emnlp2021/train.csv \
    --relation_as_special_token \
    --save_best_model \
    --seed 100 --batch_size 64 --test_batch_size 128 --save_best_model --experiment_name ""

Access the whole graph data

Check out Model Training section on the branch emnlp2021.

Results

(Note that the evaluation metric is a bit different from that in the paper. In the paper we use a grouped AUC where AUC scores within different groups are calculated and averaged. Here we report the overall AUC across all relations.)

Model	All	Ori Test Set	CSKB head	ASER edge
KG-BERT (BERT-base)	62.5	74.2	51.9	54.7
KG-BERT (RoBERTa-large)	70.9	78.0	63.4	64.6

Reproduce DISCOS using current version of ASER_norm

Checkout the DISCOS-reproduce folder under the branch emnlp2021.

cskb-population's Issues

how get merged_event.npy for GLUCOSE preprocessing step

Hi,

I am trying to preprocess GLUCOSE CSKB. As it is indicated in second step of its README (Parsed Matching step), we need to get the parser and merged eventuality cache from ASER and save it as merged_event.npy. My question is that how we can get the merged eventuality cache? In the provided link from ASER there isn't such a merged event.

thanks in advance!

Where to download train.csv

Hi,

Thank you for creating the benchmark dataset.

I found the link (https://www.kaggle.com/datasets/tianqingfang/ckbp-emnlp2021) doesn't work anymore. Where can we download train.csv now?

Thanks for your help!