Giter Site home page Giter Site logo

khan-www23's Introduction

[WWW'23] KHAN: Knowledge-Aware Hierarchical Attention Networks for Accurate Political Stance Prediction

This repository provides an implementation of KHAN as described in the paper: KHAN: Knowledge-Aware Hierarchical Attention Networks for Accurate Political Stance Prediction by Yunyong Ko, Seongeun Ryu, Soeun Han, Youngseung Jeon, Jaehoon Kim, Sohyun Park, Kyungsik Han, Hanghang Tong, and Sang-Wook Kim, In Proceedings of the ACM Web Conference (WWW) 2023.

The overview of KHAN

The overview of KHAN

  • Datasets
    • To reflect the different political knowledge of each entity, we build two political knowledge graphs, KG-lib and KG-con. Also, for extensive evaluation, we construct a large-scale political news datatset, AllSides-L, much larger (48X) than the existing largest political news article dataset.
  • Algorithm
    • We propose a novel approach to accurate political stance prediction (KHAN), employing (1) hierarchical attention networks (HAN) and (2) knowledge encoding (KE) to effectively capture both explicit and implicit factors of a news article.
  • Evaluation
    • Via extensive experiments, we demonstrate that (1) (accuracy) KHAN consistently achieves higher accuracies than all competing methods (up to 5.92% higher than the state-of-the-art method), (2) (efficiency) KHAN converges within comparable training time/epochs, and (3) (effectiveness) each of the main components of KHAN is effective in political stance prediction.

Datasets

  1. News articles datasets (SemEval, AllSides-S, AllSides-L)
Dataset # of articles Class distribution
SemEval 645 407 / 238
AllSides-S 14.7k 6.6k / 4.6k / 3.5k
AllSides-L 719.2k 112.4k / 202.9k / 99.6k / 62.6k / 241.5k
  1. Knowledge Graphs (YAGO, KG-conservative, KG-liberal)
KG dataset # of source poses # of entities # of raltions
YAGO - 123,182 1,179,040
KG-lib 219,915 5,581 29,967
KG-con 276,156 6,316 33,207
  1. Pre-trained KG embeddings (common, conservative, liberal)

Dependencies

Our code runs on the Intel i7-9700k CPU with 64GB memory and NVIDIA RTX 2080 Ti GPU with 12GB, with the following packages installed:

python 3.8.10
torch 1.11.0
torchtext 0.12.0
pandas
numpy
argparse
sklearn

How to run

python3 main.py \
  --gpu_index=0 \
  --batch_size=16 \
  --num_epochs=50 \
  --learning_rate=0.001 \
  --max_sentence=20 \
  --embed_size=256 \
  --dropout=0.3 \
  --num_layer=1 \
  --num_head=4 \
  --d_hid=128 \
  --dataset=SEMEVAL \
  --alpha=0.6 \
  --beta=0.2

Citation

Please cite our paper if you have used the code in your work. You can use the following BibTex citation:

@inproceedings{ko2023khan,
  title={KHAN: Knowledge-Aware Hierarchical Attention Networks for Accurate Political Stance Prediction},
  author={Ko, Yunyong and Ryu, Seongeun and Han, Soeun and Jeon,Youngseung and Kim, Jaehoon and Park, Sohyun and Han, Kyungsik Tong, Hanghang and Kim., Sang-Wook},
  booktitle={Proceedings of the ACM Web Conference (WWW) 2023},
  pages={1572--1583},
  year={2023},
  isbn = {9781450394161},
  publisher = {Association for Computing Machinery (ACM)},
  doi = {10.1145/3543507.3583300},
  location = {Austin, TX, USA},
}

khan-www23's People

Contributors

seongeunryu avatar sosilver2080 avatar yy-ko avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

khan-www23's Issues

Hyperparameter setting for allsides data

Dear authors. Thank you for your great work and also your efforts in sharing the code & data.

I was wondering if you can let me know which hyperparameters were used for allsides data.

With your default hyperparameter setting in the README, the performance of allsides-s test data have reached maximum of 70% accuracy like below and I could not find all the required parameters from the paper.

Again, thank you for your work and assistance.

==================================================================================
--gpu_index=0
--batch_size=16
--num_epochs=50
--learning_rate=0.001
--max_sentence=20
--embed_size=256
--dropout=0.3
--num_layer=1
--num_head=4
--d_hid=128
--dataset=ALLSIDES-S
--alpha=0.6
--beta=0.2
Count of using GPUs: 2
Count of using GPUs: 2
====================================TRAIN INFO START====================================

  • TRAINING MODEL = KHAN
    • Embedding Size = 256
    • Maximum Length = 20
    • Number of Transformer Encoder Layers = 1
    • Number of Multi-head Attentions = 4
    • Hidden Layer Dimension = 128
    • Dropout Probability = 0.3
    • Alpha = 0.6
    • Beta = 0.2
  • DATASET = ALLSIDES-S
  • BATCH SIZE = 16
  • NUM EPOCHS = 50
  • LEARNING RATE = 0.001

==================================== Training Start ====================================

  • Training data size: 13304
  • Test data size: 1479
    [2.233713901947616, 3.180492469519484, 4.20347551342812]
  • Reading Pre-trained Knowledge Embeddings...
    50
    torch.Size([256])
    Total params: 122.80M
    Fold: 1 | Epoch: 1 | Loss: 1.3780 | TrainAcc: 0.4497 | ValAcc: 0.6105 | Time: 40.98
    Fold: 1 | Epoch: 2 | Loss: 0.9669 | TrainAcc: 0.5270 | ValAcc: 0.5916 | Time: 40.51
    Fold: 1 | Epoch: 3 | Loss: 0.8768 | TrainAcc: 0.5714 | ValAcc: 0.5625 | Time: 40.43
    Fold: 1 | Epoch: 4 | Loss: 0.8125 | TrainAcc: 0.6040 | ValAcc: 0.5991 | Time: 40.43
    Fold: 1 | Epoch: 5 | Loss: 0.7928 | TrainAcc: 0.6197 | ValAcc: 0.6045 | Time: 40.58
    Fold: 1 | Epoch: 6 | Loss: 0.7621 | TrainAcc: 0.6335 | ValAcc: 0.6241 | Time: 40.37
    Fold: 1 | Epoch: 7 | Loss: 0.7471 | TrainAcc: 0.6439 | ValAcc: 0.6302 | Time: 40.31
    Fold: 1 | Epoch: 8 | Loss: 0.7436 | TrainAcc: 0.6512 | ValAcc: 0.6504 | Time: 40.32
    Fold: 1 | Epoch: 9 | Loss: 0.7038 | TrainAcc: 0.6757 | ValAcc: 0.6234 | Time: 40.48
    Fold: 1 | Epoch: 10 | Loss: 0.6852 | TrainAcc: 0.6941 | ValAcc: 0.6376 | Time: 40.37
    Fold: 1 | Epoch: 11 | Loss: 0.6664 | TrainAcc: 0.7068 | ValAcc: 0.6092 | Time: 40.42
    Fold: 1 | Epoch: 12 | Loss: 0.6353 | TrainAcc: 0.7223 | ValAcc: 0.6795 | Time: 40.41
    Fold: 1 | Epoch: 13 | Loss: 0.6127 | TrainAcc: 0.7389 | ValAcc: 0.6160 | Time: 40.42
    Fold: 1 | Epoch: 14 | Loss: 0.5859 | TrainAcc: 0.7540 | ValAcc: 0.6599 | Time: 40.39
    Fold: 1 | Epoch: 15 | Loss: 0.5598 | TrainAcc: 0.7653 | ValAcc: 0.6633 | Time: 40.53
    Fold: 1 | Epoch: 16 | Loss: 0.5337 | TrainAcc: 0.7862 | ValAcc: 0.6849 | Time: 40.34
    Fold: 1 | Epoch: 17 | Loss: 0.5187 | TrainAcc: 0.7892 | ValAcc: 0.6545 | Time: 40.52
    Fold: 1 | Epoch: 18 | Loss: 0.5091 | TrainAcc: 0.8012 | ValAcc: 0.6910 | Time: 40.27
    Fold: 1 | Epoch: 19 | Loss: 0.4569 | TrainAcc: 0.8190 | ValAcc: 0.6728 | Time: 40.29
    Fold: 1 | Epoch: 20 | Loss: 0.4471 | TrainAcc: 0.8238 | ValAcc: 0.6842 | Time: 40.32
    Fold: 1 | Epoch: 21 | Loss: 0.4305 | TrainAcc: 0.8340 | ValAcc: 0.6795 | Time: 40.32
    Fold: 1 | Epoch: 22 | Loss: 0.4135 | TrainAcc: 0.8422 | ValAcc: 0.6802 | Time: 40.24
    Fold: 1 | Epoch: 23 | Loss: 0.4013 | TrainAcc: 0.8430 | ValAcc: 0.6883 | Time: 40.26
    Fold: 1 | Epoch: 24 | Loss: 0.3912 | TrainAcc: 0.8534 | ValAcc: 0.6890 | Time: 40.31
    Fold: 1 | Epoch: 25 | Loss: 0.3343 | TrainAcc: 0.8771 | ValAcc: 0.6910 | Time: 40.36
    Fold: 1 | Epoch: 26 | Loss: 0.3321 | TrainAcc: 0.8761 | ValAcc: 0.6687 | Time: 40.28
    Fold: 1 | Epoch: 27 | Loss: 0.2972 | TrainAcc: 0.8899 | ValAcc: 0.6795 | Time: 40.23
    Fold: 1 | Epoch: 28 | Loss: 0.2997 | TrainAcc: 0.8887 | ValAcc: 0.6728 | Time: 40.29
    Fold: 1 | Epoch: 29 | Loss: 0.2795 | TrainAcc: 0.9000 | ValAcc: 0.6782 | Time: 40.30
    Fold: 1 | Epoch: 30 | Loss: 0.2816 | TrainAcc: 0.8957 | ValAcc: 0.6660 | Time: 40.23
    Fold: 1 | Epoch: 31 | Loss: 0.2515 | TrainAcc: 0.9103 | ValAcc: 0.6795 | Time: 40.37
    Fold: 1 | Epoch: 32 | Loss: 0.2407 | TrainAcc: 0.9112 | ValAcc: 0.6863 | Time: 40.35
    Fold: 1 | Epoch: 33 | Loss: 0.2225 | TrainAcc: 0.9213 | ValAcc: 0.6802 | Time: 40.33
    Fold: 1 | Epoch: 34 | Loss: 0.2215 | TrainAcc: 0.9216 | ValAcc: 0.6957 | Time: 40.29
    Fold: 1 | Epoch: 35 | Loss: 0.2202 | TrainAcc: 0.9204 | ValAcc: 0.6788 | Time: 40.44
    Fold: 1 | Epoch: 36 | Loss: 0.2118 | TrainAcc: 0.9250 | ValAcc: 0.6775 | Time: 40.34
    Fold: 1 | Epoch: 37 | Loss: 0.1921 | TrainAcc: 0.9335 | ValAcc: 0.6768 | Time: 40.32
    Fold: 1 | Epoch: 38 | Loss: 0.1914 | TrainAcc: 0.9305 | ValAcc: 0.6707 | Time: 40.18
    Fold: 1 | Epoch: 39 | Loss: 0.1904 | TrainAcc: 0.9324 | ValAcc: 0.6694 | Time: 40.15
    Fold: 1 | Epoch: 40 | Loss: 0.1721 | TrainAcc: 0.9393 | ValAcc: 0.6755 | Time: 40.25
    Fold: 1 | Epoch: 41 | Loss: 0.1718 | TrainAcc: 0.9390 | ValAcc: 0.6897 | Time: 40.23
    Fold: 1 | Epoch: 42 | Loss: 0.1819 | TrainAcc: 0.9363 | ValAcc: 0.6829 | Time: 40.25
    Fold: 1 | Epoch: 43 | Loss: 0.1778 | TrainAcc: 0.9379 | ValAcc: 0.6700 | Time: 40.27
    Fold: 1 | Epoch: 44 | Loss: 0.1685 | TrainAcc: 0.9392 | ValAcc: 0.6897 | Time: 40.25
    Fold: 1 | Epoch: 45 | Loss: 0.1669 | TrainAcc: 0.9425 | ValAcc: 0.6714 | Time: 40.21
    Fold: 1 | Epoch: 46 | Loss: 0.1521 | TrainAcc: 0.9460 | ValAcc: 0.6667 | Time: 40.26
    Fold: 1 | Epoch: 47 | Loss: 0.1477 | TrainAcc: 0.9480 | ValAcc: 0.6755 | Time: 40.08
    Fold: 1 | Epoch: 48 | Loss: 0.1444 | TrainAcc: 0.9484 | ValAcc: 0.6761 | Time: 40.00
    Fold: 1 | Epoch: 49 | Loss: 0.1621 | TrainAcc: 0.9441 | ValAcc: 0.6856 | Time: 40.00
    Fold: 1 | Epoch: 50 | Loss: 0.1584 | TrainAcc: 0.9445 | ValAcc: 0.6640 | Time: 39.97

FOLD - 1
Test Accuracy: 0.6957, Training time: 2096.58 (sec.)

loss of YAGO.RotatE file

Hello, I would like to express my gratitude for the valuable research work you have done, which has been of great help to my project.
While attempting to replicate your work, I noticed that some key files appear to be missing, including YAGO.RotatE under ./pre-trained. Therefore, I would like to inquire whether you might be able to provide these files. Once again, thank you for your work and assistance.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.