lfy79001/s3hqa

Code for ACL 2023 paper S3HQA: A Three-Stage Approach for Multi-hop Text-Table Hybrid Question Answering

S3HQA

This is the project containing source code for the paper S3HQA: A Three-Stage Approach for Multi-hop Text-Table Hybrid Question Answering in ACL 2023.

Requirements

python==3.7
torch==1.7.1+cu110
transformers==4.21.1

Data prepare

Download all data from hear .

Then unzip Data.zip .

Download bert-base-uncased model, deberta-base model, bart-large model from huggingfacehub. Or you can use them directly without downloading the model by changing the code.

Put bert-base-uncased model in ./PTM/bert-base-uncased and bart-large model in ./PTM/bart-large.

Use our retrieval data for your work （such as LLM)

If your work just focuses on the reader rather than retrieval.

Directly use train.row.json, dev.row.json and test.row.json for your experiments.

Training

Use checkpoint

If you want to get final answers of dev or test set.

First, download reader checkpoint from hear.

Then you can directly run bash read_dev.sh or bash read_test.sh to get the answers.

Train retriever

retriever step1 bash retrieve1.sh

retriever step2 bash retrieve2.sh

Train reader

bash read.sh

s3hqa's Issues

Evidence used in LLM prompting

Thanks for your great work! I'd like to learn about some details about the experiments of LLM prompting.

In your shared data "dev.row.json" and "test.row.json", each sample has a great many links with "link_labels" and "links_rank", I'd like to know how many evidence did you feed into LLM along with table and question, and how to select them. Thank you!

A question about the 'link_labels' and 'links_rank'

I've tried some experiments on the data 'dev.row.json' you released, and I am not sure about the meaning of the key "link_labels" and "links_rank".

• What's the meaning?
  • I want to know that if
    • the "links_rank" is the output of your trained retriever?
    • the "link_labels" indicates the link you want to select?
• How to select the links?
  • I've compared the two values in some cases, and I found that links with the label of '1' don't always have a higher rank. So how do you choose the links in your project, according to the rank or the labels?

Thanks a lot ~