pandinosaurus/seq2seq-prompt

Source code for COLING 2022 paper "Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models"

Seq2Seq-Prompt

Source code and dataset for Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models.

Quick links

• Overview
• Requirements
• Prepare the data
• Run AutoSeq
  • Main results
  • Search for automatic label sequence and label word mappings
• Citation

Overview

In this work we propose AutoSeq, a general-purpose, fully automatic prompting method that depends only on few-shot annotations.

Figure: Illustration of AutoSeq. We first use T5 to generate label sequence candidates given each label's training instances; we then use contrastive re-ranking to get label sequences that are more label-specific; in the end we enumerate all the combinations and re-rank by the fine-tuning performance.

Requirements

To run our code, please install all the dependencies:

pip install -r requirements.txt

Prepare the data

Please download the official GLUE and SuperGLUE data and extract the files to ./data/GLUE/TASK_NAME and ./data/SuperGLUE/TASK_NAME.

Then use the following command to generate the few-shot data:

python tools/generate_k_shot_data_for_GLUE.py
python tools/generate_k_shot_data_for_SuperGLUE.py

Run AutoSeq

To reproduce our results, please use the scripts we provide. For convenience, we use SST-2 as an example below.

Main results

AutoSeq:

task=SST-2 bash scripts/t5_seq.sh

AutoWord:

task=SST-2 bash scripts/t5_word.sh

Manual:

task=SST-2 bash scripts/t5_man.sh

Manual with eng.:

task=SST-2 bash scripts/t5_man_eng.sh

Fine-tuning:

task=SST-2 bash scripts/t5_fine_tuning.sh

Fine-tuning (Full train set):

task=SST-2 bash scripts/t5_full_fine_tuning.sh

Prompt tuning:

task=SST-2 bash scripts/t5_prompt_tuning.sh

All the results will be stored in ./log. To gather all the results, run the following command:

python tools/gather_result.py --condition "{'tag': 'exp-autoseq-SST-2', 'task_name': 'sst-2', 'few_shot_type': 'prompt'}"

We also release the original log of our experiments to facilitate the reproduction.

Search for automatic label sequence and label word mappings

You can find our generated mappings in ./auto_label_sequences and ./auto_label_words.

We first generate candidate label sequence and label word mappings by running:

python tools/generate_label_sequences.py
python tools/generate_label_words.py

The generated candidates will be saved in ./my_auto_label_sequences and ./my_auto_label_words.

Then we do prompt-based fine-tuning of all the mappings by:

task=SST-2 bash scripts/sort_mapping.sh

Finally we sort them based on dev set performance:

python tools/sort_mapping.py --condition "{'tag': 'exp-mapping-SST-2', 'task_name': 'sst-2'}" --mapping_dir auto_label_sequences

Citation

Please cite our paper if you use AutoSeq in your work:

@inproceedings{yu2022automatic,
   title={Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models},
   author={Zichun Yu and Tianyu Gao and Zhengyan Zhang and Yankai Lin and Zhiyuan Liu and Maosong Sun and Jie Zhou},
   booktitle={International Conference on Computational Linguistics (COLING)},
   year={2022}
}