lupantech/promptpg

Data and code for the ICLR 2023 paper "Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning".

promptpg's Issues

Something is wrong with the ranking...

selected exemplars were similar for different questions.

Hello, I've discovered that RL chose similar exemplars even when the query questions are different based on what I've learned from both the training logs and the resulting json files. Put otherwise, a certain number of exemplars were chosen at high frequencies.

• See shot_pids in json, where "9254" and "36539" were frequently selected while "4995" was never used.
• See cids for sample[-1] in batch in log, where [13 1] and [1 13] were dominant.

Does this suggest that those exemplars are in any way representative of the task or dataset?

Candidate examples for PromptPG at Inference time

Hi,

I am working on a method for exemplar selection for in-context learning and would like to compare it with PromptPG. From my understanding of the method, the set of candidate examples is kept the same between training and inference, but this doesn't seem to be the case in the implementation (comparing learn_policy.py:32 and run_gpt3.py:32). Is this supposed to be the case?

Thanks,
Shivanshu

Many tables do not have the correct column headers

Hi, thank you for the great dataset. There are some tables where the column headers are not right. The columns say "stem" and "leaf" which is not right. Some of the examples are: