yaolu/ordered-prompt

ordered-prompt's Issues

Gentle Warning on Incorrect Data File.

Great paper, and thanks for the opensource effort!

A gentle reminder:

The sst2 dataset provided in this ./data/sst2/test.jsonl directory is incorrect, all instances in the test.jsonl file are labeled as '0', while half of them should be '1'.

Clarification request `evaluate_file` import

Hi Yao,

I'd like to request a clarification regarding the following import:
https://github.com/yaolu/Ordered-Prompt/blob/975d1e2f70ecac895a7ae236ae1962ec748d51b2/entropy.py#L4

The first suspect, HuggingFace's evaluate library, does not implement a function called evaluate_file.
Since this function is loading a custom pickle file, it seems that evaluate might be a custom module that might be missing from this repo?

In this context what does evaluate refer to and where can I find the implementation of the function evaluate_file?

Thank you!