Do you have a plan to release a standard evaluation protocol?

[todpole3]

Do you have a plan to release a standard evaluation protocol?

[danyaljj]

The short answer is "yes"!

We are working to consolidate the split and should be able to release something soon -- hopefully in a week or two.

FYI @yizhongw

[yizhongw]

Hi @danyaljj, thanks for this work! Can I clarify - since this issue was a few weeks ago, and I saw the paper released recently, were you referring to the paper being released or is there something else coming?

In particular, I am looking for

1. The evaluation split of tasks
2. Code to reproduce the results of the paper (e.g. for finetuning and evaluating on this dataset)

If they are not yet released, it would be helpful if you could give a rough idea for when that would be released (I am trying to decide whether to include this excellent benchmark in my own work but that would depend on the timeline).

Thank you!

Thanks for your interest! We are planning to release the splits and evaluation scripts next week (hopefully by Wed). Does this work for your schedule?

[JunShern]

Hi @yizhongw, any update on this? Sorry to rush you, but for our planning it'll be helpful to know whether we will get to include this or not.

[yizhongw]

Hi @yizhongw, any update on this? Sorry to rush you, but for our planning it'll be helpful to know whether we will get to include this or not.

@JunShern Sorry for being late on this! We have merged this PR just now, and now the official evaluation splits can be found here. You can follow the instructions there for setting up your experiments.
We also released our code for Tk-Instruct model and checkpoints. Feel free to let me know if you find any issue!

[todpole3]

Thanks for the heads-up!

[JunShern]

Hi @danyaljj, thanks for this work! Can I clarify - since this issue was a few weeks ago, and I saw the paper released recently, were you referring to the paper being released or is there something else coming?

In particular, I am looking for

1. The evaluation split of tasks
2. Code to reproduce the results of the paper (e.g. for finetuning and evaluating on this dataset)

If they are not yet released, it would be helpful if you could give a rough idea for when that would be released (I am trying to decide whether to include this excellent benchmark in my own work but that would depend on the timeline).

Thank you!

[JunShern]

Amazing, yes next week should work for us! Thanks for the quick response and all the best. :)

[JunShern]

Amazing, thank you! I'll be looking into it :)