hpclab/istella22-experiments

Official software repository of the ACM SIGIR 2022 resource paper: "The Istella22 Dataset: Bridging Traditional and Neural Learning to Rank Evaluation" by D. Dato et al.

Experiments on the Istella22 Dataset

This GitHub repository contains instructions on how to replicate the results presented in:

The Istella22 Dataset: Bridging Traditional and Neural Learning to Rank Evaluation

LambdaMART

Three LambdaMART models are available in this repository. The experiments published in the Istella22 resource paper can be reproduced by following the Jupyter Notebook evaluation.

Additional features to build the MonoT5 SVM files can be found in the lambdamart/data subdirectory. The features should be pasted to the Istella22 official test.svm file to build the final test file for the MonoT5 and MonoT5 (Title + Url + Text) versions. Final test files can be produced using the paste command.

paste -d' ' test.svm monoT5.feature.svm > test.monoT5.svm
paste -d' ' test.svm monoT5.titleUrlText.svm > test.monoT5.titleUrlText.svm

Both models and features are gzipped. Unzip them before use.

MonoT5

MonoT5 models are available on Huggingface as macavaney/it5-base-istella-title_url_text and macavaney/it5-base-istella-title_url.

You can use them using the MonoT5 transformer included in this package. Example:

import pandas as pd
import pyterrier as pt ; pt.init()
from monot5 import MonoT5

model = MonoT5('macavaney/it5-base-istella-title_url')
input = pd.DataFrame([{'qid': '1', 'query': 'test', 'title': 'test document', 'url': 'https://test.com/'}])
model(input)
# qid query          title                url     score  rank
#   1  test  test document  https://test.com/ -0.005107     0

You can run experiments for both models with:

python run_monot5.py

Citation

If you use Istella22 and/or the source code shared with this GitHub repository, please cite:

The Istella22 Dataset: Bridging Traditional and Neural Learning to Rank Evaluation

@inproceedings{istella22,
    author = {Domenico Dato and Sean MacAvaney and Franco Maria Nardini and Raffaele Perego and Nicola Tonellotto},
    title = {The Istella22 Dataset: Bridging Traditional and Neural Learning to Rank Evaluation},
    booktitle = {Proceedings of ACM SIGIR 2022},
    year = {2022}
}

istella22-experiments's Issues

nDCG of the fine-tuned MonoT5 models diverge from paper

Hey all,

First of all thank you for this interesting work (I enjoyed reading the paper a lot)! After cloning and executing the project run_monot5.py we obtained the following results from the cached run files monoT5/runs:

name	P@1	P@5	P@10	nDCG@10	nDCG@20	RR	AP
MonoT5 fine-tuned title+url	0.8412	0.5991	0.3914	0.6858	0.7087	0.9025	0.7396
MonoT5 fine-tuned title+url+text	0.8581	0.5945	0.3910	0.7034	0.7268	0.9132	0.7462

Which are notably higher values in terms of nDCG than the values reported in the paper (which are ≈0.45). A student of mine also re-ran the T5 models published on huggingface without the run caching and reported similarly diverging values.

I just wanted to highlight this finding. Do you have any idea where these values are coming form?

Cheers,
Philipp