Giter Site home page Giter Site logo

hpclab / istella22-experiments Goto Github PK

View Code? Open in Web Editor NEW
2.0 5.0 0.0 264.66 MB

Official software repository of the ACM SIGIR 2022 resource paper: "The Istella22 Dataset: Bridging Traditional and Neural Learning to Rank Evaluation" by D. Dato et al.

License: MIT License

Python 47.60% Jupyter Notebook 52.40%

istella22-experiments's Introduction

Experiments on the Istella22 Dataset

This GitHub repository contains instructions on how to replicate the results presented in:

The Istella22 Dataset: Bridging Traditional and Neural Learning to Rank Evaluation

LambdaMART

Three LambdaMART models are available in this repository. The experiments published in the Istella22 resource paper can be reproduced by following the Jupyter Notebook evaluation.

Additional features to build the MonoT5 SVM files can be found in the lambdamart/data subdirectory. The features should be pasted to the Istella22 official test.svm file to build the final test file for the MonoT5 and MonoT5 (Title + Url + Text) versions. Final test files can be produced using the paste command.

paste -d' ' test.svm monoT5.feature.svm > test.monoT5.svm
paste -d' ' test.svm monoT5.titleUrlText.svm > test.monoT5.titleUrlText.svm

Both models and features are gzipped. Unzip them before use.

MonoT5

MonoT5 models are available on Huggingface as macavaney/it5-base-istella-title_url_text and macavaney/it5-base-istella-title_url.

You can use them using the MonoT5 transformer included in this package. Example:

import pandas as pd
import pyterrier as pt ; pt.init()
from monot5 import MonoT5

model = MonoT5('macavaney/it5-base-istella-title_url')
input = pd.DataFrame([{'qid': '1', 'query': 'test', 'title': 'test document', 'url': 'https://test.com/'}])
model(input)
# qid query          title                url     score  rank
#   1  test  test document  https://test.com/ -0.005107     0

You can run experiments for both models with:

python run_monot5.py

Citation

If you use Istella22 and/or the source code shared with this GitHub repository, please cite:

The Istella22 Dataset: Bridging Traditional and Neural Learning to Rank Evaluation

@inproceedings{istella22,
    author = {Domenico Dato and Sean MacAvaney and Franco Maria Nardini and Raffaele Perego and Nicola Tonellotto},
    title = {The Istella22 Dataset: Bridging Traditional and Neural Learning to Rank Evaluation},
    booktitle = {Proceedings of ACM SIGIR 2022},
    year = {2022}
}

istella22-experiments's People

Contributors

francomarianardini avatar seanmacavaney avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

istella22-experiments's Issues

nDCG of the fine-tuned MonoT5 models diverge from paper

Hey all,

First of all thank you for this interesting work (I enjoyed reading the paper a lot)! After cloning and executing the project run_monot5.py we obtained the following results from the cached run files monoT5/runs:

name P@1 P@5 P@10 nDCG@10 nDCG@20 RR AP
MonoT5 fine-tuned title+url 0.8412 0.5991 0.3914 0.6858 0.7087 0.9025 0.7396
MonoT5 fine-tuned title+url+text 0.8581 0.5945 0.3910 0.7034 0.7268 0.9132 0.7462

Which are notably higher values in terms of nDCG than the values reported in the paper (which are โ‰ˆ0.45). A student of mine also re-ran the T5 models published on huggingface without the run caching and reported similarly diverging values.

I just wanted to highlight this finding. Do you have any idea where these values are coming form?

Cheers,
Philipp

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.