airalcorn2/ranknet

My (slightly modified) Keras implementation of RankNet and PyTorch implementation of LambdaRank.

RankNet and LambdaRank

My (slightly modified) Keras implementation of RankNet (as described here) and PyTorch implementation of LambdaRank (as described here). See here for a tutorial demonstating how to to train a model that can be used with Solr.

ranknet's Issues

Some confuses about ranknet

Hi,

Sorry to disturb you. For me I have confused about the weight that the network learned. This means that for each query [q1,q2,...qn] with documents[d1,d2....dm] the network will learn the w1i[w11,w12....] for the query1 and w2i[w21,w22....] for the query2? Could you please give me some helps. Thank you very much.

Sean

split batches in one query for LambdaRank

Hi @airalcorn2 ,

Thanks for your clear code.

I am a little confused about LambdaRank that I hope you can help with. If I have a total of 2560 docs in one query, can I split it into 10 batches to do 10 backwards, or I have to include all 2560 docs in one batch to do 1 backward. Thanks.

Why lamb_upates missing a negative sign

in lambdarank.py:

lamb_updates = 1 / (1 + exped) * N * dcg_diffs.abs()

but equation (6) in [2]:
lamb(i, j) = -1 / (1 + exped) * N * dcg_diffs.abs()

network prediction examples

My 2 cents: The last part of your code generating scores was unclear to me.
Instead, the below exemplified how the trained network outputs 1 when the two vectors are uncorrelated, and something close to 0.5 when they are.
Feel free to add it to your code.
Also, maybe a jupyter notebook would make more sense, as it can display in github the results without you having to copy paste them in-line the code as comments.

(Pdb) model.predict([X_1[0:1,:], np.zeros(shape=(1,50))])
array([[0.9999989]], dtype=float32)
(Pdb) model.predict([X_1[0:1,:], np.ones(shape=(1,50))])
array([[0.9999285]], dtype=float32)
(Pdb) model.predict([X_1[0:1,:], np.random.randn(N, INPUT_DIM)])
array([[0.99992096]], dtype=float32)
(Pdb) model.predict([X_1[0:1,:], X_1[0:1,:]])
array([[0.5]], dtype=float32)
(Pdb) model.predict([X_1[0:1,:], X_2[0:1,:]])
array([[0.9995178]], dtype=float32)
(Pdb) model.predict([X_1[0:1,:], X_1[0:1,:]+0.1])
array([[0.47106665]], dtype=float32)
(Pdb) model.predict([X_1[0:1,:], X_1[0:1,:]+0.2])
array([[0.44332358]], dtype=float32)
(Pdb) model.predict([X_1[0:1,:], X_1[0:1,:]-0.2])
array([[0.5630561]], dtype=float32)
(Pdb) model.predict([X_1[0:1,:], X_1[0:1,:]+0.1*np.random.randn(N, INPUT_DIM)])
array([[0.46504748]], dtype=float32)
(Pdb) model.predict([X_1[0:1,:], X_1[0:1,:]+0.1*np.random.randn(N, INPUT_DIM)])
array([[0.5252977]], dtype=float32)
(Pdb) model.predict([X_1[0:1,:], X_1[0:1,:]+0.1*np.random.randn(N, INPUT_DIM)])
array([[0.4684026]], dtype=float32)

Ranknet to Predict New Query

Hi,
I have question regarding your keras implementation of Ranknet:
How to feed features of a new one query to the model to predict the ranking list ?

thank you

How to input the query into Ranknet?

Hi,
The model structure of Ranknet shows that it just use the feature of document as input,
Does Ranknet use query as input too?How?

Should we randomly change the order of rel_doc, irr_doc pair?

Hi,
In my training data, there are two columns rel_doc and irr_doc,
which means the ranking of doc in pos_doc should be higher than the one in irr_doc.
If the model input is a (rel_doc, irr_doc) pair,
then all of the label(ground truth) will be just 1.
Should we randomly change the order of rel_doc, irr_doc pair and convert the label to -1?

ps. In the paper From RankNet to LambdaRank to
LambdaMART: An Overview, it say the cost function is symmetric while swapping i,j and change label(S_ij) from 1 to -1,
but in my case, the S_ij is fixed(S_ij=1 or say S_ji=-1).

Possible error in ndcg_diffs calculation

I think it should be (with log2 on denominator only):

ndcg_diffs = (1 / (1 + doc_ranks[:n_rel]).log2()) - (1 / (1 + doc_ranks[n_rel:]).log2()).view(n_irr)