A5 prepare_ltr_training_data

[kbalog]

A5 prepare_ltr_training_data

[FebriantiW]

for the idf, it should use the count of whole documents in the index which is 54709 and not document count based upon search hits because it only counts document with field body.

i didn't zoom in on the search result based analyze_query so cannot answer on the 2nd one. anyone ?

[ChristofferHolmesland]

for the idf, it should use the count of whole documents in the index which is 54709
@FebriantiW

That fixed my issue, thank you! I assumed it was correct to use the document count from the search because we are calculating the IDF based on the body field.

[ChristofferHolmesland]

If anyone else is wondering about the analyze_query issue, you can ignore the terms and still pass every test in the notebook.

[thek123]

for the idf, it should use the count of whole documents in the index which is 54709 and not document count based upon search hits because it only counts document with field body.

i didn't zoom in on the search result based analyze_query so cannot answer on the 2nd one. anyone ?
I might be wrong but it said
will calculate IDF based on the 'body' field only

and it does not hold for 'trec9_index' to pass the test

[FebriantiW]

I got the same issue/numbers as @ChristofferHolmesland then later realised that in the idf formula that it considers entire document collection as N. It should pass the test even if you get the term doc count from body fields only.