Refactoring of the index and document

[castorini]

Refactoring of the index and document

[lintool]

@Peilin-Yang For the TREC documents, you might want to consider what we did for the IR-reproducibility experiments: https://github.com/lintool/IR-Reproducibility/tree/master/systems/lucene/ingester/src/main/java/luceneingester

IIRC, there the code uses an actual XML parser instead of string parsing...

[Peilin-Yang]

I think you rely on some Lucene code to do the parsing (https://github.com/lintool/IR-Reproducibility/blob/master/systems/lucene/ingester/src/main/java/luceneingester/IndexThreads.java#L25-L27)?

Yes, Lucene has an arsenal of parsing different old TREC collections, e.g. GOV2, LATimes, FT, FBIS, etc.
https://github.com/apache/lucene-solr/tree/master/lucene/benchmark/src/java/org/apache/lucene/benchmark/byTask/feeds

It will be interesting to look into it.

[lintool]

Yes, we should copy that code and integrate/refactor in Anserini.

[Peilin-Yang]

Merged