craigbryan/csi4107-tweet-search

csi4107-tweet-search's Issues

Try different analyzers

I just added an option to use Lucene's EnglishAnalyzer (EA) instead of the StandardAnalyzer (SA). They both do tokenizing and stemming (with the same word bank, I think), but the EA also does stemming (using porter stemmer) and possessive filtering (gets rid of 's). Interestingly, the EA get a better average precision, but a worse P@10 than the SA.

Need to be able to access the vocabulary generated with the index for the assignment

Explain the algorithms, data structures, and optimizations that you used in each of the three steps. How big was the vocabulary? Include a sample of 100 tokens from your vocabulary. Include the first 10 answers to queries 1 and 25. Discuss your final results.

Interpreting Hashtags

We should probably create a Tokenizer/Analyzer that interprets hashtags specifically and adds weighting to those terms.

We could find a way to add weights to these terms while we index the documents (Extend the Analyzer/Tokenizer)
or
We can inject this function into our Similarity Computation somehow (we would have to implement our own Similarity Class)

Try cosine similarity instead of the basic boolean model

Try adding relevance feedback