atif-mahmud/bubbl-spark

bubbl-spark's Issues

Distributed query using input document keywords (pipeline step 2)

The search engine should be called with the input document's keywords to determine the related documents. The search engine should be called multiple times, each call obtaining a different segment of the search results. All calls should be performed in parallel with each call being made on a different machine.

Calculate document vectors (pipeline step 4)

The document vector should be calculated for each cleaned search result. This is done by:

1. Converting the document into words (we actually textrank the document to do this).
2. Calculate the average word2vec across all all the words in the document to create our document vector

Textrank input documents (pipeline step 1)

Incoming documents must be run through textrank to obtain their keywords.

Download and clean search results (pipeline step 3)

Each document in the search results should be downloaded and cleaned. All download/cleaning operations should be performed in parallel.

Cluster documents (pipeline step 5)

Now we cluster the documents by their document vector

Title clusters (pipeline step 6)

Each document cluster is then titled. Currently this is done by ranking the most important keyword in each cluster by running FastText across the combined corpus of all the documents in the cluster.