Visualization of data

[mitchellspryn]

Visualization of data

[mitchellspryn]

Done. Notebook notebooks/Initial Analysis.ipynb (commit 89c4a74).

Takeaways:

Model Description

Input: two free-text fields, which are presumably questions.
Output: binary label, 1 if the questions are asking the same thing, 0 otherwise.

Dataset Description

400k question pairs.
150k examples of duplicate questions.
250k examples of non-duplicate questions.
350k distinct questions - some questions are frequently repeated.

Next Steps

This is a natural language processing problem. There are some fundamental building blocks that we will need:

Cleaning. We need to remove special characters, punctuation, lemmatize / stem the text, etc.
Word embedding model. We can start with one-hot, but that may not be sufficient. As our corpus doesn't contain much factual text, we may look to external sources to build word embedding models wikitext

Then, we can start looking at models:

Tf/Idf cosine similarity: this should be quick and dirty. It may not perform well, but it can serve as a baseline on which to evaluate the more advanced models.
Sequence RNN: RNNs have shown great promise in understanding these types of problems (LSTM, GRU, etc.). We have a large enough dataset that we can consider training one of these.

In addition, we should think about ways to leverage the graph-like strucuture found in this notebook via an interative prediction algorithm.