mitchellspryn/kagglequoraquestionsimilarity

Github repo for kaggle quora question similarity problem

kagglequoraquestionsimilarity's Issues

Visualization of data

Get following information:

*Label distribution
*Number of unique words
*Average sentence length
*Word distributions

The notebook should be generic so it can be used after cleaning is done.

LSTM Question similarity

Investigate using LSTM for question similarity. Start with one-hot encoding until word embedding investigation is done.

Divide up dataset into train/test

Perform stratified split on provided dataset and upload to repo

Word Embedding

Investigate word embedding model

Investigate performance of FuzzyWuzzy

https://github.com/seatgeek/fuzzywuzzy. Might be a useful feature.

Text Cleaning Strategy

Devise a consistent way to clean the text, map similar words together, etc.

Explore identfiying frequent questions

In the Initial Analysis notebook (notebooks/Initial Analysis.ipynb, commit 89c4a74), we saw that there was a pattern that certain questions were linked against often as duplicates. Perhaps we can use this as a mechanism for augmenting our duplicate identification process.

Rolling Sentiment Window exploration

There was the theory floated at the meeting on 4/29 that removing emotionally charged passages from the questions may lead to better identification of the core question being asked, in turn making our models more accurate.

Let's test that theory.

Create a notebook / code that predicts sentiment for given window sizes. Find a way that for a given window size / threshold value:

• how many posts are affected
• does it actually help identifying the core question? For now, some spare examples are fine.

Use the above two to identify optimal window size / threshold value / other algorithm details.