imvladikon / quora-question-pair Goto Github PK
View Code? Open in Web Editor NEWduplicates detection experiments on Quora Question Pairs (QQP)
License: Other
duplicates detection experiments on Quora Question Pairs (QQP)
License: Other
Write a function to count how many words were in the each sentence in dirty data set
https://github.com/imvladikon/quora-question-pair/blob/distribution_lenth_sentence_word/notebooks/galina/count_words_and_chars_dirty.ipynb
The same\different sentences ratio
Sentence length in words distribution
Sentence length in chars distribution
Misspelling https://github.com/imvladikon/quora-question-pair/issues/2
Table: length of corpus of words with and without stop-words
number of unique words with or without lemmatisation
plot number of occurrence of lemma s
bigram statistics. number of bigrams without stop words; number of bigrams after lemmatisation.
Plot number of ocurrence of the bigrams
Levenstein distance after lemmatisation for same and distinct sentences.
Count misspelled words and do a table:
Stats on word usage (count in unique questions and just bool/presence)
Some wrong examples:
=======================
Masha punctuation
What is the story of Kohinoor (Koh-i-Noor) Diamond?
what story kohinoor (koh-i-noor diamond
what story kohinoor koh noor diamond
what story kohinoor koh-i-noor diamond
Vova punctuation
What does it feel like to have sex with a man if you are (or were) a straight man?
what doe feel like sex man (or straight man
what feel like sex man straight man
what feel like sex man straight man
Vova punctuation
I'm a boy. I masturbate. I'm 13. Is it bad to masturbate?
boy. masturbate. 13. bad masturbate
i be boy i masturbate i be is bad masturbate
--- Now we don't know do we need pronouns, so I don't know what is our gold standard
Masha punctuation
What is the difference between urban and rural living in the U.S.?
what difference urban rural live u.s.
what difference urban rural live u s
Masha punctuation
How do I build and host an e-commerce website?
how build host e-commerce website
how i build host e commerce website
Masha stop words
Is black or white the absence of color?
black white absence color
is black white absence color
black white absence color
Masha numbers
Who will win the 2016 presidential elections?
who win 2016 presidential election
who win presidential election
who win 2016 presidential election
Vova question words, Masha numbers
Which phone should I buy under 15k?
phone buy 15k
which phone i buy k
which phone I buy under 15k
Masha numbers
What are the Best B2B website?
what best b2b website
what best b b website
what best b2b website
Vova punctuation
I really like this guy. He has a girlfriend, but we talk all the time and it seems like he flirts with me. How do I get him to like me?
really like guy. girlfriend talk time seem like flirt me. how get like
i really like guy he girlfriend talk time seem like flirt how i get like
i really like guy he girlfriend talk time seem like flirt how i get like
Vova lemmatization
What is the science behind this? Why does the coaster not fall down?
what science behind why doe coaster fall
what science behind why coaster fall
Masha punctuation
Do employees at Lowe's have a good work-life balance? Does this differ across positions and departments?
employee lowe good work-life balance differ across position department
do employee lowe s good work life balance does differ across position department
Write a list of plots and tables we need to prepare
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.