Giter Site home page Giter Site logo

imvladikon / quora-question-pair Goto Github PK

View Code? Open in Web Editor NEW
0.0 1.0 1.0 73.27 MB

duplicates detection experiments on Quora Question Pairs (QQP)

License: Other

Makefile 0.62% Jupyter Notebook 94.64% Python 4.62% Shell 0.12%
nlp spacy fasttext paraphrase

quora-question-pair's Introduction

quora sentence duplicates detection

Open in Colab made-with-python kaggle

The Quora dataset is composed of pairs of questions, and the task is to determine if the two questions are duplicates of each other, that is, that they have the same meaning.


├── LICENSE
├── Makefile           <- Makefile with commands like `make data` or `make train`
├── README.md          <- The top-level README for developers using this project.
├── data
│   ├── external       <- Data from third party sources.
│   ├── interim        <- Intermediate data that has been transformed.
│   ├── processed      <- The final, canonical data sets for modeling.
│   └── raw            <- The original, immutable data dump.
│
├── docs               <- A default Sphinx project; see sphinx-doc.org for details
│
├── models             <- Trained and serialized models, model predictions, or model summaries
│
├── notebooks          <- Jupyter notebooks. Naming convention is a number (for ordering),
│                         the creator's initials, and a short `-` delimited description, e.g.
│                         `1.0-jqp-initial-data-exploration`.
│
├── references         <- Data dictionaries, manuals, and all other explanatory materials.
│
├── reports            <- Generated analysis as HTML, PDF, LaTeX, etc.
│   └── figures        <- Generated graphics and figures to be used in reporting
│
├── requirements.txt   <- The requirements file for reproducing the analysis environment, e.g.
│                         generated with `pip freeze > requirements.txt`
│
├── setup.py           <- makes project pip installable (pip install -e .) so src can be imported
├── src                <- Source code for use in this project.
│   ├── __init__.py    <- Makes src a Python module
│   │
│   ├── data           <- Scripts to download or generate data
│   │   └── make_dataset.py
│   │
│   ├── features       <- Scripts to turn raw data into features for modeling
│   │   └── build_features.py
│   │
│   ├── models         <- Scripts to train models and then use trained models to make
│   │   │                 predictions
│   │   ├── predict_model.py
│   │   └── train_model.py
│   │
│   └── visualization  <- Scripts to create exploratory and results oriented visualizations
│       └── visualize.py
│
└── tox.ini            <- tox file with settings for running tox; see tox.readthedocs.io

notes

for pre-commit hooks (converting jupyter notebooks to py/html files) need to install:

pip install pre-commit
pre-commit install

see here

quora-question-pair's People

Contributors

imvladikon avatar

Watchers

 avatar

Forkers

galina-blokh

quora-question-pair's Issues

Word usage count

Stats on word usage (count in unique questions and just bool/presence)

Plots and tables

  1. The same\different sentences ratio

  2. Sentence length in words distribution

  3. Sentence length in chars distribution

  4. Misspelling https://github.com/imvladikon/quora-question-pair/issues/2

  5. Table: length of corpus of words with and without stop-words

  6. number of unique words with or without lemmatisation

  7. plot number of occurrence of lemma s

  8. bigram statistics. number of bigrams without stop words; number of bigrams after lemmatisation.

  9. Plot number of ocurrence of the bigrams

  10. Levenstein distance after lemmatisation for same and distinct sentences.

Count misspelled words in train\test\validation sets

Count misspelled words and do a table:

  • how many misspelled words
  • rate of misspelled words (misspelled words/all words)
  • how many questions consist at least one misspelled words
  • rate of misspelled questions (number questions with misspelled words/number of questions)
  • how many misspelled words could we have in 1 question (number of questions with 1 misspelled word: with 2 misspelled words ....)
  • rate of misspelled words in a sent (calculate number of misspelled words / number of words in the sentence. Round it (maybe 2 digits after the dot is enough). Number of sent with the rate)

Dealing with punctuation marks. Need to fix.

Some wrong examples:

  1. original sentence
  2. after spacy_tokenizer (Vova's)
  3. after tok_stop_lem_punct (Masha's)
  4. what we need to have

=======================
Masha punctuation
What is the story of Kohinoor (Koh-i-Noor) Diamond?
what story kohinoor (koh-i-noor diamond
what story kohinoor koh noor diamond
what story kohinoor koh-i-noor diamond

Vova punctuation
What does it feel like to have sex with a man if you are (or were) a straight man?
what doe feel like sex man (or straight man
what feel like sex man straight man
what feel like sex man straight man

Vova punctuation
I'm a boy. I masturbate. I'm 13. Is it bad to masturbate?
boy. masturbate. 13. bad masturbate
i be boy i masturbate i be is bad masturbate
--- Now we don't know do we need pronouns, so I don't know what is our gold standard

Masha punctuation
What is the difference between urban and rural living in the U.S.?
what difference urban rural live u.s.
what difference urban rural live u s

Masha punctuation
How do I build and host an e-commerce website?
how build host e-commerce website
how i build host e commerce website

Masha stop words
Is black or white the absence of color?
black white absence color
is black white absence color
black white absence color

Masha numbers
Who will win the 2016 presidential elections?
who win 2016 presidential election
who win presidential election
who win 2016 presidential election

Vova question words, Masha numbers
Which phone should I buy under 15k?
phone buy 15k
which phone i buy k
which phone I buy under 15k

Masha numbers
What are the Best B2B website?
what best b2b website
what best b b website
what best b2b website

Vova punctuation
I really like this guy. He has a girlfriend, but we talk all the time and it seems like he flirts with me. How do I get him to like me?
really like guy. girlfriend talk time seem like flirt me. how get like
i really like guy he girlfriend talk time seem like flirt how i get like
i really like guy he girlfriend talk time seem like flirt how i get like

  • Vova question, lemmatisation (d -deletion, in Masha's sent a stays after lemmatisation because she lowercases after removing stop-words, maybe it's a good way)
    Which hindi songs can be played using A D C Em chords?
    hindi song play use c em chord
    which hindi song play use a d c em chord
    which hindi song play use a d c em chord

Vova lemmatization
What is the science behind this? Why does the coaster not fall down?
what science behind why doe coaster fall
what science behind why coaster fall

Masha punctuation
Do employees at Lowe's have a good work-life balance? Does this differ across positions and departments?
employee lowe good work-life balance differ across position department
do employee lowe s good work life balance does differ across position department

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.