pasmod/paradox

An Automatic Paraphrase Detection System

Paradox: Automatic Paraphrase Identification

Given two sentences, paradox returns a continuous valued similarity score on a scale from 0 to 5, with 0 indicating that the semantics of the sentences are completely independent and 5 signifying semantic equivalence. Paradox uses Glove pre-trained models.

How to install

Paradox is dockerized! First install Docker and then run the following commands:

cd paradox
make install
make download_glove
make download_models

Training Corpus

For training, the semantic similarity corpora from SemEval (2012-2016) are used. The training data are available under /corpus.

Evaluation

The evaluation scipt reports the results on the test data set of the SemEval2016 challange. To see the resport run the following commands:

source env/bin/activate
python benchmark.py

Citation

This repository contains the code for the DeepLDA approach introduced in the following paper. Use the following bibtex entry to cite us:

@InProceedings{liebeck-EtAl:2016:SemEval,
    author    = {Liebeck, Matthias and Pollack, Philipp and Modaresi, Pashutan and Conrad, Stefan},
    title     = {HHU at SemEval-2016 Task 1: Multiple Approaches to Measuring Semantic Textual Similarity},
    booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
    month     = {June},
    year      = {2016},
    address   = {San Diego, California},
    publisher = {Association for Computational Linguistics},
    pages     = {607--613},
    url       = {TOBEFILLED-http://www.aclweb.org/anthology/W/W05/W05-0292}
}

ToDos:

• Implement topical similarity based of the LDA models.

paradox's Issues

Save the trained neural net

Train metric should be F1 not accuracy

Add training data for the remaining three languages if available

Encode sequence with vocabulary indexes like in reorderer

Experiment with other deep learning architectures

Word sequence max length via median

https://github.com/pasmod/paradox/blob/master/paradox/loaders/keras_loader.py#L40

Implement STS Similarity

Solve the problem of absolute paths

Experiment with character sequences and LSTMs

Baseline: One-Hot-Encoding of the words

Set batchsize to max length

Baseline: Scikit CountVecotizer with SVM as baseline

Add evaluation data

Add optional message to ResultLogger

Refactor the choice of the baseline estimator into a function.

estimate_svm_baseline(... ) and estimate_svm_baseline(..., True) should be wrapped

result_logger should be parallelizable

Move result_logger into logger module

Refactor evaluation.count_vectorizer_word_baseline

• Move hindi_tokenizer into separate file
• Split the creation of the pipeline and the estimate_svm_baseline into different files

Experiment with character n-grams of sizes 2 and 3

Check if the Hindi tokenizer works from right to left?

Add Normalizer to SVM Baselines

Refactor keras_test

Seperate functions for:

• get_data(language, task)
• model generation in separate files
• compile step into separate file, multiple methods for different optimizers

[feature] verbose mode for similarity transformer

I added a verbose mode for the similarity transformer using tqdm. let me know if you want a PR for this. it looks like this when running benchmark.py

the benchmark script takes a rather long time on my computer so I wanted to figure out what was going on and save the model for reuse as well

diff --git a/paradox/benchmark.py b/paradox/benchmark.py
index b0beba2..5922ac7 100644
--- a/paradox/benchmark.py
+++ b/paradox/benchmark.py
@@ -1,3 +1,4 @@
+import logging
 from metrics import pearson, mse
 from pipeline import pipeline
 import k_neighbors_regressor
@@ -5,6 +6,7 @@ import numpy as np
 import similarity
 import parser
 
+logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')
 
 def report(correlations, errors, y_pred_fold):
     print("PC:\t\t\t%0.2f\t(+/- %0.2f)" % (np.mean(correlations),
@@ -28,11 +30,14 @@ def test(model=None, categories=[]):
 pairs = parser.parse(mode="train")
 X = [pair[0] for pair in pairs]
 y = [pair[1] for pair in pairs]
-transformer = similarity.build()
+transformer = similarity.build(verbose=True)
 estimator = k_neighbors_regressor.build(n_neighbors=4)
 p = pipeline(transformers=[transformer], estimator=estimator)
 p.fit(X, y)
 
+import pickle
+with open('model.pickle', 'wb') as f:
+    pickle.dump(p, f)
 
 test(p, categories=["answer-answer"])
 test(p, categories=["question-question"])
diff --git a/paradox/similarity.py b/paradox/similarity.py
index d5b20b5..6496787 100644
--- a/paradox/similarity.py
+++ b/paradox/similarity.py
@@ -41,8 +41,8 @@ def similarity(text1, text2, levels=['surface', 'context']):
     return sims
 
 
-def build(levels=['surface', 'context']):
-    pipeline = Pipeline([('transformer', Similarity(levels=levels))])
+def build(levels=['surface', 'context'], verbose=False):
+    pipeline = Pipeline([('transformer', Similarity(levels=levels, verbose=verbose))])
     return ('similarity', pipeline)
 
 
@@ -52,15 +52,24 @@ def param_grid():
 
 
 class Similarity(BaseEstimator):
-    def __init__(self, levels=['surface']):
+    def __init__(self, levels=['surface'], verbose=False):
         self.levels = levels
+        self.verbose = verbose
 
     def fit(self, X, y):
         return self
 
     def transform(self, X):
         a = []
-        for x in X:
+
+        tqdm = lambda x: x
+        if self.verbose:
+            try:
+                from tqdm import tqdm
+            except ImportError:
+                pass
+
+        for x in tqdm(X):
             a.append(self._transform(x))
         return a

Refactor main keras call

Each model should specify the following parameters

• In which representation the data set should be loaded into (chars / words)
• get_model should be refactored
• compile_optimizer should be in the same function as the model creation; no overengeneering

Baseline: Scikit char-based CountVectorizer with SVM as baseline

Add CrossValidation

Baseline: One-Hot-Encoding of the characters

Add model checkpoints

Experiment with split random_state