marceau-h/dedoublonneur

----------- Allows to delete duplicates froml a list -----------

You have to pass a list, it will process "pivots", texts from the list that'll be used as reference texts in order to process the similarity.

The syntax is the following, it'll process a few things when initialized :

instance = avance.Avance(list_to_process, nb_pivots : int = 50)

You'll then be able to process the duplicates with the method "process"

The syntax is the following, you can explicit a few parameters:

instance.process(taillegroupe = 50, sensibilite = .0004)

"taillegroupe" is the size of each group, the smallest, the fastest to process. But a small group means that the texts are only compared to a limited sample.

"sensibilite" is the minimal similarity required between two texts in order to flag them as dulpicates --> the sensivity of the process

There's a example notebook called "test.ipynb" alongside a list so you can try the script !