Inconsistency sometimes occurs across multiple runs on the same file

[nil0x42]

Inconsistency sometimes occurs across multiple runs on the same file

[ViperelB]

It's also a little weird that using "duplicut -l 24" on a wordlist, then reusing the result with -l 16 afterwards, the size will always be smaller than doing -l 16 on the original wordlist. Technically it should be the same amount of duplicate/filtered lines taken out.

visual example:
Original = 2.03 GB -> duplicut -l 24 = 1.11 GB -> duplicut -l 16 = 563 MB
Original = 2.03 GB -> duplicut -l 16 = 867 MB

example2:
0...9999999.dict file (94.1mb) -> sort -u = same as original (94.1mb)
0...9999999.dict file (94.1mb) -> duplicut (successfully removed 0 duplicates and 0 filtered lines in 09 seconds) = 83.5mb (opened in notepad++ it seems the file suddenly stops after "8874998" while the original goes up to 9999999