grove/exmeso

External Merge Sort in Java

exmeso's Issues

Distinct option doesn't work with a small size of file

When and an input file is small enough to fit in a single chunk, distinct option doesn't remove duplicated items.

The reason seems because ExternalMergeSort#mergeSortedChunksNoPartialMerge(List sortedChunks) doesn't honor config.distinct as you see below.

    if (sortedChunks.size() == 1) {
        File sortedChunk = sortedChunks.get(0);
        return new ChunkFile<T>(sortedChunk, serializer, comparator, config.cleanup);
    } else {
        List<ChunkFile<T>> cfs = new ArrayList<ChunkFile<T>>(sortedChunks.size());
        for  (File file : sortedChunks) {
            cfs.add(new ChunkFile<T>(file, serializer, comparator, config.cleanup));
        }
        return new MergeSortedIterator<T,ChunkFile<T>>(cfs, comparator, config.distinct);
    }

It works fine if you eliminate the above optimizing if-true section.

The sample code gets an error

The sample code to sort json file gets org.codehaus.jackson.map.RuntimeJsonMappingException as you see in exception.txt.
The reason is a bug of org.codehaus.jackson.map.MappingIterator.java, you cannot invoke hasNext() two times or more when it hits END_ARRAY token, ']'.
Because codehaus is no more maintained, it's better to migrate to its successor: com.fasterxml.jackson.
You can see a sample migration in JacksonSerializer.zip.
The differences from the original code are shown in
JacksonSerializer.java.pdf.

MergeSortedIterator does not provider stable ordering

When multiple ChunkFiles contain values that compare equal, I would expect the mergeSortedChunks method to respect the order of the files in the sortedChunks parameter. Instead the MergeSortedIterator uses a PriorityQueue which does not guarantee a stable ordering.