jwhitbeck/dendrite

Dendrite is a library for querying large datasets on a single host at near-interactive speeds.

dendrite's Issues

Empty vectors are read as 'nil' by default

Writing out an empty vector will not preserve its vector type on read.

index-by does not work as expected for files-reader

AFAICT the index arg passed to f by index-by is the index in the current file, starting from 0 each time a new file is entered. I think the natural expectation would be that the index arg is the index in the entire sequence of records across all the files being parsed by files-reader.

Make the "invalid dendrite file" exception message more useful

IMO the exception message should include the path of the file that's being read. Here's a real-life example. Which file on disk is corrupt? One must add debugging statements to find out.

Caused by: java.lang.IllegalStateException: File is not a valid dendrite file.
        at dendrite.java.FileReader.readMetadata(FileReader.java:301)
        at dendrite.java.FileReader.create(FileReader.java:87)
        at dendrite.core$file_reader.invokeStatic(core.clj:140)
        at dendrite.core$file_reader.invoke(core.clj:133)
        at dendrite.core$file_reader.invokeStatic(core.clj:138)
        at dendrite.core$file_reader.invoke(core.clj:133)

bundleSize is not a configurable WriterOption

Currently, bundle size is defaulted to 256: https://github.com/jwhitbeck/dendrite/blob/master/dendrite/java-src/dendrite/java/Options.java#L343. If the write thread is outrun by the writing thread, it's possible that the batchQueue gets filled. This is a problem if the machine running dendrite doesn't have enough memory.

Another method to solve this problem might be to apply backpressure via the API (i.e. making certain write calls more expensive when a flush is necessary).

Add an option to read the first N lines from a file

Pulling only the first N lines from a file would be a useful option during development, when you want to read just a handful of lines out of a very large file to test your code.

You can approximate this behavior by using a reducer which exits early:

(let [lines-read (atom 0)
      wrapped-reducer (fn [acc v]
                        (if (>= @lines-read 10)
                          (do
                            (println "10 lines have been read")
                            (reduced acc))
                          (do (swap! lines-read inc)
                              (original-reducer acc v))))]
  ..)

However, this still takes a few seconds. From @jwhitbeck:

Indeed dendrite is currently optimized for throughput and is reading far ahead of those ten lines. The read process has two stages: (1) deserialize each active column into arrays of values (2) assemble the nested records from the flattened columnar layout. Your wrapped-reducer short-circuits (2) but doesn't impact (1).

Adding a "line-count" option is definitely the way to go, but isn't easy given the current implementation. More generally, I have a design for adding indexing/filtering capabilities and buffering less when it isn't needed which should make dendrite much snappier for interactive use.