.option('columns', 'col1,col2,col3,col4') does not preserve order

[astrolabsoftware]

.option('columns', 'col1,col2,col3,col4') does not preserve order

[JulienPeloton]

Hi Jacob,

Thanks again for your precious feedback :-)

I think there are three separate things here:

1. Order is not preserved when using the columns option (clearly a bug)
2. Is there a gain in specifing manually the schema in the multifile case?
3. Speeding up the I/O by selecting only some columns.

1), this is clearly a bug, that I found (the sorted should not be called):

spark-fits/src/main/scala/com/astrolabsoftware/sparkfits/FitsHduBintable.scala

Lines 49 to 50 in 7aa656b

	val colPositions = selectedColNames.map(
	x => getColumnPos(keyValues, x)).toList.sorted

I will open a PR and push the fix.

2), since #55, it is not obvious that not providing the schema will add overhead. As long as you know all the FITS schemas are the same, the code will perform minimal checks.

3), this is tricky. In practice, the option columns allows to skip the decoding part, but not the loading part, hence there is only a little speed-up. To do it properly, we would need to implement filters at the level of the record reader directly. This work is planned, I am just lacking of time :-(

Having said that, the proper way to get speed-up would be to upgrade the structure of spark-fits to Apache Spark Data Source V2 (we are currently using V1), recently released. But that's a lot of work, and probably needs to be done in the context of an internship as a whole project.

[JulienPeloton]

Also on 3), FITS is row-based, so when we read data it is natural to do:

(row, col)
element (0, 0) / element (0, 1) / element (0, 2) / ... 
element (1, 0) / element (1, 1) / element (1, 2) / ...
...

while for an efficient reading of columns, you would rather prefer to have a column-based format:

(row, col)
element (0, 0) / element (1, 0) / element (2, 0) / ...
element (0, 1) / element (1, 1) / element (2, 1) / ...
...

I suspect this will probably make the column filter pushdown implementation no so straightforward :-(

[JulienPeloton]

The ordering has been fixed in #77