[PROPOSAL]: support not implementation of delta lake or some other physical manifestation, but V2 data source of Spark

[MironAtHome]

[PROPOSAL]: support not implementation of delta lake or some other physical manifestation, but V2 data source of Spark

[clee704]

I agree that it could be more beneficial for the community in general in the long run if Spark supports index as a concept in Data Source V2 API. Actually, there is already work going on (SPARK-36525). If you want to contribute to that direction, then it might be better to work on Spark directly.

I think one of the values of Hyperspace is being or trying to be data format agnostic. For example, covering indexes work as long as the data source supports creating from an existing dataset a new dataset with fewer columns and laid out in certain ways (i.e. bucketing) for efficient scanning by selected columns. Data skipping indexes work as long as the data source stores data in multiple objects (i.e. files) and can compute aggregations grouped by object. So we support not only Parquet, Delta Lake, Iceberg, but also CSV and other data sources as long as they are capable of certain things.

Looking at the current code and how it's evolving, it seems the index API in Spark doesn't allow Hyperspace-like indexing subsystems that support more than one data source to be plugged in. If we want to build Hyperspace around the Data Source V2 API, then we should propose a suitable change to the API so that for example we can hook into the index API.