Support replace table

[rdblue]

Support replace table

[mike-weinberg]

Replacing a table as a single operation would replace all metadata.

Does this have to be true?

if CTAS C at t₀ (C_{t_0}) is a deterministic operation against a snapshot of a dataset D at t₀ (D_{t_0}), then isn't it possible to preserve metadata whenever the materialization program P_C (e.g. a version of a SQL/PIG query or spark program) does not change, propagating only those changes from any upstream data set?

e.g. if D_{t_1}'s metadata differs in a way that doesn't change the data (e.g. renames a column), C_{t_1} can be considered to use the same materialization program as C_{t_0} specifically because it references ID's and not names in its transformations.

In other words, if
P_{Ct_0} = P_{Ct_1} = P_{C_0}
is true, then table schema metadata for the iceberg table representing C can be preserved.

In addition, (I think) it is at minimum plausible to say:
P_{Ct_0} = P_{Ct_1}
even in the case of specific data-affecting metadata changes, for example a type promotion from short to long integer (but NOT in the case of a type demotion, which might cause P_{Ct_0} to be unsafe to apply to D_{t_1})

Seemingly the only time the metadata could not be preserved would be when the relational reference / equivalent of a FROM statement is changed such that metadata is only replaced if the data lineage cannot be considered equivalent from one version of the materialization program to the next.

As for what I'm getting at, I'd like to offer that while metadata replacement is a sensible default operation for CTAS, I think it would be useful to leave room for the possibility that CTAS could be treated as a materialized view, and therefore that metadata replacement is one of multiple reasonable behaviors for table-replacing operations.

[rdblue]

@mike-weinberg: no, it doesn't have to be true. That's why there are multiple operations: insert (do not change table metadata), replace/overwrite data (overwrite some/all data, then insert as a single operation), CTAS (create table and insert in one operation) and RTAS (drop, create, and insert in one operation).

Those cover different use cases. If you want to preserve existing metadata, then you should insert or replace data. You would only replace a table if you specifically do not care about compatibility with the previous schema. For example, if you create a report table every day and you want to update the query that produces it, you probably don't want to make DDL changes and then run a job that replaces the contents. You just want to replace whatever is there with a new version using new version's schema.

For more context, check out the logical plans we are proposing for Spark in the Standardize Logical Plans SPIP.

[mike-weinberg]

@rdblue thanks for the added context. Metadata history and lineage has been on my mind for a while so I guess my thinking and preferences are biased, but I see what you mean.

[rdblue]

This was added in d2bf002.