Share A Single File System Instance In HadoopTableOperations

[netflix]

Share A Single File System Instance In HadoopTableOperations

[rdblue]

Can you give an example of properties you're trying to set here?

We can cache file systems in HadoopTableOperations, but most of the systems I've worked on use this pattern of getting the right file system for the URI and using the FileSystem level cache.

[mccheah]

We are experimenting with using Iceberg as a temporary representation of the tables that are backed by our internal data warehouse solution. When we do so, however, we need to put the Iceberg table metadata somewhere. We want to put it on local disk, but when we put it on local disk we need to encrypt it with a one-time encryption key that only exists for the lifetime of the Spark dataset that is being read / written; So for example we're doing something like this:

Key encryptionKey = generateKey();
Configuration conf = new Configuration();
conf.set("encryption.key", encryptionKey.toString());
HadoopTables tables = new HadoopTables(conf);
// create table and insert all metadata
sparkSession.read().option("iceberg.spark.hadoop.encryption.key", encryptionKey.toString()).load(tempTablePath);

In such a case, we don't want the same file system instance - probably a local FS instance wrapped with some encryption layer - to be cached, because every time we run this code we want a different encryption key every time.

[rdblue]

Okay, how about adding the support you're talking about to HadoopTableOperations and opening a PR? That would unblock you because you'd have the caching level you need and we could further evaluate the feature.

[rdblue]

Also, why do all of the properties include "spark"?

[mccheah]

The properties here assume being injected into sparkSession.read.option. If we wanted to include them in the Table properties set instead it should be iceberg.hadoop.

[rdblue]

Properties set through Spark wouldn't need to be specific to Spark. You might use the same ones as session properties in Presto.