oracle/oci-hdfs-connector

HDFS Connector for Oracle Cloud Infrastructure

Oracle Cloud Infrastructure HDFS Connector for Object Storage

About

oci-hdfs-connector provides the public HDFS connector that integrates with the Oracle Cloud Infrastructure Object Storage Service.

The project is open source and maintained by Oracle Corp. The home page for the project is here.

Installation

See the documentation for details.

Examples

Examples can be found here.

Documentation

Full documentation, including prerequisites and installation and configuration instructions, can be found here.

HDFS Connector provides support for delegation token configurator that injects a delegation token on every request. This can be done by setting the delegation token file path in an environment variable OCI_DELEGATION_TOKEN_FILE or by setting the Hadoop property fs.oci.delegation.token.filepath

API reference can be found here.

Help

See the “Questions or Feedback?” section here.

Version

The first three numbers of HDFS connector version represent the Hadoop version that the connector was compiled and tested against. The next three numbers are for the platform, major and minor versions respectively.

• Minor version is incremented when there are no breaking changes
• Major version is incremented when there are breaking changes which might affect some users
• Platform version is incremented when there are bigger important breaking changes which would affect many users

For example, in HDFS connector version 3.3.1.x.y.z, 3.3.1 is for the Hadoop version being used and x.y.z refers to the {platform}.{major}.{minor} version of the HDFS connector.

Build

Building HDFS connector relies on Maven artifacts that are provided by the Java SDK. To obtain the artifacts, you must download the Java SDK and build it locally. You can then build the HDFS connector.

Important: The Java SDK file version that you download from the Oracle Releases page must match the HDFS connector version, which you can find in the hdfs-connector/pom.xml file in the tag block that has the groupId attribute com.oracle.oci.sdk.

Changes

See CHANGELOG.

Known Issues

You can find information on any known issues with the connector here and under the “Issues” tab of this GitHub repository. You can find information on any known issues with the SDK here and under the “Issues” tab of this GitHub repository.

Potential data corruption issue for OCI HDFS Connector with RefreshableOnNotAuthenticatedProvider

Details: If you are using version 3.2.1.1 or earlier of the OCI HDFS Connector and you use a RefreshableOnNotAuthenticatedProvider (e.g. InstancePrincipalsCustomAuthenticator, or generally for Resource Principals or Instance Principals) you may be affected by silent data corruption.

Workaround: Update the OCI HDFS Connector to version 3.2.1.3 or later. For more information about this issue and workarounds, see Potential data corruption issue for OCI HDFS Connector with RefreshableOnNotAuthenticatedProvider.

Direct link to this issue: Potential data corruption issue with OCI HDFS Connector on binary data upload with RefreshableOnNotAuthenticatedProvider

Contributing

This project welcomes contributions from the community. Before submitting a pull request, please review our contribution guide

Security

Please consult the security guide for our responsible security vulnerability disclosure process

License

Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved. This software is dual-licensed to you under the Universal Permissive License (UPL) 1.0 as shown at https://oss.oracle.com/licenses/upl or Apache License 2.0 as shown at http://www.apache.org/licenses/LICENSE-2.0. You may choose either license.

See LICENSE for more details.

oci-hdfs-connector's Issues

Pyspark py4j.protocol.Py4JJavaError: An error occurred while calling o95.csv.

I am trying to download files on Object store with a Pyspark application. I am getting the following error for that:

2020-01-21 06:27:53,860 WARN streaming.FileStreamSink: Error while looking for metadata directory
py4j.protocol.Py4JJavaError: An error occurred while calling o95.csv.
: java.io.IOException: Unable to fetch file status for: 0febff2985/companies
        at com.oracle.bmc.hdfs.store.BmcDataStore.getObjectMetadata(BmcDataStore.java:583)
        at com.oracle.bmc.hdfs.store.BmcDataStore.getFileStatus(BmcDataStore.java:508)
        at com.oracle.bmc.hdfs.BmcFilesystem.getFileStatus(BmcFilesystem.java:302)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
        at org.apache.spark.sql.execution.datasources.DataSource.$anonfun$checkAndGlobPathIfNecessary$1(DataSource.scala:557)
        at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:244)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at scala.collection.TraversableLike.flatMap(TraversableLike.scala:244)
        at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:241)
        at scala.collection.immutable.List.flatMap(List.scala:355)
        at org.apache.spark.sql.execution.datasources.DataSource.checkAndGlobPathIfNecessary(DataSource.scala:545)
        at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:359)
        at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
        at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:615)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.oracle.bmc.model.BmcException: (-1, null, false) Processing exception while communicating to: https://objectstorage.us-phoenix-1.oraclecloud.com/ (outbound opc-request-id: ASLKASJLAJDLADLAJSDALS)
        at com.oracle.bmc.http.internal.RestClient.convertToBmcException(RestClient.java:578)
        at com.oracle.bmc.http.internal.RestClient.head(RestClient.java:519)
        at com.oracle.bmc.objectstorage.ObjectStorageClient.lambda$null$38(ObjectStorageClient.java:970)
        at com.oracle.bmc.retrier.BmcGenericRetrier.lambda$execute$0(BmcGenericRetrier.java:50)
        at com.oracle.bmc.waiter.GenericWaiter.execute(GenericWaiter.java:54)
        at com.oracle.bmc.retrier.BmcGenericRetrier.execute(BmcGenericRetrier.java:46)
        at com.oracle.bmc.objectstorage.ObjectStorageClient.lambda$headObject$39(ObjectStorageClient.java:966)
        at com.oracle.bmc.retrier.BmcGenericRetrier.lambda$execute$0(BmcGenericRetrier.java:50)
        at com.oracle.bmc.waiter.GenericWaiter.execute(GenericWaiter.java:54)

at com.oracle.bmc.retrier.BmcGenericRetrier.execute(BmcGenericRetrier.java:46)
        at com.oracle.bmc.objectstorage.ObjectStorageClient.headObject(ObjectStorageClient.java:960)
        at com.oracle.bmc.hdfs.store.BmcDataStore.getObjectMetadata(BmcDataStore.java:553)
        ... 25 more
Caused by: shaded.oracle.javax.ws.rs.ProcessingException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection
        at shaded.oracle.org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:287)
        at shaded.oracle.org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:753)
        at shaded.oracle.org.glassfish.jersey.internal.Errors.process(Errors.java:316)
        at shaded.oracle.org.glassfish.jersey.internal.Errors.process(Errors.java:298)
        at shaded.oracle.org.glassfish.jersey.internal.Errors.process(Errors.java:229)
        at shaded.oracle.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414)
        at shaded.oracle.org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:752)
        at shaded.oracle.org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:419)
        at shaded.oracle.org.glassfish.jersey.client.JerseyInvocation$Builder.head(JerseyInvocation.java:383)
        at com.oracle.bmc.http.internal.ForwardingInvocationBuilder.head(ForwardingInvocationBuilder.java:186)
        at com.oracle.bmc.http.internal.RestClient.head(RestClient.java:516)
        ... 35 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection
        at shaded.oracle.org.glassfish.jersey.client.HttpUrlConnectorProvider$DefaultConnectionFactory.getConnection(HttpUrlConnectorProvider.java:300)
        at shaded.oracle.org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:335)
        at shaded.oracle.org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:282)
        at shaded.oracle.org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:278)
        ... 45 more

Please Let me know what could be causing this issue.

hadoop fs -cat doesn't work with Instance Principal authenticator

I have Hadoop 2.6.0-cdh5.15.1, with OCI HDFS driver 2.7.7.0.

I cannot get content of a file on OCI using:

$ hadoop fs -text oci://bdw-demo-bucket@oraclebigdatadb/pocitani_pismen.txt

I writes an error:

text: Unable to fetch file status for: pocitani_pismen.txt

However, I can list the file, so the driver somehow works:

$ hadoop fs -text oci://bdw-demo-bucket@oraclebigdatadb/pocitani_pismen.txt
-rw-rw-rw-   1        838 2019-01-04 18:21 oci://bdw-demo-bucket@oraclebigdatadb/pocitani_pismen.txt

And I can get the file content using OCI Python SDK. So it is not a permission/policies issue.

In core-site.xml, I have:

  <property>
    <name>fs.oci.client.hostname</name>
    <value>https://objectstorage.us-phoenix-1.oraclecloud.com</value>
  </property>
  <property>
    <name>fs.oci.client.custom.authenticator</name>
    <value>com.oracle.bmc.hdfs.auth.InstancePrincipalsCustomAuthenticator</value>
  </property>

When I run the hadoop with more verbose logging:

HADOOP_ROOT_LOGGER="DEBUG,console" hadoop fs -text oci://bdw-demo-bucket@oraclebigdatadb/pocitani_pismen.txt

I can see error:

19/01/08 16:39:52 DEBUG internal.RequestSignerImpl: Could not sign request
java.lang.SecurityException: class "javax.annotation.Nonnull"'s signer information does not match signer information of other classes in the same package

Full console log from the command:
hadoop-fs-text-instance-principal-auth-problem.log

How to use "Authenticator" as described in Oracle document

I followed the steps, however, I never succeed. Just ask question about how it works. I have create dynamic groups, policies, and compartment following docs. Then, what is the next??? I create a VM in that compartment and the VM will automatically have the privatekey File?

BmcDataStore -> rename is not thread safe. throwing java.io.IOException: Unable to perform rename

https://github.com/oracle/oci-hdfs-connector/blob/master/hdfs-connector/src/main/java/com/oracle/bmc/hdfs/store/BmcDataStore.java

BmcDataStore -> rename is not thread safe and throwing IOException. when using spark to store dataset to object storage using this connector we are hitting across this issue. data access to object storage through connector should be thread safe.

Stacktrace:
Stacktrace_2.txt

Code reference from where exception is getting thrown:

private void rename(final String sourceObject, final String destinationObject)
throws IOException {
LOG.debug("Attempting to rename {} to {}", sourceObject, destinationObject);
try {
final String newEntityTag =
new RenameOperation(
this.objectStorage,
this.requestBuilder.renameObject(
sourceObject, destinationObject))
.call();
this.statistics.incrementWriteOps(1); // 1 put
LOG.debug("Newly renamed object has eTag {}", newEntityTag);
} catch (final Exception e) {
LOG.debug("Failed to rename {} to {}", sourceObject, destinationObject, e);
throw new IOException("Unable to perform rename", e); <-------- ERROR line ------>
}
}

ls: Unable to fetch file status for HDFS, connector version: 3.3.0.3

Runing ENV:
1、HDP 3.1.4
2、connector version: 3.3.0.3
3、java version "1.8.0_261"

[root@bigdata-hadoop-1 ~]# hadoop fs -ls oci://bigdata@ocichina001/hello.txt
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/oci-hdfs-full-3.3.0.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/04/09 15:36:08 INFO hdfs.BmcFilesystem: Attempting to initialize filesystem with URI oci://bigdata@ocichina001/hello.txt
21/04/09 15:36:08 INFO hdfs.BmcFilesystem: Initialized filesystem for namespace ocichina001 and bucket bigdata
21/04/09 15:36:08 INFO store.BmcDataStoreFactory: Using connector version: 3.3.0.3
21/04/09 15:36:08 INFO store.BmcDataStoreFactory: Setting retry timeout to 30 seconds
21/04/09 15:36:09 INFO bmc.Services: Registering new service: Services.BasicService(serviceName=OBJECTSTORAGE, serviceEndpointPrefix=objectstorage, serviceEndpointTemplate=https://objectstorage.{region}.{secondLevelDomain})
21/04/09 15:36:11 INFO util.JavaRuntimeUtils: Determined JRE version as Java_8
21/04/09 15:36:11 INFO http.DefaultConfigurator: Setting connector provider to HttpUrlConnectorProvider
21/04/09 15:36:11 INFO http.DefaultConfigurator: Configuring non-buffering for HttpUrlConnectorProvider
21/04/09 15:36:11 INFO store.BmcDataStoreFactory: Using endpoint https://ocichina001.compat.objectstorage.ap-tokyo-1.oraclecloud.com/
21/04/09 15:36:11 INFO objectstorage.ObjectStorageClient: Setting endpoint to https://ocichina001.compat.objectstorage.ap-tokyo-1.oraclecloud.com/
21/04/09 15:36:11 INFO store.BmcDataStore: Using upload configuration: UploadConfiguration(minimumLengthForMultipartUpload=128, lengthPerUploadPart=128, maxPartsForMultipartUpload=10000, enforceMd5BeforeUpload=false, enforceMd5BeforeMultipartUpload=false, allowMultipartUploads=true, allowParallelUploads=true, disableAutoAbort=false)
21/04/09 15:36:11 INFO store.BmcDataStore: Object metadata caching disabled
21/04/09 15:36:11 INFO store.BmcDataStore: fs.oci.caching.object.parquet.enabled is enabled, setting parquet cache spec to 'maximumSize=10240,expireAfterWrite=15m'
21/04/09 15:36:12 INFO hdfs.BmcFilesystem: Setting working directory to oci://bigdata@ocichina001/user/root, and initialized uri to oci://bigdata@ocichina001
21/04/09 15:36:12 INFO store.BmcDataStore: Getting metadata for key: 'hello.txt'
21/04/09 15:36:16 INFO bmc.ClientRuntime: Using SDK: Oracle-JavaSDK/1.35.0
21/04/09 15:36:16 INFO bmc.ClientRuntime: User agent set to: Oracle-JavaSDK/1.35.0 (Linux/4.14.35-1902.305.4.el7uek.x86_64; Java/1.8.0_112; Java HotSpot(TM) 64-Bit Server VM/25.112-b15) Oracle-HDFS_Connector/3.3.0.3
ls: Unable to fetch file status for: hello.txt

Invalid host:port authority in Kerberized cluster

It appears that this connector does not work in clusters where Kerberos authentication is enabled. Failure trace (sensitive details changed) is:
java.lang.IllegalArgumentException: Does not contain a valid host:port authority: bucket@namespace at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) at org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:290) at org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:318) at org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:540) at org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:524) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) at org.apache.hadoop.tools.mapred.CopyOutputFormat.checkOutputSpecs(CopyOutputFormat.java:124) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:270) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:182) at org.apache.hadoop.tools.DistCp.run(DistCp.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:493)
This occurs just after the BmcFilesystem initialization. It seems plausible that overriding the getCanonicalServiceName method from org.apache.hadoop.fs.FileSystem, in order to return a null value could be a potential fix.

To reproduce: Use connector on cluster with hadoop.security.authentication = kerberos

`ArrayIndexOutOfBoundsException` in read-ahead mode

I am getting error when I am using read-ahead mode:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 15639
	at com.oracle.bmc.hdfs.store.BmcReadAheadFSInputStream.read(BmcReadAheadFSInputStream.java:68)
	at java.io.FilterInputStream.read(FilterInputStream.java:83)
	at com.homesoft.spark.SparkClient.testBmcReadAhead(SparkClient.java:114)
	at com.homesoft.spark.SparkClient.main(SparkClient.java:102)

Here is the minimal reproducible test:

        try (final BmcFilesystem filesystem = BmcFilesystemTest.getOOSFilesystem("oci://bucket@tenancy",
                ReadMode.READ_AHEAD,
                (Integer) BmcProperties.READ_AHEAD_BLOCK_SIZE.getDefaultValue());
             final InputStream inputStream = filesystem.open(new Path(filename))) {
            logger.info("Reading file {} byte-by-byte", filename);
            while (-1 != inputStream.read()) ;
            logger.info("Successfully read file {}", filename);
        } catch (IOException | URISyntaxException e) {
            e.printStackTrace();
        }

where

public class BmcFilesystemTest {
...
    public static BmcFilesystem getOOSFilesystem(String storageUri, ReadMode readMode, int readAheadBlockSize) throws IOException, URISyntaxException {
        final BmcFilesystem filesystem = new BmcFilesystem();
        final Configuration configuration = getOOSConfiguration(readMode, readAheadBlockSize);

        filesystem.initialize(new URI(storageUri), configuration);
        return filesystem;
    }
...
    public static Configuration getOOSConfiguration(ReadMode readMode, int readAheadBlockSize) {
        final Configuration configuration = new Configuration();
        configuration.set("fs.oci.client.hostname", "https://objectstorage.us-ashburn-1.oraclecloud.com");
        configuration.set("fs.oci.client.auth.tenantId", "<tenancy OCID>");
        configuration.set("fs.oci.client.auth.userId", "<user OCID>");
        configuration.set("fs.oci.client.auth.fingerprint", "<fingerprint>");
        configuration.set("fs.oci.client.auth.pemfilepath", "<path to the private key>");
        configuration.set("fs.oci.client.auth.passphrase", "");
        configuration.set(BmcConstants.IN_MEMORY_READ_BUFFER_KEY, String.valueOf(ReadMode.IN_MEMORY == readMode));
        configuration.set(BmcConstants.READ_AHEAD_KEY, String.valueOf(ReadMode.READ_AHEAD == readMode));
        configuration.set(BmcConstants.READ_AHEAD_BLOCK_SIZE_KEY, String.valueOf(readAheadBlockSize));

        configuration.set("fs.oci.client.apache.connection.closing.strategy", "immediate"); // to avoid reading entire stream
        return configuration;
    }

---

I am not sure if there is some logic I am not aware of but it looks like the bug is here:

    @Override
    public int read() throws IOException {
        LOG.debug("{}: Reading single byte at position {}", this, filePos);
        if (dataPos == -1) {
            fillBuffer();
        }
        if (dataPos == -1) {
            return -1;
        }
        filePos++;
        return Byte.toUnsignedInt(data[dataCurOffset++]);
    }

Whenever we read byte-by-byte, the data buffer is being filled only once: when dataPos == -1. After that nobody checks if the buffer has to be refilled.

I would suggest the following fix:

    @Override
    public int read() throws IOException {
        LOG.debug("{}: Reading single byte at position {}", this, filePos);
        if (dataPos == -1) {
            fillBuffer();
        }
        if (dataPos == -1) {
            return -1;
        }
        if (data.length == 1 + dataCurOffset) {
          dataPos = -1;
        }
        filePos++;
        return Byte.toUnsignedInt(data[dataCurOffset++]);
    }

Here are the last lines with DEBUG output enabled on the BmcReadAheadFSInputStream:

[2022-04-01 20:41:57,224] com.oracle.bmc.hdfs.store.BmcReadAheadFSInputStream DEBUG - ReadAhead Stream for sample.xlsx: Reading single byte at position 15637
[2022-04-01 20:41:57,225] com.oracle.bmc.hdfs.store.BmcReadAheadFSInputStream DEBUG - ReadAhead Stream for sample.xlsx: Reading single byte at position 15638
[2022-04-01 20:41:57,226] com.oracle.bmc.hdfs.store.BmcReadAheadFSInputStream DEBUG - ReadAhead Stream for sample.xlsx: Reading single byte at position 15639
[2022-04-01 20:42:44,535] com.oracle.bmc.hdfs.store.BmcReadAheadFSInputStream DEBUG - ReadAhead Stream for sample.xlsx: Closing
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 15639
	at com.oracle.bmc.hdfs.store.BmcReadAheadFSInputStream.read(BmcReadAheadFSInputStream.java:68)
	at java.io.FilterInputStream.read(FilterInputStream.java:83)
	at com.homesoft.spark.SparkClient.testBmcReadAhead(SparkClient.java:114)
	at com.homesoft.spark.SparkClient.main(SparkClient.java:102)

I.e. it attempted to read behind the file limits:

$ oci os object head -bn bucket -ns tenancy --name "sample.xlsx"
{
  "accept-ranges": "bytes",
  "access-control-allow-credentials": "true",
  "access-control-allow-methods": "POST,PUT,GET,HEAD,DELETE,OPTIONS",
  "access-control-allow-origin": "*",
  "access-control-expose-headers": "accept-ranges,access-control-allow-credentials,access-control-allow-methods,access-control-allow-origin,content-length,content-md5,content-type,date,etag,last-modified,opc-client-info,opc-client-request-id,opc-request-id,storage-tier,version-id,x-api-id",
  "content-length": "15639",
  "content-md5": "9kPGjn8vsb/iYQpAO92Hnw==",
  "content-type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
  "date": "Sat, 02 Apr 2022 01:46:22 GMT",
  "etag": "b25578a9-c839-4066-b932-4375c9027024",
  "last-modified": "Tue, 15 Mar 2022 21:47:13 GMT",
  "opc-client-request-id": "2204E36376B74F8A9E468294C6F4A7BD",
  "opc-request-id": "iad-1:a0P863M9NBeezUMrut68Ogv9OCJIY3MJiCvFHIMtyirQ98Rb_hpiIji7WTylLeqU",
  "storage-tier": "Standard",
  "version-id": "b934ca25-c585-4d5f-bc49-0bc85bc07199",
  "x-api-id": "native"
}
$

BmcDataStore -> createDirectory is not threadsafe. throwing java.io.IOException: Unable to put object

https://github.com/oracle/oci-hdfs-connector/blob/master/hdfs-connector/src/main/java/com/oracle/bmc/hdfs/store/BmcDataStore.java

BmcDataStore -> createDirectory is not thread safe and throwing IOException. when using spark to store dataset to object storage using this connector we are hitting across this issue. data access to object storage through connector should be thread safe.

Stacktrace:
ExceptionStack_1.txt

Code reference from where exception is getting thrown:

/**
* Creates a pseudo directory at the given path.
*
* @param path
* The path to create a directory object at.
* @throws IOException
* if the operation cannot be completed.
*/
public void createDirectory(final Path path) throws IOException {
// nothing to do for the "root" directory
if (this.isRootDirectory(path)) {
LOG.debug("Root directory specified, nothing to create");
return;
}

       final String key = this.pathToDirectory(path);

       LOG.debug("Attempting to create directory {} with object {}", path, key);

       final ByteArrayInputStream bais = new ByteArrayInputStream(new byte[0]);
       final PutObjectResponse response;
       try {
           response = this.objectStorage.putObject(this.requestBuilder.putObject(key, bais, 0L));
           this.statistics.incrementWriteOps(1);
           LOG.debug("Created directory at {} with etag {}", path, response.getETag());
       } catch (final BmcException e) {
           // if running jobs in parallel, it's possible multiple threads try to
           // create the directory at the same time, which might lead to 409 conflicts.
           // also allowing 412 (even though we don't set im/inm headers) as we
           // basically just want to ensure the placeholder directory object exists
           if (e.getStatusCode() != 409 && e.getStatusCode() != 412) {
               LOG.debug("Failed to create directory for {}", key, e);
               throw new IOException("Unable to put object", e);              <-------- ERROR line ------>
           }
           LOG.debug(
                   "Exception while creating directory, ignoring {} {}",
                   e.getStatusCode(),
                   e.getMessage());
       }
   }

There is a vulnerability in guava 28.0-jre,upgrade recommended

oci-hdfs-connector/hdfs-connector/pom.xml

Lines 258 to 260 in a93631e

	<groupId>com.google.guava</groupId>
	<artifactId>guava</artifactId>
	<version>28.0-jre</version>

CVE-2020-8908

Recommended upgrade version：30.0-jre

Potential data corruption issue for OCI HDFS Connector with RefreshableOnNotAuthenticatedProvider

Update 2020-11-03: We believe we have fixed this issue in version 3.2.1.3. If you are using a RefreshableOnNotAuthenticatedProvider (Instance Principals or Resource Principals, for example), we urge you to update to version 3.2.1.3 or later.

Original issue
If you are using any of the OCI HDFS Connector and you use a RefreshableOnNotAuthenticatedProvider (e.g. InstancePrincipalsCustomAuthenticator, or generally for Resource Principals or Instance Principals) you may be affected by silent data corruption.

Description

When using a RefreshableOnNotAuthenticatedProvider, the OCI Java SDK will attempt to acquire a new security token when receiving a 401 response, which indicates authentication problems, and then automatically retry the request. In this situation, the clients fail to reset the stream position for requests that upload streams. As a result, an empty stream is successfully uploaded, and success may be returned, even though the actual stream content was not uploaded.

Affected and Not Affected Code

It only happens when an authentication details provider that implements RefreshableOnNotAuthenticatedProvider is used, e.g.:

• Affected: InstancePrincipalsCustomAuthenticator
• Affected: InstancePrincipalsAuthenticationDetailsProvider
• Affected: ResourcePrincipalAuthenticationDetailsProvider

It does not happen for other authentication details provider that do not implement said interface, e.g.:

• Not affected: SimpleAuthenticationDetailsProvider
• Not affected: ConfigFileAuthenticationDetailsProvider

Workarounds

We believe we have fixed this issue in OCI Java SDK version 1.25.2. If you are using a RefreshableOnNotAuthenticatedProvider (InstancePrincipalsCustomAuthenticator in particular, or generally Instance Principals or Resource Principals, for example) with the OCI HDFS Connector, we urge you to update to version 1.25.2 of the OCI Java SDK by overriding the version of the OCI Java SDK that the OCI HDFS Connector uses.

We apologize for any inconvenience this issue may cause you. We will release an updated version of the OCI HDFS Connector, which uses the fixed version 1.25.2 of the OCI Java SDK, shortly. Please watch this GitHub issue for updates.

hive external table of OCI object storage , run select count(*) return error : No FileSystem for scheme "oci"

Hadoop version:
1、HDP 3.1.4.0-315
2、Hive 3.1.0
3、hdfs-connector: oci-hdfs-full-3.3.0.7.0.1

logs:
0: jdbc:hive2://bigdata-hadoop-2.sub070606371> show create table ssb_customer_txt_obj;
DEBUG : Acquired the compile lock.
INFO : Compiling command(queryId=hive_20210701170404_946bf166-b352-47aa-98a5-4fd45a04e09d): show create table ssb_customer_txt_obj
DEBUG : Encoding valid txns info 2040:9223372036854775807:: txnid:2040
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:createtab_stmt, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=hive_20210701170404_946bf166-b352-47aa-98a5-4fd45a04e09d); Time taken: 3.461 seconds
INFO : Executing command(queryId=hive_20210701170404_946bf166-b352-47aa-98a5-4fd45a04e09d): show create table ssb_customer_txt_obj
INFO : Starting task [Stage-0:DDL] in serial mode
DEBUG : Task getting executed using mapred tag : hive_20210701170404_946bf166-b352-47aa-98a5-4fd45a04e09d,userid=root
INFO : Completed executing command(queryId=hive_20210701170404_946bf166-b352-47aa-98a5-4fd45a04e09d); Time taken: 1.54 seconds
INFO : OK
DEBUG : Shutting down query show create table ssb_customer_txt_obj
+----------------------------------------------------+
| createtab_stmt |
+----------------------------------------------------+
| CREATE EXTERNAL TABLE ssb_customer_txt_obj( |
| c_custkey int, |
| c_name string, |
| c_address string, |
| c_city string, |
| c_nation string, |
| c_region string, |
| c_phone string, |
| c_mktsegment string) |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' |
| WITH SERDEPROPERTIES ( |
| 'field.delim'='|', |
| 'serialization.format'='|') |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.mapred.TextInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' |
| LOCATION |
| 'oci://bigdata@ocichina001/ssb100_data/customer' |
| TBLPROPERTIES ( |
| 'bucketing_version'='2', |
| 'discover.partitions'='true', |
| 'transient_lastDdlTime'='1625018554') |
+----------------------------------------------------+
24 rows selected (5.365 seconds)

0: jdbc:hive2://bigdata-hadoop-2.sub070606371>
0: jdbc:hive2://bigdata-hadoop-2.sub070606371> select * from ssb_customer_txt_obj limit 2;
DEBUG : Acquired the compile lock.
INFO : Compiling command(queryId=hive_20210701171017_9a358f1f-1c0c-43d0-ad3c-1a55777e821f): select * from ssb_customer_txt_obj limit 2
DEBUG : Encoding valid txns info 2042:9223372036854775807:: txnid:2042
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:ssb_customer_txt_obj.c_custkey, type:int, comment:null), FieldSchema(name:ssb_customer_txt_obj.c_name, type:string, comment:null), FieldSchema(name:ssb_customer_txt_obj.c_address, type:string, comment:null), FieldSchema(name:ssb_customer_txt_obj.c_city, type:string, comment:null), FieldSchema(name:ssb_customer_txt_obj.c_nation, type:string, comment:null), FieldSchema(name:ssb_customer_txt_obj.c_region, type:string, comment:null), FieldSchema(name:ssb_customer_txt_obj.c_phone, type:string, comment:null), FieldSchema(name:ssb_customer_txt_obj.c_mktsegment, type:string, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20210701171017_9a358f1f-1c0c-43d0-ad3c-1a55777e821f); Time taken: 6.69 seconds
INFO : Executing command(queryId=hive_20210701171017_9a358f1f-1c0c-43d0-ad3c-1a55777e821f): select * from ssb_customer_txt_obj limit 2
INFO : Completed executing command(queryId=hive_20210701171017_9a358f1f-1c0c-43d0-ad3c-1a55777e821f); Time taken: 0.342 seconds
INFO : OK
DEBUG : Shutting down query select * from ssb_customer_txt_obj limit 2
+---------------------------------+------------------------------+---------------------------------+------------------------------+--------------------------------+--------------------------------+-------------------------------+------------------------------------+
| ssb_customer_txt_obj.c_custkey | ssb_customer_txt_obj.c_name | ssb_customer_txt_obj.c_address | ssb_customer_txt_obj.c_city | ssb_customer_txt_obj.c_nation | ssb_customer_txt_obj.c_region | ssb_customer_txt_obj.c_phone | ssb_customer_txt_obj.c_mktsegment |
+---------------------------------+------------------------------+---------------------------------+------------------------------+--------------------------------+--------------------------------+-------------------------------+------------------------------------+
| 1 | Customer#000000001 | j5JsirBM9P | MOROCCO 0 | MOROCCO | AFRICA | 25-989-741-2988 | BUILDING |
| 2 | Customer#000000002 | 487LW1dovn6Q4dMVym | JORDAN 1 | JORDAN | MIDDLE EAST | 23-768-687-3665 | AUTOMOBILE |
+---------------------------------+------------------------------+---------------------------------+------------------------------+--------------------------------+--------------------------------+-------------------------------+------------------------------------+
2 rows selected (14.117 seconds)

0: jdbc:hive2://bigdata-hadoop-2.sub070606371> select count() from ssb_customer_txt_obj;
DEBUG : Acquired the compile lock.
INFO : Compiling command(queryId=hive_20210701170416_14defefe-bc86-465d-8a47-b5f16557c5a7): select count() from ssb_customer_txt_obj
DEBUG : Encoding valid txns info 2041:9223372036854775807:: txnid:2041
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20210701170416_14defefe-bc86-465d-8a47-b5f16557c5a7); Time taken: 16.057 seconds
INFO : Executing command(queryId=hive_20210701170416_14defefe-bc86-465d-8a47-b5f16557c5a7): select count() from ssb_customer_txt_obj
INFO : Query ID = hive_20210701170416_14defefe-bc86-465d-8a47-b5f16557c5a7
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
DEBUG : Task getting executed using mapred tag : hive_20210701170416_14defefe-bc86-465d-8a47-b5f16557c5a7,userid=root
INFO : Subscribed to counters: [] for queryId: hive_20210701170416_14defefe-bc86-465d-8a47-b5f16557c5a7
INFO : Tez session hasn't been created yet. Opening session
DEBUG : Adding local resource: scheme: "hdfs" host: "bigdata-hadoop-1.sub07060637140.bddsvcn.oraclevcn.com" port: 8020 file: "/data/tmp/hive/root/_tez_session_dir/d7333501-dddc-4398-bb21-9cd0f3f7d777-resources/orai18n.jar"
DEBUG : Adding local resource: scheme: "hdfs" host: "bigdata-hadoop-1.sub07060637140.bddsvcn.oraclevcn.com" port: 8020 file: "/data/tmp/hive/root/_tez_session_dir/d7333501-dddc-4398-bb21-9cd0f3f7d777-resources/osdt_cert.jar"
DEBUG : Adding local resource: scheme: "hdfs" host: "bigdata-hadoop-1.sub07060637140.bddsvcn.oraclevcn.com" port: 8020 file: "/data/tmp/hive/root/_tez_session_dir/d7333501-dddc-4398-bb21-9cd0f3f7d777-resources/oraclepki.jar"
DEBUG : Adding local resource: scheme: "hdfs" host: "bigdata-hadoop-1.sub07060637140.bddsvcn.oraclevcn.com" port: 8020 file: "/data/tmp/hive/root/_tez_session_dir/d7333501-dddc-4398-bb21-9cd0f3f7d777-resources/xdb.jar"
DEBUG : Adding local resource: scheme: "hdfs" host: "bigdata-hadoop-1.sub07060637140.bddsvcn.oraclevcn.com" port: 8020 file: "/data/tmp/hive/root/_tez_session_dir/d7333501-dddc-4398-bb21-9cd0f3f7d777-resources/hive-hcatalog-core.jar"
DEBUG : Adding local resource: scheme: "hdfs" host: "bigdata-hadoop-1.sub07060637140.bddsvcn.oraclevcn.com" port: 8020 file: "/data/tmp/hive/root/_tez_session_dir/d7333501-dddc-4398-bb21-9cd0f3f7d777-resources/osdt_jce.jar"
DEBUG : Adding local resource: scheme: "hdfs" host: "bigdata-hadoop-1.sub07060637140.bddsvcn.oraclevcn.com" port: 8020 file: "/data/tmp/hive/root/_tez_session_dir/d7333501-dddc-4398-bb21-9cd0f3f7d777-resources/ucp.jar"
DEBUG : Adding local resource: scheme: "hdfs" host: "bigdata-hadoop-1.sub07060637140.bddsvcn.oraclevcn.com" port: 8020 file: "/data/tmp/hive/root/_tez_session_dir/d7333501-dddc-4398-bb21-9cd0f3f7d777-resources/ojdbc8.jar"
DEBUG : Adding local resource: scheme: "hdfs" host: "bigdata-hadoop-1.sub07060637140.bddsvcn.oraclevcn.com" port: 8020 file: "/data/tmp/hive/root/_tez_session_dir/d7333501-dddc-4398-bb21-9cd0f3f7d777-resources/osdt_core.jar"
DEBUG : Adding local resource: scheme: "hdfs" host: "bigdata-hadoop-1.sub07060637140.bddsvcn.oraclevcn.com" port: 8020 file: "/data/tmp/hive/root/_tez_session_dir/d7333501-dddc-4398-bb21-9cd0f3f7d777-resources/osh.jar"
INFO : Dag name: select count() from ssb_customer_txt_obj (Stage-1)
DEBUG : DagInfo: {"context":"Hive","description":"select count() from ssb_customer_txt_obj"}
DEBUG : Setting Tez DAG access for queryId=hive_20210701170416_14defefe-bc86-465d-8a47-b5f16557c5a7 with viewAclString=, modifyStr=root,hive
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1625031757474_0001_1_00, diagnostics=[Vertex vertex_1625031757474_0001_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: ssb_customer_txt_obj initializer failed, vertex=vertex_1625031757474_0001_1_00 [Map 1], org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "oci"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:268)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:239)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
]
ERROR : Vertex killed, vertexName=Reducer 2, vertexId=vertex_1625031757474_0001_1_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1625031757474_0001_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]
ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
INFO : org.apache.tez.common.counters.DAGCounter:
INFO : AM_CPU_MILLISECONDS: 2640
INFO : AM_GC_TIME_MILLIS: 124
ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1625031757474_0001_1_00, diagnostics=[Vertex vertex_1625031757474_0001_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: ssb_customer_txt_obj initializer failed, vertex=vertex_1625031757474_0001_1_00 [Map 1], org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "oci"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:268)

    VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

---

at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:239)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1625031757474_0001_1_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1625031757474_0001_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
DEBUG : Shutting down query select count(*) from ssb_customer_txt_obj

    VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

---

Map 1 container INITIALIZING -1 0 0 -1 0 0
Reducer 2 container INITED 1 0 0 1 0 0

VERTICES: 00/02 [>>--------------------------] 0% ELAPSED TIME: 6.86 s

Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1625031757474_0001_1_00, diagnostics=[Vertex vertex_1625031757474_0001_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: ssb_customer_txt_obj initializer failed, vertex=vertex_1625031757474_0001_1_00 [Map 1], org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "oci"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:268)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:239)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1625031757474_0001_1_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1625031757474_0001_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 (state=08S01,code=2)

Missing fs.oci.buffer.dir

There is a huge among of space needed for the buffer dir during copy process of big files from hdfs to object storage.

The s3 connector has the option to configure the buffer dir (fs.s3.buffer.dir). This is missing for oci I guess.

OCI configurations get ignored in pyspark

I am trying to connect to Object store using the hdfs connector in Pyspark.
I'm setting the oci configuration while creating the spark session object as follows:

SparkSession.builder  \
                   .appName(..) \
                   .master(..) \
                   .jar('spark.jars', "<oci-hdfs-connector-jar>") \
                   .config('fs.oci.client.hostname', '..')  \
                   .config('fs.oci.client.auth.pemfilepath'. '..') \
                   ..

When I try to run this, I get warnings as follows:

Warning: Ignoring non-spark config property: fs.oci.client.hostname=....
Warning: Ignoring non-spark config property: fs.oci.client.auth.fingerprint=....
Warning: Ignoring non-spark config property: fs.oci.client.auth.tenantId=....
Warning: Ignoring non-spark config property: fs.oci.client.auth.pemfilepath=....

How to set the oci configuration in Pyspark?

Add shading logic used to create oci-hdfs-full-<version>.jar

oci-connector-hdfs's pom currently does not have the logic being used to create the uber jar: oci-hdfs-full-version-.jar present in the release assets.
If there are not any concerns, can we please add that shading logic to oci-connector-hdfs's pom as well?

I wanted to shade a couple of more dependencies in that uber-jar.

Full über jar contains unshaded problematic third party libraries

In my particular case, my Spark job is using objectweb ASM and we were getting surprising results.
Eventually tracked down to the Spark classpath containing conflicting versions of the ASM library coming from the oci-hdfs-connector.
The connector has as number of similar third-party library dependencies included that are bound to conflict with Spark applications.

Log4j vulnerability in OCI HDFS Connector

Update 2021-12-20 : We've updated the version of log4j dependencies to 2.17.0 in our latest release of the OCI HDFS Connector (version 3.3.1.0.3.0). The latest version is available via github source/releases and maven to download and use.

---

Update 2021-12-20 : Another vulnerability was discovered in version 2.16.0 of log4j that allows an attacker with control over Thread Context Map data to cause a denial of service when a crafted string is interpreted. This vulnerability has been published as CVE-2021-45105. We're working on releasing a new version of the OCI HDFS Connector with log4j version 2.17.0 that has the fix for the aforementioned vulnerability.

---

Update 2021-12-15 : We've updated the version of log4j dependencies to 2.16.0 in our latest release of the OCI HDFS Connector (version 3.3.1.0.2.0). The latest version is available via github source/releases and maven to download and use.

---

On Thursday (December 9th), a 0-day exploit in the popular Java logging library log4j was discovered that results in Remote Code Execution (RCE) by logging a certain string. The vulnerability has now been published as CVE-2021-44228. This affects all versions of log4j that are <= 2.14.1

Since all the previous versions of the OCI HDFS Connector have a direct dependency on log4j versions <= 2.14.1, this vulnerability can also be exploited in the OCI HDFS Connector. We will soon release a fix for the tool which will upgrade the version of log4j version to 2.17.0, that contains the fix for the aforementioned vulnerability.

Workaround

As a workaround, customers who use the older versions of the OCI HDFS Connector will need to define a dependency on version 2.17.0 of the following packages, in their project pom file, in order to override the older versions of log4j coming from the OCI HDFS Connector :
org.apache.logging.log4j:log4j-core
org.apache.logging.log4j:log4j-slf4j-impl
org.apache.logging.log4j:log4j-1.2-api
org.apache.logging.log4j:log4j

(Comment updated to reflect that CVE-2021-45046 requires upgrading to 2.17.0 or newer.)

Using HDFS connector with latest OCI SDK

I'm trying to use the latest version of the oci-hdfs connector.

If I use it through dependency on oci-hdfs-full and I'm getting following exception:

java.lang.NoSuchMethodError: com.oracle.bmc.objectstorage.model.Bucket.getApproximateSize()Ljava/lang/Long;

This is probably caused by the fact that another part of the project consumes more recent version OCI SDK (1.4.0) which is not compatible and clashes. Any chance of releasing oci-hdfs-full with up-to-date OCI SDK?

Trying to depend on oci-hdfs instead I'm getting:

Caused by: java.lang.NoClassDefFoundError: jersey/repackaged/com/google/common/base/Function
	at shaded.oracle.org.glassfish.jersey.client.ClientConfig$State.<init>(ClientConfig.java:150)
	at shaded.oracle.org.glassfish.jersey.client.ClientConfig.<init>(ClientConfig.java:468)
	at shaded.oracle.org.glassfish.jersey.client.JerseyClientBuilder.<init>
.....
	at shaded.oracle.javax.ws.rs.client.ClientBuilder.newBuilder(ClientBuilder.java:86)
	at com.oracle.bmc.http.internal.RestClientFactory.createClient(RestClientFactory.java:154)
	at com.oracle.bmc.http.internal.RestClientFactory.create(RestClientFactory.java:141)
	at com.oracle.bmc.objectstorage.ObjectStorageClient.<init>(ObjectStorageClient.java:202)

Is additional dependency needed? Which?

Support for Apache Flink

Wanted to know if there is a similar library for integrating Apache flink with OCI Object Storage?

exists()/mkdirs() race in BmcFileBackedOutputStream.java

oci-hdfs-connector/hdfs-connector/src/main/java/com/oracle/bmc/hdfs/store/BmcFileBackedOutputStream.java

Line 65 in aeeac0e

if (!dir.exists() && !dir.mkdirs()) {

There is a race between java.io.File::exists() and java.io.File:mkdirs() in createBufferFile().

First an existence of a directory is checked and if it does not exist, the directory is created.

If two threads are executing this code, following schedule is possible:

1. threadA: exists("/tmp/hadoop-root") returns false
2. threadB: exists("/tmp/hadoop-root") returns false
3. threadA: mkdirs("/tmp/hadoop-root") succeeds
4. threadB: mkdirs("/tmp/hadoop-root") fails and IOException is thrown

BmcDataStore -> renameDirectory fails to rename when a file has special characters(*,$,^,?,+,|)

https://github.com/oracle/oci-hdfs-connector/blob/854920265be7d787a001f66f205a5022da50fe0b/hdfs-connector/src/main/java/com/oracle/bmc/hdfs/store/BmcDataStore.java

We use Spark to store a file to ObjectStore using this connector. This issue occurs when a file/folder has special characters. It is observed for *,$,^,?,+,| but not limited to these.

Analysis:

The renameDirectory() method in BmcDataStore uses the below code to find a new name. String's replaceFIrst method takes the first param as a regex and we need to escape regex characters if it is present in input.

final String newObjectName =
objectToRename.replaceFirst(sourceDirectory, destinationDirectory);

Example to reproduce:
String objectToRename = "test$11/_temporary/0/_temporary/attempt_20210507141458_0001_m_000000_1/";
String sourceDir= "test$11/_temporary/0/_temporary/attempt_20210507141458_0001_m_000000_1/";
String destDir = "test11/_temporary/0/task_20210507141458_0001_m_000000/";
String newName = objectToRename.replaceFirst(sourceDir,destDir);
System.out.println("ObjectToRename:"+objectToRename);
System.out.println("newName:"+newName);

Output:
ObjectToRename: test$11/_temporary/0/_temporary/attempt_20210507141458_0001_m_000000_1/
newName: test$11/_temporary/0/_temporary/attempt_20210507141458_0001_m_000000_1/

Since both ObjectToRename and newName values are same, ObjectStorage rename API was throwing "Caused by: com.oracle.bmc.model.BmcException: (400, SourceNameSameAsNewName, false) Source name should not be the same as new name (opc-request-id: iad-1:OVKmOCJnZquWhV284uYL-g1DlTA0pnW4XUpjhVTrgB0322Ua_hFGxfVkZcOIqxUK)"

Stacktrace

[03:25:24.898 [task-result-getter-0] WARN o.a.spark.scheduler.TaskSetManager - Lost task 0.0 in stage 0.0 (TID 0, 172.18.0.12, executor 0): org.apache.spark.SparkException: Task failed while writing rows.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Unable to perform rename
at com.oracle.bmc.hdfs.store.BmcDataStore.rename(BmcDataStore.java:246)
at com.oracle.bmc.hdfs.store.BmcDataStore.renameDirectory(BmcDataStore.java:228)
at com.oracle.bmc.hdfs.BmcFilesystem.rename(BmcFilesystem.java:468)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:578)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:549)
at org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:50)
at org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:77)
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:225)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:78)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
... 10 more
Caused by: com.oracle.bmc.model.BmcException: (400, SourceNameSameAsNewName, false) Source name should not be the same as new name (opc-request-id: iad-1:JKNpip3rbb03hw5UpYH_0KJtPiYT5jmsYvgl8r0UUfYkoubuT0673UvDlRML1Rdb)
at com.oracle.bmc.http.internal.ResponseHelper.throwIfNotSuccessful(ResponseHelper.java:138)
at com.oracle.bmc.http.internal.ResponseConversionFunctionFactory$ValidatingParseResponseFunction.apply(ResponseConversionFunctionFactory.java:88)
at com.oracle.bmc.http.internal.ResponseConversionFunctionFactory$ValidatingParseResponseFunction.apply(ResponseConversionFunctionFactory.java:84)
at com.oracle.bmc.objectstorage.internal.http.RenameObjectConverter$1.apply(RenameObjectConverter.java:81)
at com.oracle.bmc.objectstorage.internal.http.RenameObjectConverter$1.apply(RenameObjectConverter.java:69)
at com.oracle.bmc.objectstorage.ObjectStorageClient.lambda$null$86(ObjectStorageClient.java:1745)
at com.oracle.bmc.retrier.BmcGenericRetrier.doFunctionCall(BmcGenericRetrier.java:88)
at com.oracle.bmc.retrier.BmcGenericRetrier.lambda$execute$0(BmcGenericRetrier.java:59)
at com.oracle.bmc.waiter.GenericWaiter.execute(GenericWaiter.java:55)
at com.oracle.bmc.retrier.BmcGenericRetrier.execute(BmcGenericRetrier.java:50)
at com.oracle.bmc.objectstorage.ObjectStorageClient.lambda$renameObject$87(ObjectStorageClient.java:1737)
at com.oracle.bmc.retrier.BmcGenericRetrier.doFunctionCall(BmcGenericRetrier.java:88)
at com.oracle.bmc.retrier.BmcGenericRetrier.lambda$execute$0(BmcGenericRetrier.java:59)
at com.oracle.bmc.waiter.GenericWaiter.execute(GenericWaiter.java:55)
at com.oracle.bmc.retrier.BmcGenericRetrier.execute(BmcGenericRetrier.java:50)
at com.oracle.bmc.objectstorage.ObjectStorageClient.renameObject(ObjectStorageClient.java:1731)
at com.oracle.bmc.hdfs.store.RenameOperation.call(RenameOperation.java:38)
at com.oracle.bmc.hdfs.store.BmcDataStore.rename(BmcDataStore.java:241)
... 22 more](url)

OCI HDFS connector in read-ahead mode doesn't emit `bytesRead` input metric

OCI HDFS connector in read-ahead mode doesn't emit bytesRead input metric:

We were using default mode and were reading the number of read rows and read bytes using the following metrics:

• stageCompleted.stageInfo.taskMetrics.inputMetrics.bytesRead
• stageCompleted.stageInfo.taskMetrics.inputMetrics.recordsRead

inside a listener's event override def onStageCompleted(stageCompleted: SparkListenerStageCompleted).

But after enabling "read ahead" mode, the bytesRead metric is always zero.

The content type is not supported

The oci-hdfs-connector doesn't provide possibility to setup content type for data set. For example,

outputDataset.coalesce(1)
 .write().mode(SaveMode.Append)
.format("com.databricks.spark.csv")
 .option("header", "true")
 .option("encoding", "UTF-8")
 .csv(outputFolder);

keep object without content type and it will be application/x-octet-stream (default OCI content).

Using Authenticator

I checked the oci-hdfs-connector code. It still using the 4 tuples to do authentication. User just don't need to setup these 4 tuples in the core-site.xml file. However, in your private BasicAuthenticationDetailsProvider createAuthenticator method, it is still looking for the 4 tuples. It means, these 4 tuples should be setup either in Java environment variables or in other config places.

Object Storage Connection through Proxy

Hi Team,

Environmnet were I run this Connector has to make all outbound call through HTTP Proxy, but I don't see Proxy support in this connector. can you guys help me on this? this is blocking our major release.

Thanks.

How to configure the filesystem implementation in spark configuration

[Issue]
Unable to read the files in OCI Object Storage in Databricks with error

org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "oci"

[Details]
I have configure the oci-hdfs-connector in databricks cluster and configure the auth in spark config

fs.oci.client.custom.clientfs.impl com.oracle.bmc.hdfs.BmcFilesystem
fs.oci.client.auth.tenantId ocid1.tenancy.oc1..aaaaaxxxx
fs.oci.client.auth.userId ocid1.user.oc1..aaaaaaxxxx
fs.oci.client.auth.fingerprint a8:5b:41:70:63:31:b2:8f:5xxx
spark.master local[*, 4]
spark.databricks.cluster.profile singleNode
fs.oci.client.auth.bucketnamespace OCIxxx
fs.oci.client.hostname https://objectstorage.us-phoenix-1.oraclecloud.com
fs.oci.client.auth.pemfilepath /dbfs/FileStore/oci/oci_private.pem

I would love to know how to replace/overwrite the spark config to include the BmcFilesystem and make databricks recognize the oci scheme in Notebook.

Closing stream and read fails, possible stale connection on upgrade from 3.2.1.3 to 3.3.0.7.0.1 with Jersey connector

I've just upgraded from 3.2.1.3 to 3.3.0.7.0.1.

Apache Spark 3.1.2
Hadoop 2.7.4

I've seen our performance degrade significantly on accessing large files from Spark jobs (~ 1G compressed json files).
With the default Apache Connector, the logs contained many partial read and retry errors.
So I switched back to the Jersey HTTPConnector.

With this connector, the following warnings are in the log:

21/07/18 02:28:47 WARN ObjectStorageClient: getObject returns a stream, please make sure to close the stream to avoid any indefinite hangs
21/07/18 02:28:47 WARN ResponseHelper: Wrapping response stream into auto closeable stream, do disable this, pleaseuse ResponseHelper.shouldAutoCloseResponseInputStream(false)
21/07/18 02:32:54 WARN BmcDirectFSInputStream: Read failed, possibly a stale connection. Will re-attempt.
java.io.IOException: Total bytes processed (950272) does not match content-length (485179266)

presto access hive OCI object external table failed. return Unable to extract password

env:
1、presto-server-0.251
2、HDP 3.1.4.0-315
3、hdfs-connector : oci-hdfs-full-3.3.0.7.0.1.jar

logs:

[root@bigdata-hadoop-1 presto-server-0.251]# presto-cli --server bigdata-hadoop-1:8089 --catalog hive --schema ssb100_poc
presto:ssb100_poc> show tables;
Table

customer
customer_orc
dates
dates_orc
lineorder
lineorder_flat
lineorder_flat_orc
lineorder_orc
lineorder_partion_orc
part
part_orc
ssb_customer_orc
ssb_customer_txt_obj
ssb_date_txt_obj
ssb_dates_orc
ssb_lineorder_orc
ssb_part_orc
ssb_supplier_orc
supplier
supplier_orc
(20 rows)

Query 20210701_091214_00001_c45wy, FINISHED, 3 nodes
Splits: 53 total, 53 done (100.00%)
0:19 [20 rows, 651B] [1 rows/s, 35B/s]

presto:ssb100_poc> show create table ssb_customer_txt_obj;
Create Table

CREATE TABLE hive.ssb100_poc.ssb_customer_txt_obj (
c_custkey integer,
c_name varchar,
c_address varchar,
c_city varchar,
c_nation varchar,
c_region varchar,
c_phone varchar,
c_mktsegment varchar
)
WITH (
external_location = 'oci://bigdata@ocichina001/ssb100_data/customer',
format = 'TEXTFILE'
)
(1 row)

Query 20210701_091239_00002_c45wy, FINISHED, 1 node
Splits: 1 total, 1 done (100.00%)
0:04 [0 rows, 0B] [0 rows/s, 0B/s]

presto:ssb100_poc> select * from ssb_customer_txt_obj limit 2;

Query 20210701_091255_00004_c45wy, FAILED, 1 node
Splits: 17 total, 0 done (0.00%)
0:07 [0 rows, 0B] [0 rows/s, 0B/s]

Query 20210701_091255_00004_c45wy failed: Unable to extract password

presto:ssb100_poc> select count(*) from ssb_customer_txt_obj;
Query 20210701_091315_00005_c45wy failed: Unable to extract password

presto:ssb100_poc>

===========================================================================

vi server.log

2021-07-01T17:17:40.396+0800 DEBUG dispatcher-query-11 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is QUEUED
2021-07-01T17:17:40.409+0800 DEBUG dispatcher-query-13 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is WAITING_FOR_RESOURCES
2021-07-01T17:17:40.884+0800 DEBUG dispatcher-query-14 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is PLANNING
2021-07-01T17:17:40.891+0800 DEBUG dispatcher-query-11 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is DISPATCHING
2021-07-01T17:17:40.981+0800 DEBUG query-execution-4 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.1.0 is PLANNED
2021-07-01T17:17:40.989+0800 DEBUG query-execution-7 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is PLANNED
2021-07-01T17:17:41.001+0800 DEBUG dispatcher-query-11 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is STARTING
2021-07-01T17:17:41.001+0800 DEBUG query-execution-7 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.1.0 is SCHEDULING
2021-07-01T17:17:41.031+0800 DEBUG query-execution-7 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is SCHEDULING
2021-07-01T17:17:41.037+0800 DEBUG query-execution-4 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is SCHEDULED
2021-07-01T17:17:41.037+0800 DEBUG dispatcher-query-11 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is RUNNING
2021-07-01T17:17:41.095+0800 DEBUG query-execution-4 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is RUNNING
2021-07-01T17:17:41.138+0800 INFO hive-hive-3 java.io.serialization ObjectInputFilter {0}: {1}, array length: {2}, nRefs: {3}, depth: {4}, bytes: {5}, ex: {6}
2021-07-01T17:17:41.156+0800 DEBUG dispatcher-query-13 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is FAILED
2021-07-01T17:17:41.156+0800 DEBUG query-execution-5 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy failed
com.facebook.presto.spi.PrestoException: Unable to extract password
at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:128)
at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:47)
at com.facebook.presto.hive.util.ResumableTasks.access$000(ResumableTasks.java:20)
at com.facebook.presto.hive.util.ResumableTasks$1.run(ResumableTasks.java:35)
at com.facebook.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Unable to extract password
at com.oracle.bmc.hdfs.store.BmcPropertyAccessor$4.get(BmcPropertyAccessor.java:169)
at com.oracle.bmc.hdfs.store.BmcPropertyAccessor$4.get(BmcPropertyAccessor.java:132)
at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.createAuthenticator(BmcDataStoreFactory.java:362)
at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.buildClient(BmcDataStoreFactory.java:193)
at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.createClient(BmcDataStoreFactory.java:101)
at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.createDataStore(BmcDataStoreFactory.java:88)
at com.oracle.bmc.hdfs.BmcFilesystem.initialize(BmcFilesystem.java:143)
at org.apache.hadoop.fs.PrestoFileSystemCache.createFileSystem(PrestoFileSystemCache.java:149)
at org.apache.hadoop.fs.PrestoFileSystemCache.getInternal(PrestoFileSystemCache.java:89)
at org.apache.hadoop.fs.PrestoFileSystemCache.get(PrestoFileSystemCache.java:62)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at com.facebook.presto.hive.cache.HiveCachingHdfsConfiguration.lambda$getConfiguration$0(HiveCachingHdfsConfiguration.java:68)
at com.facebook.presto.hive.cache.HiveCachingHdfsConfiguration$CachingJobConf.createFileSystem(HiveCachingHdfsConfiguration.java:104)
at org.apache.hadoop.fs.PrestoFileSystemCache.get(PrestoFileSystemCache.java:59)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at com.facebook.presto.hive.HdfsEnvironment.lambda$getFileSystem$0(HdfsEnvironment.java:71)
at com.facebook.presto.hive.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:23)
at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:70)
at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:64)
at com.facebook.presto.hive.StoragePartitionLoader.loadPartition(StoragePartitionLoader.java:163)
at com.facebook.presto.hive.DelegatingPartitionLoader.loadPartition(DelegatingPartitionLoader.java:78)
at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:192)
at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:40)
at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:121)
... 7 more
Caused by: java.io.IOException: Configuration problem with provider path.
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1966)
at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1926)
at com.oracle.bmc.hdfs.store.BmcPropertyAccessor$4.get(BmcPropertyAccessor.java:140)
... 32 more
Caused by: java.io.IOException: Invalid secret key format
at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:850)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.(AbstractJavaKeyStoreProvider.java:106)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:49)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:41)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:100)
at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:71)
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1947)
... 34 more

2021-07-01T17:17:41.157+0800 DEBUG query-execution-4 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.1.0 is ABORTED
2021-07-01T17:17:41.157+0800 DEBUG query-execution-7 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is ABORTED
2021-07-01T17:17:41.160+0800 DEBUG page-buffer-client-callback-14 com.facebook.presto.operator.PageBufferClient Request to http://10.0.0.4:8089/v1/task/async/20210701_091740_00007_c45wy.0.0.0/results/0/0 failed java.util.concurrent.CancellationException: Task was cancelled.
2021-07-01T17:17:41.348+0800 INFO dispatcher-query-13 com.facebook.presto.event.QueryMonitor TIMELINE: Query 20210701_091740_00007_c45wy :: Transaction:[43bc0f84-e357-4636-8e2a-bab7eb6ed6ae] :: elapsed 751ms :: planning 117ms :: scheduling 634ms :: running 0ms :: finishing 634ms :: begin 2021-07-01T17:17:40.396+08:00 :: end 2021-07-01T17:17:41.147+08:00

Problem with certificates

I am currently running spark on a docker container and I have installed the HDFS connector.
My connectivity is behind a gateway that requires a root certificate, the certificate was added to "java/jre/lib/security/cacerts" however when trying to access OCI bucket file, I get the below error:

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323) at sun.security.validator.Validator.validate(Validator.java:271) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:223) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ... 61 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:451) ... 67 more

How to properly configure?