Allowing S3Filesystem to handle an s3 object for custom s3 configs

[fsspec]

Allowing S3Filesystem to handle an s3 object for custom s3 configs

[mrocklin]

Have you taken a look at the code to S3FileSystem.connect?

    def connect(self, refresh=False):
        """
        Establish S3 connection object.

        Parameters
        ----------
        refresh : bool (True)
            Whether to use cached filelists, if already read
        """
        anon, key, secret, kwargs, ckwargs, token, ssl = \
              (self.anon, self.key, self.secret, self.kwargs,
               self.client_kwargs, self.token, self.use_ssl)

        # Include the current PID in the connection key so that different
        # SSL connections are made for each process.
        tok = tokenize(anon, key, secret, kwargs, ckwargs, token,
                       ssl, os.getpid())
        if refresh:
            self._conn.pop(tok, None)
        logger.debug("Open S3 connection.  Anonymous: %s", self.anon)
        if self.anon:
            # TODO: test addition of kwargs (e.g., S3 data centre)
            from botocore import UNSIGNED
            conf = Config(connect_timeout=self.connect_timeout,
                          read_timeout=self.read_timeout,
                          signature_version=UNSIGNED)
            self.session = boto3.Session(**self.kwargs)
        else:
            conf = Config(connect_timeout=self.connect_timeout,
                          read_timeout=self.read_timeout)
            self.session = boto3.Session(self.key, self.secret, self.token,
                                         **self.kwargs)
        if tok not in self._conn:
            s3 = self.session.client('s3', config=conf, use_ssl=ssl,
                                     **self.client_kwargs)
            self._conn[tok] = s3
        return self._conn[tok]

It might be that what you want to do is pretty doable with modest changes (or perhaps even no changes?) A contribution that allowed this would be very welcome.

[jreback]

#69

this exactly the use case (see the tests for an example)

[jgerardsimcock]

I tried that and keep getting the following:

TypeError: __init__() got an unexpected keyword argument 'client_kwargs'

[jreback]

you need to clone and install
this is in master but not released

[jreback]

@martindurant could do a release at some point

[jgerardsimcock]

Yes. I went through that as well. In my source code, I have that as an option but still getting this error.

[jgerardsimcock]

Thank you. This has been resolved.