Using a proxy to access S3

[asieira]

Using a proxy to access S3

[piskvorky]

Hmm, couldn't we do this using by extending the functionality added in #30 ?

[asieira]

Not sure if I understand what you have in mind, but I don't think we can use the same argument here.

On S3OpenWrite, the current usage for **kw is to pass it along to initiate_multipart_upload, not connect_s3: https://github.com/piskvorky/smart_open/blob/master/smart_open/smart_open_lib.py#L402

We could, however, add a new argument to smart_open that is a dict with keyword arguments to be passed along to connect_s3 in a similar way.

[asieira]

Ok, I thought of something simpler and more backwards-compatible. What if smart_open properly handled the case where uri was an instance of boto.s3.key.Key and used its name and bucket properties instead of creating new ones?

This way the calling code would have full control on how it created that Key instance, proxies and all.

[piskvorky]

Hmm, interesting idea. As long as it's backward compatible (strings work the same way), we could add support for other types of input, why not.

Any comments @ziky90 @VincTheSecond ?

[ziky90]

I like the idea. It would be useful for me as well to be able to use proxy parameter.

[asieira]

The challenge is that for this to work, I would change the constructors for both S3OpenRead and S3OpenWrite to receive a boto.s3.key.Key instance, which would not be strictly backwards compatible if anyone was using them directly.

[asieira]

I just submitted #38 with a fully functional version of this idea. I'll still need some help fixing one of the tests, which I temporarily commented out.

[tmylk]

Merged in #38