Smart open does not work with S3 buckets containing dots

[ziky90]

Smart open does not work with S3 buckets containing dots

[piskvorky]

Thanks for reporting. Last time I looked, boto3 was unfinished and had many issues.

But if it works better, we could (should) switch to boto3 of course. This will need careful testing (and updating all code) though, as the two libraries are not API-compatible IIRC.

[hobson]

Similar problem for me. Perhaps smart_open isn't being used recursively at the contained objects within models like LsiModel (LsiModel.Projection object is split into two pickles, and smart_open not being used to open them separately). I have no problem opening these pickles with smart_open using the exact URIs given in the warning message.

lsi = LsiModel.load(s3_uri('lsi_model_both_130K'))
WARNING:root:failed to load projection from s3://XYZ@bucket-name/lsi_model_both_130K.projection: [Errno 2] No such file or directory: 's3://XYZ@bucket-name/lsi_model_both_130K.projection.u.npy'

[piskvorky]

That's right, the numpy objects are loaded using numpy's native load operation, which doesn't support s3 or smart_open.

The easiest work-around is to store your object using pickle (a single file) and smart_open, avoiding gensim's load/save that split the object into multiple files. Or pass separately=[] to lsi.save(), that should work too.

[hobson]

Perfect! Hadn't thought of the single-file workaround. Thanks!

[evdoks]

It happens with Python >= 2.7.9. A solution to this issue would be to set calling_format=ProtocolIndependentOrdinaryCallingFormat() when calling smart_open and passing calling_format to boto.connect_s3. Additionally, you need to set S3Connection.DefaultHost to S3 endpoint with region name: e.g.,
S3Connection.DefaultHost='s3-eu-west-1.amazonaws.com'. Since currently smart_open does not support calling_format, having support for it will require a pull request. I could provide one, if needed.

[tmylk]

@evdoks Thanks for the troubleshooting. A PR and a test would be greatly appreciated.

[piskvorky]

@evdoks IIRC, in one of the recent PRs (merged already), we added passing parameters from smart_open directly through to S3.

Does that help with passing calling_format here?

[tmylk]

calling_format explicitly added in #83

[mpenkov]

@ziky90 We replaced boto with boto3 in the S3 subsystem. Could you please check if the problem with dots in bucket names still occurs?

[menshikh-iv]

I close issue because it works fine with bucket name with dots (like com.test-gensim.wow) and current master fbc82cc