putStream timeout

[automattic]

putStream timeout

[domenic]

This has happened to me before when I forget to immediately pipe a stream (or call putStream) after getting it. If you don't do that, you'll miss data events and Amazon will get confused, giving you a 400. This often only shows up in strange cases, like larger files or different uploading clients.

I don't know if that's your exact problem though. It could be our bug, certainly. If you can get a small comparison test case of code that works vs some that doesn't I'll gladly look into it.

[ThiagoMiranda]

exports.awsPut = function(req, res) {
var headers =  {
    'Content-Length' : req.headers['content-length']
};
var amazonReq = knoxClient.putStream(req, '/slideDigest/' + req.query["fileName"], headers, function(err, amazonRes){
    var stringData = "";
    amazonRes.on("data", function(rData){
        stringData += rData;
    });
    amazonRes.on("end", function(){
        console.log(amazonRes.statusCode);
    });

});

amazonReq.on("progress", function(e){
    console.info(e)
});
};

This doesn't works.
In production I have something like that:

    var amazonRequest = client.request('PUT', '/' + dest
        + '?partNumber=' + params.query.partNumber + '&uploadId='
        + params.query.uploadId, {
    'Content-Length' : req.headers['content-length']
});
req.pipe(amazonRequest);

And it works

[ThiagoMiranda]

The strange thing is that on the progress event all files ( the ones that give me a 200 or 400 ) achieve 100%

[domenic]

What is happening to req before you pass it in to awsPut (if anything)?

[ThiagoMiranda]

Nothing. This is just a simple router from the expressjs

[domenic]

I wonder if it has to do with how client.put(...) (which is called by client.putStream) adds Expect: '100-continue'. Note that request('PUT', ...) does not add that. Still not in a position to test this properly for a day or two, but you could try adding Expect: undefined or Expect: "" to your headers hash and see if that fixes things. Very interesting...

[ThiagoMiranda]

Strangely when I've arrived to the office today the problem is gone. I'm able to upload larger files to the Amazon Bucket without any problems. The stranger thing is that my production code doesn't behave like that ( and we have a bunch of clients uploading files larger than 500mb simultaneous ).
Anyway, Thanks for the help and if this happens again I'll keep you posted.

[zhaoda]

I also encountered this problem, my code below

var wstream = client.put(dest, {
  'Content-Length': stat.size,
  'Content-Type': 'image/jpeg',
  'x-amz-acl': 'public-read'
})

wstream.on('error', function(err) {
  cb(err)
})

wstream.on('response', function(res) {
  cb(null, res)
})

wstream.end()

when i delete wstream.end(), it's ok.

[domenic]

That code just immediately closes the stream it tries to put to. That will of course cause a timeout since the connection is waiting for stat.size bytes but you sent it zero bytes.

[zhaoda]

@domenic you are right, thanks.