Uploading a stream to a block blob extremely slow

[azure]

Uploading a stream to a block blob extremely slow

[XiaoningLiu]

Hi @connor4312

I think main reason for the slow uploading is because BlockBlobURL.upload is not a parallelism uploading, it's a convenience layer uploading method.

For high performance uploading, please use uploadStreamToBlockBlob and other public methods provided in highlevel.ts. They fully support parallel upload.

BTW, how do you know "This indicates that BlockBlobUrl.upload does not consume the file stream? uploadStreamToBlockBlob internally calls into BlockBlobURL.upload to upload chunks.

[connor4312]

how do you know "This indicates that BlockBlobUrl.upload does not consume the file stream? uploadStreamToBlockBlob internally calls into BlockBlobURL.upload to upload chunks.

I can tell this because the Parser stops emitting entry events. In the quoted section of the docs, they say that's how they work. Using .upload causes it to stall, but using uploadStreamToBlockBlob works, and just piping to a temp file on the filesystem also works.

I think main reason for the slow uploading is because BlockBlobURL.upload is not a parallelism uploading, it's a convenience layer uploading method. For high performance uploading, please use uploadStreamToBlockBlob and other public methods provided in highlevel.ts. They fully support parallel upload.

Should this be the way it works? As a naive consumer I would expect blob.upload() to "just work". Instead I got mysterious timeouts, and just randomly tried uploadStreamToBlockBlob because I was looking through Github issues and stumbled upon it mentioned in another issue. It's very non-obvious that attempting parallelism with blob.upload() would fail in this way.

[XiaoningLiu]

Hi @connor4312

Is the timeout error thrown by blobURL.upload? If there are any errors thrown by blobURL.upload, please share the full error message to us. And you can also enable the logging, and share the logs to us for debugging.

  const pipeline = StorageURL.newPipeline(sharedKeyCredential, {
    logger: {
      log: console.log,
      minimumLogLevel: HttpPipelineLogLevel.INFO
    }
  });

At least, you should not use blobUrl.upload in this scenario. Because blobUrl.upload accepts a stream factory method instead of a stream. The stream factory method needs to return a new stream starting from the offset 0 in data source every time. This is because blobUrl.upload will retry when network broken, and it needs a new stream to start a new HTTP request. But method () => file will share the existing file stream. I'm not sure this is the root cause. But I cannot think of other reasons besides the poor network speed, which may stall a upload, because the readable stream is directly passed into https://github.com/axios/axios the underline request module.

parser.on('entry', file => {
  blobUrl.upload(Aborter.none, () => file, file.size, {
    blobHTTPHeaders: {
      blobContentType: mime.getType(file.path)!,
    },
  });
});

We will make the documentation about blobURL.upload and highlevel APIs more clear.