Azure Storage Blob SDK Uploads Only 250MB

[brt5586]

Azure Storage Blob SDK Uploads Only 250MB

[msundman78]

Thanks @pglombardo. After some more digging I found that ActiveStorage is using SAS tokens for DirectUpload, which does not use the x-ms-version header for selecting the API version. Instead the API version is selected when generating the SAS URL in the SignedVersion (sv) parameter, or the optional api-version parameter [1].

The SignedVersion parameter is also defined in the STG_VERSION constant but in the Common module instead of the Blob module.

I also found that in API version 2020-12-06 and higher the Signature format was changed slightly making them incompatible with the current azure-storage-blob implementation [2]. However, all API versions since 2019-12-12 support 5 GB single PUT uploads [3].

Below code in config/initializers/azure.rb using the latest supported API version solves the problem (Tested with a 3GB upload):

require "azure/storage/blob"
module Azure::Storage::Common::Default
  remove_const(:STG_VERSION) if const_defined?(:STG_VERSION)
  const_set(:STG_VERSION, "2020-10-02")
end

[1] https://learn.microsoft.com/en-us/rest/api/storageservices/versioning-for-the-azure-storage-services#authorize-requests-by-using-a-shared-access-signature
[2] https://learn.microsoft.com/en-us/rest/api/storageservices/create-service-sas#version-2020-12-06-and-later
[3] https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob?tabs=microsoft-entra-id#remarks

[github-actions]

Hello @brt5586, thanks for contributing to the Password Pusher community! We will respond as soon as possible.

[pglombardo]

Hi @brt5586 - thanks this is useful info. I'm traveling for the next couple weeks but I'll take a closer look and see if I can make the improvement once I get back.

[pglombardo]

Hi @brt5586 I previously didn't have this documented well but could you confirm that you put in the proper CORS configuration for the Azure storage? One of the CORS configurations is to allow the x-ms-blob-type header.

Updated CORS documentation for Azure is here: https://github.com/pglombardo/PasswordPusher/blob/master/Configuration.md#azure-storage-cors-configuration

[msundman78]

I'm having the same problem, being limited to uploading max 256MB files to Azure Blob Storage due to the old API version being used. According to this thread:
Azure/azure-storage-ruby#172

It should be possible to pass the following setting to force a newer version:
Azure::Storage::Blob::Default::STG_VERSION = "xxxx-xx-xx"

I believe 2024-05-04 is the current version recommended to be used:
https://learn.microsoft.com/en-us/rest/api/storageservices/versioning-for-the-azure-storage-services

Is that anything you can use from PWP?

[pglombardo]

Hi @msundman78 - that's helpful thanks. I dug in a little bit and Azure support doesn't seem to be in great shape. I'll document here for the record.

Password Pusher uses ActiveStorage to upload/store/retrieve files. This is the section on how to configure Azure storage.

It references the azure-storage-blob Ruby gem which hasn't been updated since 2021. :-(

When I go to the source code for that gem, it's been deprecated by Microsoft with no clear alternative.

This is the discussion in the Rails community: rails/rails#49983

@msundman78 you might be able to manually set that value. You could create a file config/initializers/azure.rb with the following content:

require "azure-storage-ruby"
Azure::Storage::Blob::Default::STG_VERSION = "2024-05-04"

[pglombardo]

@msundman78 - that is an excellent investigation and solution! You should also post this in rails/rails#49983 for others in the Rails community that are facing this.

For anyone else reading this thread and using Azure as a backend, could you confirm that the fix above works for you? If so, I'll roll this initializer into the main code base.

Thanks @msundman78!