azure-samples/azure-sdk-for-python-storage-blob-upload-download

How to upload and download blobs from Azure Blob Storage with Python

page_type	languages	products	description	urlFragment
sample	python	azure azure-storage	How to upload and download blobs from Azure Blob Storage with Python.	upload-download-blobs-python

How to upload and download blobs from Azure Blob Storage with Python

This sample shows how to do the following operations of Storage Blobs with Storage SDK

• Create a Storage Account using the Azure Portal.
• Create a container.
• Upload a file to block blob.
• List blobs.
• Download a blob to file.
• Delete a blob.
• Delete the container.

Prerequisites

If you don't have an Azure subscription, create a free account before you begin.

Create a Storage Account using the Azure Portal

Step 1 : Create a new general-purpose Storage Account to use for this tutorial.

• Go to the Azure Portal and log in using your Azure account.
• Select New > Storage > Storage account.
• Select your Subscription.
• For Resource group, create a new one and give it a unique name.
• Enter a name for your storage Account.
• Select the Location to use for your Storage Account.
• Set Account kind to StorageV2(general purpose v2).
• Set Performance to Standard.
• Set Replication to Locally-redundant storage (LRS).
• Set Secure transfer required to Disabled.
• Check Review + create and click Create to create your Storage Account.

Step 2 : Copy and save Connection string.

After your Storage Account is created. Click on it to open it. Select Settings > Access keys > Key1/key, copy the associated Connection string to the clipboard, then paste it into a text editor for later use.

Put the connection string in an environment variable

This solution requires a connection string be stored in an environment variable securely on the machine running the sample. Follow one of the examples below depending on your operating system to create the environment variable. If using Windows close your open IDE or shell and restart it to be able to read the environment variable.

Linux

export AZURE_STORAGE_CONNECTIONSTRING="<YourConnectionString>"

Windows

setx AZURE_STORAGE_CONNECTIONSTRING "<YourConnectionString>"

Set up

First, clone the repository on your machine:

git clone https://github.com/Azure-Samples/azure-sdk-for-python-storage-blob-upload-download.git

Then, install the dependencies:

pip install -r requirements.txt

Finally, execute the following command to run this sample:

python example.py

Use latest Storage SDK

The storage SDK package version here is 2.x.x, if you are using the latest version of the storage SDK package, please reference to the following examples:

• blob_samples_hello_world.py - Examples for common Storage Blob tasks:
  • Create a container
  • Create a block, page, or append blob
  • Upload a file to blob
  • Download a blob
  • Delete a blob
  • Delete the container
• blob_samples_enumerate_blobs.py- Examples to enumerate blobs
  • List blobs.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

azure-sdk-for-python-storage-blob-upload-download's Issues

Create large blob

Please provide us with the following information:

This issue is for a: (mark with an x)

- [X] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Use 'create_blob_from_path' with file more than 100Mb

Any log messages given by the failure

No error logs

Expected/desired behavior

Blob to be created

OS and Version?

Linux Ubuntu 18.04

Versions

Mention any other details that might be useful

To solve issue, max_connection=1 must be added as a parameter.

---

request not authorized

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

If I run the example, I get

azure.common.AzureHttpError: This request is not authorized to perform this operation. ErrorCode: AuthorizationFailure
<?xml version="1.0" encoding="utf-8"?>
<Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.
RequestId:ae1b306b-e01c-004c-645d-fb3536000000
Time:2019-04-25T11:55:55.8180602Z</Message></Error>

API list for Phython SDK

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

I can't find documentation on python sdk's APIs

Any log messages given by the failure

n/a

Expected/desired behavior

having list of API supported by python SDK would increase the productivity.

OS and Version?

n/a

Versions

n/a

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Incorrect padding

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

git clone
python example.py

Any log messages given by the failure

None

Expected/desired behavior

Run or error

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)
Windows 10

Versions

8/11/2019 date which ever version is active

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Does the example require public access?

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [ x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Do what the doc said.

Any log messages given by the failure

I wonder if the storage account require public access. Does public access means allow anonymous read access? I'm confused by the wording in access control.

Exception=Public access is not permitted on this storage account. ErrorCode: PublicAccessNotPermitted

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

doesn't find file

OS: Raspbain Stretch
Python: 3.5.3

I have a strange problem:
if i want upload image1 works...

but if i choose image2 throw exception

Invalid Padding running example.py

This issue is for a:

- [x] bug report
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

export AZURE_STORAGE_CONNECTIONSTRING="........"
git clone https://github.com/Azure-Samples/azure-sdk-for-python-storage-blob-upload-download.git
pip install -r requirements.txt
python example.py

Any log messages given by the failure

$ python example.py 
Incorrect padding

Expected/desired behavior

Runs example stuff.

OS and Version?

$ uname -a
Linux excession 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 GNU/Linux

Versions

$ python --version
Python 3.8.8

$ pip list
Package              Version
-------------------- ---------
azure-common         1.1.27
azure-storage-blob   2.1.0
azure-storage-common 2.1.0
certifi              2021.5.30
cffi                 1.14.6
charset-normalizer   2.0.5
cryptography         3.4.8
idna                 3.2
pip                  20.2.3
pycparser            2.20
python-dateutil      2.8.2
requests             2.26.0
setuptools           49.2.1
six                  1.16.0
urllib3              1.26.6

Mention any other details that might be useful

Same issue as #17 and an error that occurs in other contexts (django-storages)

Here's an example of using BlobBlockService manually in iPython:

In [4]: blob_service_client = BlockBlobService(account_name=AZURE_ACCOUNT_NAME, account_key=AZURE_ACCOUNT_KEY)

In [5]: blob_service_client.exists(AZURE_CONTAINER)
---------------------------------------------------------------------------
Error                                     Traceback (most recent call last)
.../site-packages/azure/storage/common/_auth.py in _add_authorization_header(self, request, string_to_sign)
     68         try:
---> 69             signature = _sign_string(self.account_key, string_to_sign)
     70             auth_string = 'SharedKey ' + self.account_name + ':' + signature

.../site-packages/azure/storage/common/_common_conversion.py in _sign_string(key, string_to_sign, key_is_base64)
     90     if key_is_base64:
---> 91         key = _decode_base64_to_bytes(key)
     92     else:

.../site-packages/azure/storage/common/_common_conversion.py in _decode_base64_to_bytes(data)
     80         data = data.encode('utf-8')
---> 81     return base64.b64decode(data)
     82 

/usr/lib/python3.9/base64.py in b64decode(s, altchars, validate)
     86         raise binascii.Error('Non-base64 digit found')
---> 87     return binascii.a2b_base64(s)
     88 

Error: Incorrect padding

During handling of the above exception, another exception occurred:

AzureSigningError                         Traceback (most recent call last)
<ipython-input-5-7d5053a16991> in <module>
----> 1 blob_service_client.exists(AZURE_CONTAINER)

.../site-packages/azure/storage/blob/baseblobservice.py in exists(self, container_name, blob_name, snapshot, timeout)
   1757             expected_errors = [_CONTAINER_NOT_FOUND_ERROR_CODE] if blob_name is None \
   1758                 else [_CONTAINER_NOT_FOUND_ERROR_CODE, _BLOB_NOT_FOUND_ERROR_CODE]
-> 1759             self._perform_request(request, expected_errors=expected_errors)
   1760 
   1761             return True

.../site-packages/azure/storage/common/storageclient.py in _perform_request(self, request, parser, parser_args, operation_context, expected_errors)
    400                                 client_request_id_prefix,
    401                                 exception_str_in_one_line)
--> 402                     raise ex
    403 
    404                 logger.info("%s Operation failed: checking if the operation should be retried. "

.../site-packages/azure/storage/common/storageclient.py in _perform_request(self, request, parser, parser_args, operation_context, expected_errors)
    372                 except AzureException as ex:
    373                     retry_context.exception = ex
--> 374                     raise ex
    375                 except Exception as ex:
    376                     retry_context.exception = ex

.../site-packages/azure/storage/common/storageclient.py in _perform_request(self, request, parser, parser_args, operation_context, expected_errors)
    312                     try:
    313                         # request can be signed individually
--> 314                         self.authentication.sign_request(request)
    315                     except AttributeError:
    316                         # session can also be signed

.../site-packages/azure/storage/common/_auth.py in sign_request(self, request)
     90             self._get_canonicalized_resource_query(request)
     91 
---> 92         self._add_authorization_header(request, string_to_sign)
     93         logger.debug("String_to_sign=%s", string_to_sign)
     94 

.../site-packages/azure/storage/common/_auth.py in _add_authorization_header(self, request, string_to_sign)
     73             # Wrap any error that occurred as signing error
     74             # Doing so will clarify/locate the source of problem
---> 75             raise _wrap_exception(ex, AzureSigningError)
     76 
     77     def sign_request(self, request):

AzureSigningError: Incorrect padding

from storage.blob import BlockBlobService ModuleNotFoundError: No module named 'storage'

I'm working on python version 3.7.1 and I have already installed the suggested packages:

MacBook-Pro:Downloads $ pip3 freeze | grep azure
azure-common==1.1.17
azure-nspkg==3.0.2
azure-storage==0.36.0
azure-storage-blob==1.4.0
azure-storage-common==1.4.0

However, when I try the import from storage.blob import BlockBlobService I get this weird error about no module called AZURE