Giter Site home page Giter Site logo

azure / azurite Goto Github PK

View Code? Open in Web Editor NEW
1.8K 42.0 317.0 6.18 MB

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies

License: MIT License

TypeScript 98.91% Dockerfile 0.02% JavaScript 0.51% C# 0.41% Go 0.14%
azure-storage blob-storage storage-emulator azurite docker-image cross-platform blob-service

azurite's Issues

Test Failing : Azure-Storage-Node - BlobContainer - createContainer - should work when the headers are set to string type:

Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:201:18

 // creating again will result in a duplicate error
        blobService.createContainer(containerName, function (createError2, container2) {
          assert.equal(createError2.code, Constants.BlobErrorCodeStrings.CONTAINER_ALREADY_EXISTS);
          assert.equal(container2, null);
          blobService.removeAllListeners('sendingRequestEvent');

azure-storage-node tests
base.js:266
BlobContainer
createContainer
should work when the headers are set to string type:
Uncaught AssertionError [ERR_ASSERTION]: undefined == 'ContainerAlreadyExists'
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:201:18
at finalCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:568:7)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\filters\retrypolicyfilter.js:189:13
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:801:17
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:1014:11
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:800:15
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:571:5)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)

Table operation errors

I’ve been playing with Azurite some, and there appears there may be a problem with it’s tables implementation. Regardless of platform, I get errors when trying to add entities, delete entities, and delete tables. Some details:

  • We use azure-storage-node for sending requests.
  • We expect a certain result object when adding entities succeeds. However, we are getting an empty result object. This result object is supposed to provide information about the new entity, such as an etag.
  • When attempting to delete entities or tables, we get errors like the following (may be related to #17, #19):
POST /devstoreaccount1/test01 204 2.216 ms - -
GET /devstoreaccount1/test01?%24top=1000 200 1.560 ms - 575
GET /devstoreaccount1/test01?%24top=1000 200 0.747 ms - 575
POST /devstoreaccount1/$batch 500 1.124 ms - 45
**PANIC** Something unexpected happened! Table Storage Emulator may be in an inconsistent state!
TypeError: Cannot read property 'tableName' of undefined
    at BbPromise.try (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\lib\middleware\table\validation.js:24:48)
    at tryCatcher (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\util.js:16:23)
    at Function.Promise.attempt.Promise.try (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\method.js:39:29)
    at module.exports (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\lib\middleware\table\validation.js:18:18)
    at Layer.handle [as handle_request] (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\router\layer.js:95:5)
    at trim_prefix (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\router\index.js:317:13)
    at C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\router\index.js:284:7
    at Function.process_params (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\router\index.js:335:12)
    at next (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\router\index.js:275:10)
    at C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\body-parser\lib\read.js:130:5
    at invokeCallback (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\raw-body\index.js:224:16)
    at done (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\raw-body\index.js:213:7)
    at IncomingMessage.onEnd (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\raw-body\index.js:273:7)
    at emitNone (events.js:106:13)
   at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1055:12)
GET /devstoreaccount1/test01?%24top=1000 200 0.869 ms - 575

Test Failing : Azure-Storage-Node - BlobContainer - listBlobs - shoud work

Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:709:26

// Test listing multiple blobs
              listBlobsWithoutPrefix(null, null, function() {
                assert.equal(blobs.length, 2);

                var entries = 0;
                blobs.forEach(function (blob) {
                  assert.notEqual(blob.creationTime, null);
                  assert.deepStrictEqual(typeof blob.creationTime, 'string');

                  if (blob.name === blobName1) {
                    entries += 1;
                  }
                  else if (blob.name === blobName2) {
                    entries += 2;
                  }
                });

                assert.equal(entries, 3);

azure-storage-node tests
base.js:266
BlobContainer
listBlobs
should work:
Uncaught AssertionError [ERR_ASSERTION]: undefined != null
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:709:26
at Array.forEach ()
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:708:23
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:682:13
at finalCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:5824:7)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\filters\retrypolicyfilter.js:189:13
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:801:17
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:1014:11
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:800:15
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:5827:5)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)

`blobPort` option doesn't work

  1. Launch Azurite with blobPort=10100
  2. Observe output

Expected:
"Azure Blob Storage Emulator listening on port 10100"

Actual:
"Azure Blob Storage Emulator listening on port 10000"

PS C:\Users\cralvord> azurite -l C:\Users\cralvord\azurite\ -blobPort=10100 -queuePort=10101 -tablePort=10102

 _______                   _
(_______)                 (_)  _
 _______ _____ _   _  ____ _ _| |_ _____
|  ___  (___  ) | | |/ ___) (_   _) ___ |
| |   | |/ __/| |_| | |   | | | |_| ____|
|_|   |_(_____)____/|_|   |_| \__)_____)

Azurite, Version 2.6.5
A lightweight server clone of Azure Storage

Azure Table Storage Emulator listening on port 10102
Azure Queue Storage Emulator listening on port 10101
Azure Blob Storage Emulator listening on port 10000

Table Filters Returning No Results

I'm using Azurite, Version 2.6.5 on Mac OSX

I perform the following request and I receive all results from the database:

GET /devstoreaccount1/Signposts HTTP/1.1
Host: 127.0.0.1:10002
Authorization: SharedKey devstoreaccount1:Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
x-ms-date: 1527335502
Content-Type: application/json
x-ms-version: 2017-07-29
Cache-Control: no-cache
Postman-Token: 463d93a9-df19-49d1-8e59-eaf509432d41

{
    "value": [
        {
            "Timestamp": "2018-04-19T12:20:36.776Z",
            "title": "Benefits - GOV.UK",
            "description": "The best place to find government services and information. Simpler, clearer, faster",
            "url": "https://www.gov.uk/browse/benefits",
            "image": "/static/images/govuk.png",
            "categories": "Benefits",
            "PartitionKey": "Signposts",
            "RowKey": "574d5f400d3e106660158ac3"
        },
        {
            "Timestamp": "2018-04-19T12:20:36.791Z",
            "title": "Business & Self-Employment - GOV.UK",
            "description": "The best place to find government services and information. Simpler, clearer, faster",
            "url": "https://www.gov.uk/browse/business",
            "image": "/static/images/govuk.png",
            "categories": "Business",
            "PartitionKey": "Signposts",
            "RowKey": "574d5f400d3e106660158ac4"
        }
    ]
}

If I submit a request with a query (in this case on the PartitionKey) I get no results:

GET /devstoreaccount1/Signposts?$filter=PartitionKey%20eq%20%27Signposts%27 HTTP/1.1
Host: 127.0.0.1:10002
Authorization: SharedKey devstoreaccount1:Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
x-ms-date: 1527335659
Content-Type: application/json
x-ms-version: 2017-07-29
Cache-Control: no-cache
Postman-Token: ae88e955-2304-409d-820f-60e5b0a7c6d3

{
    "value": []
}

This query was replayed via postman but I first tried the query via the Micosoft.WindowsAzure.Storage library which gave the same results.

Is the query wrong or is this a bug in Azurite?

Thanks

Simon

Requesting queue policies failed with azure-cli and Storage Account Explorer

Hi,

Currently I’m not able to list queue policies with azure cli and I’m also not able to list queue policies with the Storage Account explorer. But the policies will be created successfully.

I recognized that the storage account emulator is answering the get request different than azurite. There are some magic characters in front of the response and a zero at the end.

Storage Account Emulator 5.3.0.0
Azurite 2.6.5
Azure CLI 2.0.30
Windows 10 + WSL

1. Storage Account Emulator + azure cli

az storage queue create --name test-queue
az storage queue policy list --queue-name test-queue

RESPONSE (Fiddler):

3E
<?xml version="1.0" encoding="utf-8"?><SignedIdentifiers />
0
az storage queue policy create --name read --queue-name test-queue --permissions rp --start 2018-04-01 --expiry 2018-04-02
az storage queue policy list --queue-name test-queue

RESPONSE (Fiddler):

F7
<?xml version="1.0" encoding="utf-8"?><SignedIdentifiers><SignedIdentifier><Id>read</Id><AccessPolicy><Start>2018-04-01T00:00:00.0000000Z</Start><Expiry>2018-04-02T00:00:00.0000000Z</Expiry><Permission>rp</Permission></AccessPolicy></SignedIdentifier></SignedIdentifiers>
0

2. Azurite + azure cli

az storage queue create --name test-queue
az storage queue policy list –queue-name test-queue

RESPONSE (Fiddler):

<?xml version="1.0" encoding="utf-8"?>
<SignedIdentifiers/>
az storage queue policy create --name read --queue-name test-queue --permissions rp --start 2018-04-01 --expiry 2018-04-02
az storage queue policy list --queue-name test-queue

azure cli returns nothing. But the response from azurite looks like:
RESPONSE (Fiddler):

<?xml version="1.0" encoding="utf-8"?>
<SignedIdentifiers>
    <SignedIdentifiers>
        <Id>read</Id>
        <AccessPolicy>
            <Start>2018-04-01</Start>
            <Expiry>2018-04-02</Expiry>
            <Permission>rp</Permission>
        </AccessPolicy>
    </SignedIdentifiers>
</SignedIdentifiers>

As you can see the magic characters in front of the response are not there and the trailing zero is also not available.

Another issue because of the issue above is that azure cli requests the queue policies first, if there are some policies available it merges the policies (old+new) and sends them back. But this is not possible with azurite. All policies will be overwritten by the new one.

Currently I circumventing the issue by using custom scripts for creating queues and policies but for debugging and some test environment related topics I need to use azure cli and the Storage Account Explorer.

Best
nobiehl

Test Failing : Azure-Storage-Node - BlobContainer - setContainerAcl - should work with policies

Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:586:01

Assuming that it is caused by either failing set or get container ACL code...

blobService.setContainerAcl(containerName, signedIdentifiers, options, function (setAclError, setAclContainer1, setResponse1) {
        assert.equal(setAclError, null);
        assert.notEqual(setAclContainer1, null);
        assert.ok(setResponse1.isSuccessful);
        
        setTimeout(function () {
          blobService.getContainerAcl(containerName, function (getAclError, getAclContainer1, getResponse1) {
            assert.equal(getAclError, null);
            assert.notEqual(getAclContainer1, null);
            assert.equal(getAclContainer1.publicAccessLevel, BlobUtilities.BlobContainerPublicAccessType.BLOB);
            assert.equal(getAclContainer1.signedIdentifiers.readwrite.Expiry.getTime(), readWriteExpiryDate.getTime());
            assert.ok(getResponse1.isSuccessful);
            
            options.publicAccessLevel = BlobUtilities.BlobContainerPublicAccessType.CONTAINER;
            blobService.setContainerAcl(containerName, null, options, function (setAclError2, setAclContainer2, setResponse2) {
              assert.equal(setAclError2, null);
              assert.notEqual(setAclContainer2, null);
              assert.ok(setResponse2.isSuccessful);

azure-storage-node tests
base.js:266
BlobContainer
setContainerAcl
should work with policies:
Uncaught TypeError: Cannot read property 'split' of undefined
at Object.exports.parse (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\util\iso8061date.js:48:22)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\models\aclresult.js:105:44
at Array.forEach ()
at Object.exports.parse (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\models\aclresult.js:101:26)
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:874:68)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)

EACCES error on startup

Azurite Version: 2.6.5
OS Version: Windows 10

I get a continuous stream of error outputs when running Azurite from a PowerShell prompt (see below). However, if I run PowerShell as administrator and allow Node.js through the firewall, everything appears to work normally.

Sounds like a potential security risk.

PS> azurite

 _______                   _
(_______)                 (_)  _
 _______ _____ _   _  ____ _ _| |_ _____
|  ___  (___  ) | | |/ ___) (_   _) ___ |
| |   | |/ __/| |_| | |   | | | |_| ____|
|_|   |_(_____)____/|_|   |_| \__)_____)

Azurite, Version 2.6.5
A lightweight server clone of Azure Storage

events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: listen EACCES 0.0.0.0:10002
    at Object._errnoException (util.js:1022:11)
    at _exceptionWithHostPort (util.js:1044:20)
    at Server.setupListenHandle [as _listen2] (net.js:1334:19)
    at listenInCluster (net.js:1392:12)
    at Server.listen (net.js:1476:7)
    at Function.listen (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\application.js:618:24)
    at env.init.then.then (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\lib\AzuriteTable.js:52:35)
    at tryCatcher (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\util.js:16:23)
    at Promise._settlePromiseFromHandler (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:512:31)
    at Promise._settlePromise (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:569:18)
    at Promise._settlePromise0 (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:614:10)
    at Promise._settlePromises (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:693:18)
    at Promise._fulfill (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:638:18)
    at C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\nodeback.js:42:21
    at Timeout._onTimeout (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\lokijs\src\lokijs.js:2633:15)
    at ontimeout (timers.js:475:11)
events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: listen EACCES 0.0.0.0:10001
    at Object._errnoException (util.js:1022:11)
    at _exceptionWithHostPort (util.js:1044:20)
    at Server.setupListenHandle [as _listen2] (net.js:1334:19)
    at listenInCluster (net.js:1392:12)
    at Server.listen (net.js:1476:7)
    at Function.listen (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\application.js:618:24)
    at env.init.then (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\lib\AzuriteQueue.js:48:35)
    at tryCatcher (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\util.js:16:23)
    at Promise._settlePromiseFromHandler (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:512:31)
    at Promise._settlePromise (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:569:18)
    at Promise._settlePromise0 (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:614:10)
    at Promise._settlePromises (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:693:18)
    at Promise._fulfill (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:638:18)
    at C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\nodeback.js:42:21
    at xfs.stat (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\fs-extra\lib\mkdirs\mkdirs.js:56:16)
    at C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\graceful-fs\polyfills.js:287:18
events.js:183
      throw er; // Unhandled 'error' event
      ^
...

I noticed that if I run PowerShell as administrator and allow Node.js through the firewall, everything appears to work normally.

Incorrect blob type when fetching the copy status

Following code uploads a 16MB binary and then copies it on itself:

CloudStorageAccount cloud = CloudStorageAccount.parse("DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;");
CloudBlobContainer container = cloud.createCloudBlobClient().getContainerReference("test");
container.deleteIfExists();
container.create();

for (int i = 0; i < 100; i++) {
    CloudBlockBlob blob = container.getBlockBlobReference("test" + i);
    System.out.print("Uploading: " + blob.getName() + "... ");
    blob.uploadFromByteArray(new byte[16 * 1024 * 1024], 0, 16 * 1024 * 1024);
    System.out.println("Done");

    blob.startCopy(blob);
    CopyStatus status;
    while (true) {
        blob.downloadAttributes();
        status = blob.getCopyState().getStatus();
        if (status != CopyStatus.PENDING) {
            break;
        }
        Thread.sleep(200);
    }
    Assert.assertEquals(CopyStatus.SUCCESS, status);
}

Following exception is thrown randomly after a few iterations:

com.microsoft.azure.storage.StorageException: Incorrect Blob type, please use the correct Blob type to access a blob on the server. Expected BLOCK_BLOB, actual UNSPECIFIED.

	at com.microsoft.azure.storage.blob.CloudBlob$8.preProcessResponse(CloudBlob.java:1238)
	at com.microsoft.azure.storage.blob.CloudBlob$8.preProcessResponse(CloudBlob.java:1204)
	at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:139)
	at com.microsoft.azure.storage.blob.CloudBlob.downloadAttributes(CloudBlob.java:1197)
	at com.microsoft.azure.storage.blob.CloudBlob.downloadAttributes(CloudBlob.java:1164)
	at org.apache.jackrabbit.oak.segment.azure.LoopFailTest.test(LoopFailTest.java:28)

Tag latest releases

The latest release on npm is v2.6.5, but on GitHub the last tag was v2.6.3. Could you please tag the 2 missing versions on Github?

Blob container no longer accessible

Azurite 2.6.5
OSX 10.13.3

Summary: Blob container which was once accessible from both python drivers (from a django app) as well as the azure storage explorer is now throwing 404s for images previously accessible. Also blob container itself doesn't even show up in the Storage Explorer.

When Azurite is running the requests to the container now 404.

GET /devstoreaccount1/test/images/product-picture-system-overview-1200x900_P2Xd.height-150.png 404 0.807 ms - 131
GET /devstoreaccount1/test/images/buddy-logo.original.png 404 0.752 ms - 131
GET /devstoreaccount1/test/images/buddy_bg.original.png 404 0.526 ms - 131

The Azure Storage explorer which once worked now does not show the container.
In the folder I use to store data I can see the various json configs. as well as the __blobstorage__ folder with my files in it. The __azurite_db_blob__.json does indeed have my container config.

{
    "filename": "azureStorage/__azurite_db_blob__.json",
    "collections": [{
        "name": "Containers",
        "data": [{
            "metaProps": {},
            "entityType": "Container",
            "leaseState": "available",
            "access": "private",
            "name": "test",
            "etag": "hofpfmX+/GTL3Uk/7EKWIm16ETU",
            "meta": {
                "revision": 0,
                "created": 1519943461794,
                "version": 0
            },
            "$loki": 1
        }],
        "idIndex": [1],
        "binaryIndices": {},
        "constraints": null,
        "uniqueNames": [],
        "transforms": {},
        "objType": "Containers",
        "dirty": false,
        "cachedIndex": null,
        "cachedBinaryIndex": null,
        "cachedData": null,
        "adaptiveBinaryIndices": true,
        "transactional": false,
        "cloneObjects": false,
        "cloneMethod": "parse-stringify",
        "asyncListeners": false,
        "disableMeta": false,
        "disableChangesApi": true,
        "disableDeltaChangesApi": true,
        "autoupdate": false,
        "serializableIndices": true,
        "ttl": null,
        "maxId": 1,
        "DynamicViews": [],
        "events": {
            "insert": [null],
            "update": [null],
            "pre-insert": [],
            "pre-update": [],
            "close": [],
            "flushbuffer": [],
            "error": [],
            "delete": [null],
            "warning": [null]
        },
        "changes": []
    }, {
        "name": "ServiceProperties",
        "data": [],
        "idIndex": [],
        "binaryIndices": {},
        "constraints": null,
        "uniqueNames": [],
        "transforms": {},
        "objType": "ServiceProperties",
        "dirty": false,
        "cachedIndex": null,
        "cachedBinaryIndex": null,
        "cachedData": null,
        "adaptiveBinaryIndices": true,
        "transactional": false,
        "cloneObjects": false,
        "cloneMethod": "parse-stringify",
        "asyncListeners": false,
        "disableMeta": false,
        "disableChangesApi": true,
        "disableDeltaChangesApi": true,
        "autoupdate": false,
        "serializableIndices": true,
        "ttl": null,
        "maxId": 0,
        "DynamicViews": [],
        "events": {
            "insert": [null],
            "update": [null],
            "pre-insert": [],
            "pre-update": [],
            "close": [],
            "flushbuffer": [],
            "error": [],
            "delete": [null],
            "warning": [null]
        },
        "changes": []
    }, {
        "name": "test",
        "data": [{
            "metaProps": {},
            "entityType": "BlockBlob",
            "leaseState": "available",
            "access": "private",
            "name": "original_images/bundle-enterprise.png",
            "id": "QXRlc3RvcmlnaW5hbF9pbWFnZXMvYnVuZGxlLWVudGVycHJpc2UucG5n",
            "uri": "azureStorage/__blobstorage__/zZYc_Gf94S_K5DBYFoUEcw9qdT8=",
            "snapshot": false,
            "committed": true,
            "size": 28311,
            "etag": "cy4zKx42DoteaPpGlJb9eQEsDO8",
            "contentType": "image/png",
            "meta": {
                "revision": 1,
                "created": 1519943666535,
                "version": 0,
                "updated": 1519943666538
            },
            "$loki": 1
        },

No idea how to troubleshoot this.

createTableIfNotExists fails with TableNotFound when table does not exist

I'm using azure/azure-storage-node.
This is the code that fails (in azure/azure-storage-node)

There is the check

else if (responseObject.error && responseObject.error.statusCode === Constants.HttpConstants.HttpResponseCodes.NotFound) {
      responseObject.error = null;
      responseObject.tableResult.exists = false;
      responseObject.response.isSuccessful = true;
    }

But the responseObject is as following:

{ error:
   { StorageError: <?xml version="1.0" encoding="utf-8"?><Error><Code>TableNotFound</Code><Message>The table specified does not exist.</Message></Error>
    at Function.StorageServiceClient._normalizeError (/Users/alexander/Development/Sandbox/azurite-check/node-service/node_modules/azure-storage/lib/common/services/storageserviceclient.js:1153:12)
    at TableService.StorageServiceClient._processResponse (/Users/alexander/Development/Sandbox/azurite-check/node-service/node_modules/azure-storage/lib/common/services/storageserviceclient.js:738:50)
    at Request.processResponseCallback [as _callback] (/Users/alexander/Development/Sandbox/azurite-check/node-service/node_modules/azure-storage/lib/common/services/storageserviceclient.js:311:37)
    at Request.self.callback (/Users/alexander/Development/Sandbox/azurite-check/node-service/node_modules/request/request.js:186:22)
    at emitTwo (events.js:126:13)
    at Request.emit (events.js:214:7)
    at Request.<anonymous> (/Users/alexander/Development/Sandbox/azurite-check/node-service/node_modules/request/request.js:1163:10)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at IncomingMessage.<anonymous> (/Users/alexander/Development/Sandbox/azurite-check/node-service/node_modules/request/request.js:1085:12)
     name: 'StorageError',
     message: '<?xml version="1.0" encoding="utf-8"?><Error><Code>TableNotFound</Code><Message>The table specified does not exist.</Message></Error>' },
  response:
   { isSuccessful: false,
     statusCode: 404,
     body: '<?xml version="1.0" encoding="utf-8"?><Error><Code>TableNotFound</Code><Message>The table specified does not exist.</Message></Error>',
     headers:
      { 'x-powered-by': 'Express',
        'content-type': 'text/html; charset=utf-8',
        'content-length': '133',
        etag: 'W/"85-QChnKDYjVl8qDySHFUJOL2CNEFU"',
        date: 'Tue, 10 Apr 2018 08:09:42 GMT',
        connection: 'keep-alive' },
     md5: undefined },
  contentMD5: undefined,
  length: undefined,
  operationEndTime: 2018-04-10T08:09:39.194Z,
  targetLocation: 0,
  outputStreamSent: false,
  tableResult: { isSuccessful: false, statusCode: 404, TableName: 'mytable' } }

So responseObject.error.statusCode is undefined.

Test Failing : Azure-Storage-Node - BlobContainer - createContainerIfNotExists - should create a container if not exists:

Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:236:24

blobService.createContainerIfNotExists(containerName, function (createError3) {
                assert.notEqual(createError3, null);
                assert.equal(createError3.code, 'ContainerBeingDeleted');
                done();
              });

azure-storage-node tests
base.js:266
BlobContainer
createContainerIfNotExists
should create a container if not exists:
Uncaught AssertionError [ERR_ASSERTION]: null != null
+ expected - actual
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:236:24
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:642:9
at finalCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:568:7)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\filters\retrypolicyfilter.js:189:13
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:801:17
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:1014:11
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:800:15
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:571:5)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)

Unable to connect Azurite using azure javascript library

Hi,

I am trying to connect azurite using azure javascript library.

When I make a get request to any container, the browser tries to send preflight request which is failed with error code 501-not implemented. Even when I make a options request using fiddler, it fails with same error.

What configuration I need to do to make it work.

Help request: How to setup Azurite HA?

I was wondering that how others have been solved high availability on their Azurite deployments?

Old skool way mapping storage from NAS to two nodes most probably works but then storage system it selves is SPOF.

On Azure they are using data replication but which is good/tested option for it wilt Azurite?

*PANIC* when looking for non-existent row

There seems to be an inconsistency in how this emulator handles non-existent rows compared to how "real" Azure Table Storage handles them. I discovered this while using Streamstone, which builds event streaming capabilities on top of Azure Table Storage, and while I haven't been able to discern exactly what operations they're doing on the table storage, I have been able to build a minimal example that reproduces the problem.

To reproduce, make sure you have the dotnet CLI installed, and run

mkdir repro && cd repro
dotnet new console
dotnet add package Streamstone

Then replace the contents of Program.cs with this:

using System;
using System.Threading.Tasks;
using Microsoft.WindowsAzure.Storage;
using Streamstone;

namespace AzuritePanicOnStreamstoneTryOpenStream
{
    class Program
    {
        public static async Task Main(string[] args)
        {
            // Replace with CloudStorageAccount.Parse("your connection string"); for using a real Azure Storage Account
            var account = CloudStorageAccount.DevelopmentStorageAccount;

            var table = account
                .CreateCloudTableClient()
                .GetTableReference("Events");
            await table.CreateIfNotExistsAsync();

            var partition = new Partition(table, "foo");

            await Stream.TryOpenAsync(partition);
        }
    }
}

Finally, run dotnet run to see the program fail with

Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: Unexpected response code, Expected:OK or NotFound, Received:InternalServerError
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Core\Executor\Executor.cs:line 315
   at Streamstone.Stream.OpenStreamOperation.ExecuteAsync()
   at AzuritePanicOnStreamstoneTryOpenStream.Program.Main(String[] args) in C:\Work\OSS\Throwaway repros\AzuritePanicOnStreamstoneTryOpenStream\Program.cs:line 17
   at AzuritePanicOnStreamstoneTryOpenStream.Program.<Main>(String[] args)

If I look at the logs for my docker container running Azurite, I see

GET /devstoreaccount1/Events(PartitionKey='foo',RowKey='SS-HEAD') 500 0.388 ms - 27
**PANIC** Something unexpected happened! Table Storage Emulator may be in an inconsistent state!
ReferenceError: partitionKey is not defined
    at TableStorageManager.queryEntities (/opt/azurite/lib/core/table/TableStorageManager.js:104:40)
    at QueryEntities.process (/opt/azurite/lib/actions/table/QueryEntities.js:12:29)
    at Object.actions.(anonymous function) [as QueryEntity] (/opt/azurite/lib/middleware/table/actions.js:52:19)
    at BbPromise.try (/opt/azurite/lib/middleware/table/actions.js:18:38)
    at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/opt/azurite/node_modules/bluebird/js/release/method.js:39:29)
    at module.exports (/opt/azurite/lib/middleware/table/actions.js:17:18)
    at Layer.handle [as handle_request] (/opt/azurite/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/opt/azurite/node_modules/express/lib/router/index.js:317:13)
    at /opt/azurite/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/opt/azurite/node_modules/express/lib/router/index.js:335:12)
    at next (/opt/azurite/node_modules/express/lib/router/index.js:275:10)
    at BbPromise.try (/opt/azurite/lib/middleware/table/validation.js:32:9)
    at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/opt/azurite/node_modules/bluebird/js/release/method.js:39:29)
    at module.exports (/opt/azurite/lib/middleware/table/validation.js:18:18)

However, if I connect to a real Azure Storage Account instead of the emulator, the program runs without error.

Test Failing : Azure-Storage-Node - BlobContainer - getContainerProperties - should work

Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:317:20

blobService.getContainerProperties(containerName, function (getError, container2, getResponse) {
          assert.equal(getError, null);
          assert.notEqual(container2, null);
          if (container2) {
            assert.equal('unlocked', container2.lease.status);
            assert.equal('available', container2.lease.state);
            assert.equal(null, container2.lease.duration);
            assert.notEqual(null, container2.requestId);
            assert.notEqual(container2.hasImmutabilityPolicy, null);
            assert.deepStrictEqual(typeof container2.hasImmutabilityPolicy, 'boolean');
            assert.notEqual(container2.hasLegalHold, null);
            assert.deepStrictEqual(typeof container2.hasLegalHold, 'boolean');

azure-storage-node tests

   BlobContainer
     getContainerProperties
       should work:
 Uncaught AssertionError [ERR_ASSERTION]: undefined != null
  at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:317:20
  at finalCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:698:7)
  at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\filters\retrypolicyfilter.js:189:13
  at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:801:17
  at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:1014:11
  at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:800:15
  at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:701:5)
  at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
  at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
  at Request.<anonymous> (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
  at IncomingMessage.<anonymous> (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
  at endReadableNT (_stream_readable.js:1045:12)
  at _combinedTickCallback (internal/process/next_tick.js:138:11)
  at process._tickCallback (internal/process/next_tick.js:180:9)

Test Failing : Azure-Storage-Node - BlobContainer - setContainerAcl - should work

Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:516:18

var options = {publicAccessLevel: BlobUtilities.BlobContainerPublicAccessType.BLOB};
        blobService.setContainerAcl(containerName, null, options, function (setAclError, setAclContainer1, setResponse1) {
          assert.equal(setAclError, null);
          assert.notEqual(setAclContainer1, null);
          assert.ok(setResponse1.isSuccessful);

azure-storage-node tests
base.js:266
BlobContainer
setContainerAcl
should work:
Uncaught AssertionError [ERR_ASSERTION]: { Error
at Function.StorageServiceClient._normalizeError (E:\repo\azurite\Azurite\externaltests\azure-storage-node\ == null
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:516:18
at finalCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:949:7)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\filters\retrypolicyfilter.js:189:13
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:801:17
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:1014:11
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:800:15
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:952:5)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)

Test Failing : Azure-Storage-Node - BlobContainer - createContainer - should work

Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js

        // creating again will result in a duplicate error
        blobService.createContainer(containerName, function (createError2, container2) {
          assert.equal(createError2.code, Constants.BlobErrorCodeStrings.CONTAINER_ALREADY_EXISTS);
          assert.equal(container2, null);

azure-storage-node tests
base.js:266
BlobContainer
createContainer
should work:
Uncaught AssertionError [ERR_ASSERTION]: undefined == 'ContainerAlreadyExists'
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:166:18
at finalCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:568:7)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\filters\retrypolicyfilter.js:189:13
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:801:17
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:1014:11
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:800:15
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:571:5)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)

Copied blob has incorrect content type

After copying a blob, the copy has a content type of "application/octet-stream" instead of whatever was on the source blob.

Example powershell code to reproduce below.
When it runs against a real Azure storage account, the copy's resultant content type is "image/png" as expected.

New-Item "emptyfile.zzz"

#Upload as blob
$ctx = New-AzureStorageContext -Local
$container = New-AzureStorageContainer "mime-bug-repro" -Context $ctx
$originalBlob = Set-AzureStorageBlobContent -Blob "emptyfile.zzz" -Container $container.Name -File "emptyfile.zzz" -BlobType Block -Properties @{"ContentType" = "image/png"} -Context $ctx

#Initiate copy
$copyJob = Start-AzureStorageBlobCopy -SrcBlob $originalBlob.Name -SrcContainer $container.Name -DestBlob "copy-of-emptyfile.zzz" -DestContainer $container.Name -Context $ctx -DestContext $ctx
#Wait for copy.
while(($copyJob | Get-AzureStorageBlobCopyState).Status -ne "Success")
{
    sleep 5
}

#Get copy
$copiedBlob = Get-AzureStorageBlob -Container $container.Name -Blob "copy-of-emptyfile.zzz" -Context $ctx 

#Check ContentType
"The content type of the copy is: $($copiedBlob.ContentType)"

#Clean up file and container
$container | Remove-AzureStorageContainer -Force
Remove-Item "emptyfile.zzz"

Could we format logs better?

Currently (working with 0.9.7), if I open Docker container logs in a text reader I see something like

�[0mGET /account/customer?restype=container&comp=list �[32m200 �[0m19.399 ms - 255�[0m

Escape chars are so annoying :(
Could we turn them off?

Also, would be really helpful to print timestamp in some way, e.g. HH:mm:ss.SSS

Thanks a lot!

Get Blob returns "501 Not Implemented"

Problem description

While Get Blob requests seem to work perfectly fine from the az CLI, 501 errors are systematically returned when using the Blob SDK for Go.

CLI results

$ az storage blob download \
    -f /tmp/test.zip -n test.zip --container-name tests \
    --connection-string 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;'
output 
Finished[#############################################################]  100.0000%
{
  "content": null,
  "deleted": false,
  "metadata": {},
  "name": "test.zip",
  "properties": {
    "appendBlobCommittedBlockCount": null,
    "blobTier": null,
    "blobTierChangeTime": null,
    "blobTierInferred": false,
    "blobType": "BlockBlob",
    "contentLength": 3108,
    "contentRange": "bytes 0-3107/3108",
    "contentSettings": {
      "cacheControl": null,
      "contentDisposition": null,
      "contentEncoding": null,
      "contentLanguage": null,
      "contentMd5": "undefined",
      "contentType": "application/zip"
    },
    "copy": {
      "completionTime": null,
      "id": null,
      "progress": null,
      "source": null,
      "status": null,
      "statusDescription": null
    },
    "deletedTime": null,
    "etag": "\"5y1Za9ku1PTKog6mbmce+DM9xLQ\"",
    "lastModified": "2018-06-22T18:26:16+00:00",
    "lease": {
      "duration": null,
      "state": null,
      "status": null
    },
    "pageBlobSequenceNumber": null,
    "remainingRetentionDays": null,
    "serverEncrypted": null
  },
  "snapshot": null
}

SDK results

package main

import (
	"context"
	"fmt"
	"net/url"

	"github.com/Azure/azure-storage-blob-go/2017-07-29/azblob"
)

const (
	accountName        = "devstoreaccount1"
	accountKey         = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
	blobEndpoint       = "127.0.0.1:10000"
	blobEndpointScheme = "http"
	blobContainer      = "tests"
	blobName           = "test.zip"
)

func main() {
	credential := azblob.NewSharedKeyCredential(accountName, accountKey)

	pipeline := azblob.NewPipeline(credential, azblob.PipelineOptions{})

	containerURL := azblob.NewContainerURL(url.URL{
		Scheme: blobEndpointScheme,
		Host:   blobEndpoint,
		Path:   "/" + blobContainer,
	}, pipeline)

	blobURL := containerURL.NewBlockBlobURL(blobName)

	ctx := context.Background()

	resp, err := blobURL.Download(ctx, 0, 0, azblob.BlobAccessConditions{}, false)
	if err != nil {
		fmt.Print(err)
		return
	}
	fmt.Print(resp.ContentType())
}
$ go run main.go
output 
-> blob/vendor/github.com/Azure/azure-storage-blob-go/2017-07-29/azblob.NewResponseError, /go/src/blob/vendor/github.com/Azure/azure-storage-blob-go/2017-07-29/azblob/zz_generated_response_error.go:29
===== RESPONSE ERROR (ServiceCode=) =====
Description=failed to unmarshal response body, Details: (none)
   GET http://127.0.0.1:10000/tests/test.zip?timeout=61
   Authorization: REDACTED
   User-Agent: [Azure-Storage/0.1 (go1.10.3; darwin)]
   X-Ms-Client-Request-Id: [e574d5cf-4592-443b-517c-13decb79d178]
   X-Ms-Date: [Fri, 22 Jun 2018 18:36:59 GMT]
   X-Ms-Version: [2017-07-29]
   --------------------------------------------------------------------------------
   RESPONSE Status: 501 Not Implemented
   Connection: [keep-alive]
   Content-Length: [229]
   Content-Type: [text/html; charset=utf-8]
   Date: [Fri, 22 Jun 2018 18:36:59 GMT]
   Etag: [W/"e5-ZoqHfPrsJlaK3wwTnhkcerEarRI"]
   X-Powered-By: [Express]


EOF

Extra information

Tested using the Docker image:

$ docker run --rm -e executable=blob -p 10000:10000 arafato/azurite
$ export AZURE_STORAGE_CONNECTION_STRING='DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;'
$ az storage container create -n tests
$ az storage blob upload -c tests -f test.zip -n test.zip

Connect with Storage Explorer

I already looked into the previous issue related to storage explorer. Is it possible to connect when running on Mac OSX? I tried to connect using the connection string provided in the Readme (below) but I get an unhelpful error from explorer.

DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;

screen shot 2018-03-26 at 12 30 26 am

Queues are lost when stopping docker contrainer

I'm running Azurite via Docker and what I've found is even though I'm storing the blob & table data next to my project the queue data isn't stored there as well. If I then stop & start the container any queues I had are lost.

This behavior doesn't match what the Storage Emulator does since that persists queue data when restarting. It'd be great if Azurite did this as well. I'd prefer the queues stored alongside my project with the other data, but even persisting them on the container would be better than losing them.

storage services authentication

I am playing with the emulator. Seems like it does not support authentication,
https://docs.microsoft.com/en-us/rest/api/storageservices/authentication-for-the-azure-storage-services
The emulator provides static account and key.

AccountName =:: devstoreaccount1
AccountKey ::= Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

My expectation that it should fail incoming API request with a wrong account and/or key, but it seems azurite does not care much about authentication. Perhaps I am missing some configuration option?

Store the state of queues for creating pre initialized Docker images

Hi,

Is it possible to persist the state of the queues, like the blob containers?
I don't need to persist messages, but persisting queues, policies and metadata could be helpful.

I would like to create a docker image for our test environment, where blob containers and queues are pre initialize.

Currently I'm using a custom start script to run azurit and create containers and queue, but this might not be the way to go.

Best
nobiehl

Use Typescript

Strategy:

  • Convert existing code to ES6 (require => import) - #63
  • Convert JS to TS - with minimal tsconfig and lint configuration

Each step should be merged separately, so that the git history does not look like a mess and it will be possible to track back cleanly.
This approach will also allow for "easier" PR reviews and put the codebase in a position where by we can start to nicer typescript features such as async await

Docker image on Windows

I use AppVeyor as my build server and they only support Windows images (this week they soft released a paid upgrade to support Windows and Linux, but I don't want to pay extra just to use Azurite for my tests).

Would it be possible to publish both a Linux and Windows based docker image?

For now my build scripts check to see if they're running on AppVeyor and if so start up the Windows storage emulator instead of setting up Azurite. I'd prefer to run the same for dev & ci though.

I tried using the node package instead of docker but when you run that it blocks and the build never completes. Docker seems to be the best solution for both local dev & ci.

**PANIC** while creating storage container policy

Hi,

I'm using azurite version: 2.6.5 in a docker container on Windows 10 and docker-ce in WSL bash.

When I try to create a storage container policy with:

export AZURE_STORAGE_CONNECTION_STRING=BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;

docker run --rm -d -p 10000:10000 arafato/azurite:2.6.5
az storage container create --name test
az storage container policy create --name read --permission rl --container-name test

I'm getting the following error:

PUT /devstoreaccount1/test?restype=container&timeout=30 201 9.546 ms - 0
GET /devstoreaccount1/test?comp=acl&restype=container 200 6.874 ms - 59
PANIC Something unexpected happened! Blob Storage Emulator may be in an inconsistent state!
TypeError: Cannot read property '0' of undefined
at parseStringAsync.then (/opt/azurite/lib/xml/Serializers.js:19:81)
at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/opt/azurite/node_modules/bluebird/js/release/promise.js:512:31)
at Promise._settlePromise (/opt/azurite/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseCtx (/opt/azurite/node_modules/bluebird/js/release/promise.js:606:10)
at Async._drainQueue (/opt/azurite/node_modules/bluebird/js/release/async.js:138:12)
at Async._drainQueues (/opt/azurite/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:763:18)
at tryOnImmediate (timers.js:734:5)
at processImmediate (timers.js:716:5)
PANIC Something unexpected happened! Blob Storage Emulator may be in an inconsistent state!
TypeError: Cannot read property '0' of undefined
at parseStringAsync.then (/opt/azurite/lib/xml/Serializers.js:19:81)
at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/opt/azurite/node_modules/bluebird/js/release/promise.js:512:31)
at Promise._settlePromise (/opt/azurite/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseCtx (/opt/azurite/node_modules/bluebird/js/release/promise.js:606:10)
at Async._drainQueue (/opt/azurite/node_modules/bluebird/js/release/async.js:138:12)
at Async._drainQueues (/opt/azurite/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:763:18)
at tryOnImmediate (timers.js:734:5)
at processImmediate (timers.js:716:5)
PANIC Something unexpected happened! Blob Storage Emulator may be in an inconsistent state!
TypeError: Cannot read property '0' of undefined
at parseStringAsync.then (/opt/azurite/lib/xml/Serializers.js:19:81)
at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/opt/azurite/node_modules/bluebird/js/release/promise.js:512:31)
at Promise._settlePromise (/opt/azurite/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseCtx (/opt/azurite/node_modules/bluebird/js/release/promise.js:606:10)
at Async._drainQueue (/opt/azurite/node_modules/bluebird/js/release/async.js:138:12)
at Async._drainQueues (/opt/azurite/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async.js:17:14)js:17:1
4)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async
.js:17
:14)
at runCallback (timers.js:763:18)
at tryOnImmediate (timers.js:734:5)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async
.UT /devstoreaccount1/test?comp=acl&restype=container - - ms - -
js:17:14)
at runCallback (timers.js:763:18) ) at tryOnImmediate (timers.js:734:5)
at processImmediate (timers.js:716:5)
PUT /devstoreaccount1/test?comp=acl&restype=container - - ms - - )
PANIC Something unexpected happened! Blob Storage Emulator may be in an inconsistent state! js:17:14TypeError: Cannot read property '0' of undefined .
at parseStringAsync.then (/opt/azurite/lib/xml/Serializers.js:19:81)
at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/opt/azurite/node_modules/bluebird/js/release/promise.js:512:31
) at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async
at Promise._settlePromise (/opt/azurite/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseFromHandler (/opt/azurite/node_modules/bluebird/js/release/promise.js:512:31) at Promise._settlePromise (/opt/azurite/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseCtx (/opt/azurite/node_modules/bluebird/js/release/promise.js:606:10)
at Async._drainQueue (/opt/azurite/node_modules/bluebird/js/release/async.js:138:12) at Async._drainQueues (/opt/azurite/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:763:18) at tryOnImmediate (timers.js:734:5)
at processImmediate (timers.js:716:5)
PANIC Something unexpected happened! Blob Storage Emulator may be in an inconsistent state!
TypeError: Cannot read property '0' of undefined
at parseStringAsync.then (/opt/azurite/lib/xml/Serializers.js:19:81)
at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/opt/azurite/node_modules/bluebird/js/release/promise.js:512:31)
at Promise._settlePromise (/opt/azurite/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseCtx (/opt/azurite/node_modules/bluebird/js/release/promise.js:606:10) at Async._drainQueue (/opt/azurite/node_modules/bluebird/js/release/async.js:138:12)
at Async._drainQueues (/opt/azurite/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:763:18)
at tryOnImmediate (timers.js:734:5)
at processImmediate (timers.js:716:5)PUT /devstoreaccount1/test?comp=acl&restype=container - - ms - -
PUT /devstoreaccount1/test?comp=acl&restype=container - - ms - -
PUT /devstoreaccount1/test?comp=acl&restype=container - - ms - -
PUT /devstoreaccount1/test?comp=acl&restype=container - - ms - -

Azure Table Query not working

I am doing a simple search on partitionkey and rowkey -> no data is being returned. If I remove all filters from the filter I get the data back.

I have tried this using WindowsAzure.Storage nuget package lib and also the Microsoft Azure Storage Explorer - both produce the same result

Standalone windows executables

In the Nuget package: "Azurite is packaged into a single self-contained executable"
(Side note: Opening the .nupkg as an archive reveals it's actually 3 executables: blob.exe, queue.exe, and table.exe)

Is there any reason that those aren't being published outside of the Nuget package? It seems more convenient to me than having to install it to a particular VS solution. (And easier than messing with Node.js. And less memory intensive than using Docker.)

Docker: console not functioning

When I attempt to connect to console, I get this error message:

OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"bash\": executable file not found in $PATH": unknown

Maybe that's deliberate?

Thanks!

Unable to set CORS rules programatically

In order to push to a storage queue from another origin, i need to add the appropriate CORS rule. At start up i try to set this programatically for the queue service....

  const properties = {
    Cors: {
      CorsRule: [{
        AllowedOrigins: ['*'],
        AllowedMethods: ['POST', 'GET', 'HEAD', 'PUT'],
        AllowedHeaders: ['*'],
        ExposedHeaders: ['*'],
        MaxAgeInSeconds: 3600
      }]
    }
  }

  qService.setServiceProperties(properties, function (error) {
    if (error) {
      throw error
    }
  })

This fails with the following error....

StorageError: Not Implemented yet.
    at Function.StorageServiceClient._normalizeError (~/dev/_spikes-poc/sas/server/node_modules/azure-storage/lib/common/services/storageserviceclient.js:1160:12)
    at QueueService.StorageServiceClient._processResponse (~/dev/_spikes-poc/sas/server/node_modules/azure-storage/lib/common/services/storageserviceclient.js:744:50)
    at Request.processResponseCallback [as _callback] (~/dev/_spikes-poc/sas/server/node_modules/azure-storage/lib/common/services/storageserviceclient.js:317:37)
    at Request.self.callback (~/dev/_spikes-poc/sas/server/node_modules/request/request.js:185:22)
    at emitTwo (events.js:126:13)
    at Request.emit (events.js:214:7)
    at Request.<anonymous> (~/dev/_spikes-poc/sas/server/node_modules/request/request.js:1157:10)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at IncomingMessage.<anonymous> (~/dev/_spikes-poc/sas/server/node_modules/request/request.js:1079:12)

I also tried hitting the associated rest endpoints, but get 501 not implemented...

GET /devstoreaccount1?restype=service&comp=properties HTTP/1.1
501: Not Implemented

PUT /devstoreaccount1?restype=service&comp=properties HTTP/1.1
501: Not Implemented

Is there another convention for setting CORS rules in Azurite?

Test Failing : Azure-Storage-Node - BlobContainer - setContainerAcl - should work with signed identifiers

Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:634:01

Assuming issue is generated here, either in how Azurite writes or return the data:

var options = {publicAccessLevel: BlobUtilities.BlobContainerPublicAccessType.OFF};
      blobService.setContainerAcl(containerName, signedIdentifiers, options, function (setAclError, setAclContainer, setAclResponse) {
        assert.equal(setAclError, null);
        assert.notEqual(setAclContainer, null);
        assert.ok(setAclResponse.isSuccessful);
        setTimeout(function () {
          blobService.getContainerAcl(containerName, function (getAclError, containerAcl, getAclResponse) {
            assert.equal(getAclError, null);
            assert.notEqual(containerAcl, null);
            assert.notEqual(getAclResponse, null);

            if (getAclResponse) {
              assert.equal(getAclResponse.isSuccessful, true);
            }

            assert.equal(containerAcl.signedIdentifiers.id1.Start.getTime(), new Date('2009-10-10T00:00:00.123Z').getTime());

azure-storage-node tests
base.js:266
BlobContainer
setContainerAcl
should work with signed identifiers:
Uncaught TypeError: Cannot read property 'split' of undefined
at Object.exports.parse (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\lib\common\util\iso8061date.js:48:22)
at E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\lib\common\models\aclresult.js:105:44
at Array.forEach ()
at Object.exports.parse (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\lib\common\models\aclresult.js:101:26)
at processResponseCallback (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:874:68)
at Request.processResponseCallback [as _callback] (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)

Doesn't work when trying to acquire a lease with Azure function core tool

Hi there,

I have an Azure durable function running locally through Azure core function tools. The main entry function is running fine, but it has problem of triggering a orchestration function through orchestration client. From the container log, I can see requests to PUT /devstoreaccount1/myfunc-leases/default/myfunc-control-00?comp=lease failed with 400 error. It seems to have problem with x-ms-proposed-lease-id.

Is there anyway I can see the full request in log?

I have no problem of running this against a real storage account.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.