azure / azurite Goto Github PK
View Code? Open in Web Editor NEWA lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
License: MIT License
A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
License: MIT License
Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:201:18
// creating again will result in a duplicate error
blobService.createContainer(containerName, function (createError2, container2) {
assert.equal(createError2.code, Constants.BlobErrorCodeStrings.CONTAINER_ALREADY_EXISTS);
assert.equal(container2, null);
blobService.removeAllListeners('sendingRequestEvent');
azure-storage-node tests
base.js:266
BlobContainer
createContainer
should work when the headers are set to string type:
Uncaught AssertionError [ERR_ASSERTION]: undefined == 'ContainerAlreadyExists'
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:201:18
at finalCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:568:7)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\filters\retrypolicyfilter.js:189:13
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:801:17
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:1014:11
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:800:15
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:571:5)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
See the request here: https://feedback.azure.com/forums/217298-storage/suggestions/10221816-add-support-for-https-endpoints-to-azure-storage-e
What would it take to support an HTTPS endpoint ? Could we do it withy dummy certs ?
I’ve been playing with Azurite some, and there appears there may be a problem with it’s tables implementation. Regardless of platform, I get errors when trying to add entities, delete entities, and delete tables. Some details:
POST /devstoreaccount1/test01 204 2.216 ms - -
GET /devstoreaccount1/test01?%24top=1000 200 1.560 ms - 575
GET /devstoreaccount1/test01?%24top=1000 200 0.747 ms - 575
POST /devstoreaccount1/$batch 500 1.124 ms - 45
**PANIC** Something unexpected happened! Table Storage Emulator may be in an inconsistent state!
TypeError: Cannot read property 'tableName' of undefined
at BbPromise.try (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\lib\middleware\table\validation.js:24:48)
at tryCatcher (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\util.js:16:23)
at Function.Promise.attempt.Promise.try (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\method.js:39:29)
at module.exports (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\lib\middleware\table\validation.js:18:18)
at Layer.handle [as handle_request] (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\router\layer.js:95:5)
at trim_prefix (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\router\index.js:317:13)
at C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\router\index.js:284:7
at Function.process_params (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\router\index.js:335:12)
at next (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\router\index.js:275:10)
at C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\body-parser\lib\read.js:130:5
at invokeCallback (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\raw-body\index.js:224:16)
at done (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\raw-body\index.js:213:7)
at IncomingMessage.onEnd (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\raw-body\index.js:273:7)
at emitNone (events.js:106:13)
at IncomingMessage.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1055:12)
GET /devstoreaccount1/test01?%24top=1000 200 0.869 ms - 575
Based on the tests for issue #12, it appears that Azurite might not actually be validating the requests against the associated access policy.
Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:709:26
// Test listing multiple blobs
listBlobsWithoutPrefix(null, null, function() {
assert.equal(blobs.length, 2);
var entries = 0;
blobs.forEach(function (blob) {
assert.notEqual(blob.creationTime, null);
assert.deepStrictEqual(typeof blob.creationTime, 'string');
if (blob.name === blobName1) {
entries += 1;
}
else if (blob.name === blobName2) {
entries += 2;
}
});
assert.equal(entries, 3);
azure-storage-node tests
base.js:266
BlobContainer
listBlobs
should work:
Uncaught AssertionError [ERR_ASSERTION]: undefined != null
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:709:26
at Array.forEach ()
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:708:23
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:682:13
at finalCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:5824:7)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\filters\retrypolicyfilter.js:189:13
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:801:17
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:1014:11
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:800:15
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:5827:5)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
blobPort=10100
Expected:
"Azure Blob Storage Emulator listening on port 10100"
Actual:
"Azure Blob Storage Emulator listening on port 10000"
PS C:\Users\cralvord> azurite -l C:\Users\cralvord\azurite\ -blobPort=10100 -queuePort=10101 -tablePort=10102
_______ _
(_______) (_) _
_______ _____ _ _ ____ _ _| |_ _____
| ___ (___ ) | | |/ ___) (_ _) ___ |
| | | |/ __/| |_| | | | | | |_| ____|
|_| |_(_____)____/|_| |_| \__)_____)
Azurite, Version 2.6.5
A lightweight server clone of Azure Storage
Azure Table Storage Emulator listening on port 10102
Azure Queue Storage Emulator listening on port 10101
Azure Blob Storage Emulator listening on port 10000
I'm using Azurite, Version 2.6.5 on Mac OSX
I perform the following request and I receive all results from the database:
GET /devstoreaccount1/Signposts HTTP/1.1
Host: 127.0.0.1:10002
Authorization: SharedKey devstoreaccount1:Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
x-ms-date: 1527335502
Content-Type: application/json
x-ms-version: 2017-07-29
Cache-Control: no-cache
Postman-Token: 463d93a9-df19-49d1-8e59-eaf509432d41
{
"value": [
{
"Timestamp": "2018-04-19T12:20:36.776Z",
"title": "Benefits - GOV.UK",
"description": "The best place to find government services and information. Simpler, clearer, faster",
"url": "https://www.gov.uk/browse/benefits",
"image": "/static/images/govuk.png",
"categories": "Benefits",
"PartitionKey": "Signposts",
"RowKey": "574d5f400d3e106660158ac3"
},
{
"Timestamp": "2018-04-19T12:20:36.791Z",
"title": "Business & Self-Employment - GOV.UK",
"description": "The best place to find government services and information. Simpler, clearer, faster",
"url": "https://www.gov.uk/browse/business",
"image": "/static/images/govuk.png",
"categories": "Business",
"PartitionKey": "Signposts",
"RowKey": "574d5f400d3e106660158ac4"
}
]
}
If I submit a request with a query (in this case on the PartitionKey) I get no results:
GET /devstoreaccount1/Signposts?$filter=PartitionKey%20eq%20%27Signposts%27 HTTP/1.1
Host: 127.0.0.1:10002
Authorization: SharedKey devstoreaccount1:Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
x-ms-date: 1527335659
Content-Type: application/json
x-ms-version: 2017-07-29
Cache-Control: no-cache
Postman-Token: ae88e955-2304-409d-820f-60e5b0a7c6d3
{
"value": []
}
This query was replayed via postman but I first tried the query via the Micosoft.WindowsAzure.Storage library which gave the same results.
Is the query wrong or is this a bug in Azurite?
Thanks
Simon
See the request here: https://feedback.azure.com/forums/217298-storage/suggestions/1500423-storage-emulator-with-storage-transaction-counting
Can we make it easy for developers to simulate their applications, and get an accurate billing estimate ?
Hi,
Currently I’m not able to list queue policies with azure cli and I’m also not able to list queue policies with the Storage Account explorer. But the policies will be created successfully.
I recognized that the storage account emulator is answering the get request different than azurite. There are some magic characters in front of the response and a zero at the end.
Storage Account Emulator 5.3.0.0
Azurite 2.6.5
Azure CLI 2.0.30
Windows 10 + WSL
az storage queue create --name test-queue
az storage queue policy list --queue-name test-queue
RESPONSE (Fiddler):
3E
<?xml version="1.0" encoding="utf-8"?><SignedIdentifiers />
0
az storage queue policy create --name read --queue-name test-queue --permissions rp --start 2018-04-01 --expiry 2018-04-02
az storage queue policy list --queue-name test-queue
RESPONSE (Fiddler):
F7
<?xml version="1.0" encoding="utf-8"?><SignedIdentifiers><SignedIdentifier><Id>read</Id><AccessPolicy><Start>2018-04-01T00:00:00.0000000Z</Start><Expiry>2018-04-02T00:00:00.0000000Z</Expiry><Permission>rp</Permission></AccessPolicy></SignedIdentifier></SignedIdentifiers>
0
az storage queue create --name test-queue
az storage queue policy list –queue-name test-queue
RESPONSE (Fiddler):
<?xml version="1.0" encoding="utf-8"?>
<SignedIdentifiers/>
az storage queue policy create --name read --queue-name test-queue --permissions rp --start 2018-04-01 --expiry 2018-04-02
az storage queue policy list --queue-name test-queue
azure cli returns nothing. But the response from azurite looks like:
RESPONSE (Fiddler):
<?xml version="1.0" encoding="utf-8"?>
<SignedIdentifiers>
<SignedIdentifiers>
<Id>read</Id>
<AccessPolicy>
<Start>2018-04-01</Start>
<Expiry>2018-04-02</Expiry>
<Permission>rp</Permission>
</AccessPolicy>
</SignedIdentifiers>
</SignedIdentifiers>
As you can see the magic characters in front of the response are not there and the trailing zero is also not available.
Another issue because of the issue above is that azure cli requests the queue policies first, if there are some policies available it merges the policies (old+new) and sends them back. But this is not possible with azurite. All policies will be overwritten by the new one.
Currently I circumventing the issue by using custom scripts for creating queues and policies but for debugging and some test environment related topics I need to use azure cli and the Storage Account Explorer.
Best
nobiehl
Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:586:01
Assuming that it is caused by either failing set or get container ACL code...
blobService.setContainerAcl(containerName, signedIdentifiers, options, function (setAclError, setAclContainer1, setResponse1) {
assert.equal(setAclError, null);
assert.notEqual(setAclContainer1, null);
assert.ok(setResponse1.isSuccessful);
setTimeout(function () {
blobService.getContainerAcl(containerName, function (getAclError, getAclContainer1, getResponse1) {
assert.equal(getAclError, null);
assert.notEqual(getAclContainer1, null);
assert.equal(getAclContainer1.publicAccessLevel, BlobUtilities.BlobContainerPublicAccessType.BLOB);
assert.equal(getAclContainer1.signedIdentifiers.readwrite.Expiry.getTime(), readWriteExpiryDate.getTime());
assert.ok(getResponse1.isSuccessful);
options.publicAccessLevel = BlobUtilities.BlobContainerPublicAccessType.CONTAINER;
blobService.setContainerAcl(containerName, null, options, function (setAclError2, setAclContainer2, setResponse2) {
assert.equal(setAclError2, null);
assert.notEqual(setAclContainer2, null);
assert.ok(setResponse2.isSuccessful);
azure-storage-node tests
base.js:266
BlobContainer
setContainerAcl
should work with policies:
Uncaught TypeError: Cannot read property 'split' of undefined
at Object.exports.parse (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\util\iso8061date.js:48:22)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\models\aclresult.js:105:44
at Array.forEach ()
at Object.exports.parse (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\models\aclresult.js:101:26)
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:874:68)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
Azurite Version: 2.6.5
OS Version: Windows 10
I get a continuous stream of error outputs when running Azurite from a PowerShell prompt (see below). However, if I run PowerShell as administrator and allow Node.js through the firewall, everything appears to work normally.
Sounds like a potential security risk.
PS> azurite
_______ _
(_______) (_) _
_______ _____ _ _ ____ _ _| |_ _____
| ___ (___ ) | | |/ ___) (_ _) ___ |
| | | |/ __/| |_| | | | | | |_| ____|
|_| |_(_____)____/|_| |_| \__)_____)
Azurite, Version 2.6.5
A lightweight server clone of Azure Storage
events.js:183
throw er; // Unhandled 'error' event
^
Error: listen EACCES 0.0.0.0:10002
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at Server.setupListenHandle [as _listen2] (net.js:1334:19)
at listenInCluster (net.js:1392:12)
at Server.listen (net.js:1476:7)
at Function.listen (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\application.js:618:24)
at env.init.then.then (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\lib\AzuriteTable.js:52:35)
at tryCatcher (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\util.js:16:23)
at Promise._settlePromiseFromHandler (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:512:31)
at Promise._settlePromise (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:569:18)
at Promise._settlePromise0 (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:614:10)
at Promise._settlePromises (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:693:18)
at Promise._fulfill (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:638:18)
at C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\nodeback.js:42:21
at Timeout._onTimeout (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\lokijs\src\lokijs.js:2633:15)
at ontimeout (timers.js:475:11)
events.js:183
throw er; // Unhandled 'error' event
^
Error: listen EACCES 0.0.0.0:10001
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at Server.setupListenHandle [as _listen2] (net.js:1334:19)
at listenInCluster (net.js:1392:12)
at Server.listen (net.js:1476:7)
at Function.listen (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\express\lib\application.js:618:24)
at env.init.then (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\lib\AzuriteQueue.js:48:35)
at tryCatcher (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\util.js:16:23)
at Promise._settlePromiseFromHandler (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:512:31)
at Promise._settlePromise (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:569:18)
at Promise._settlePromise0 (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:614:10)
at Promise._settlePromises (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:693:18)
at Promise._fulfill (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\promise.js:638:18)
at C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\bluebird\js\release\nodeback.js:42:21
at xfs.stat (C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\fs-extra\lib\mkdirs\mkdirs.js:56:16)
at C:\Users\cralvord\AppData\Roaming\npm\node_modules\azurite\node_modules\graceful-fs\polyfills.js:287:18
events.js:183
throw er; // Unhandled 'error' event
^
...
I noticed that if I run PowerShell as administrator and allow Node.js through the firewall, everything appears to work normally.
Following code uploads a 16MB binary and then copies it on itself:
CloudStorageAccount cloud = CloudStorageAccount.parse("DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;");
CloudBlobContainer container = cloud.createCloudBlobClient().getContainerReference("test");
container.deleteIfExists();
container.create();
for (int i = 0; i < 100; i++) {
CloudBlockBlob blob = container.getBlockBlobReference("test" + i);
System.out.print("Uploading: " + blob.getName() + "... ");
blob.uploadFromByteArray(new byte[16 * 1024 * 1024], 0, 16 * 1024 * 1024);
System.out.println("Done");
blob.startCopy(blob);
CopyStatus status;
while (true) {
blob.downloadAttributes();
status = blob.getCopyState().getStatus();
if (status != CopyStatus.PENDING) {
break;
}
Thread.sleep(200);
}
Assert.assertEquals(CopyStatus.SUCCESS, status);
}
Following exception is thrown randomly after a few iterations:
com.microsoft.azure.storage.StorageException: Incorrect Blob type, please use the correct Blob type to access a blob on the server. Expected BLOCK_BLOB, actual UNSPECIFIED.
at com.microsoft.azure.storage.blob.CloudBlob$8.preProcessResponse(CloudBlob.java:1238)
at com.microsoft.azure.storage.blob.CloudBlob$8.preProcessResponse(CloudBlob.java:1204)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:139)
at com.microsoft.azure.storage.blob.CloudBlob.downloadAttributes(CloudBlob.java:1197)
at com.microsoft.azure.storage.blob.CloudBlob.downloadAttributes(CloudBlob.java:1164)
at org.apache.jackrabbit.oak.segment.azure.LoopFailTest.test(LoopFailTest.java:28)
The latest release on npm is v2.6.5
, but on GitHub the last tag was v2.6.3
. Could you please tag the 2 missing versions on Github?
Azurite 2.6.5
OSX 10.13.3
Summary: Blob container which was once accessible from both python drivers (from a django app) as well as the azure storage explorer is now throwing 404s for images previously accessible. Also blob container itself doesn't even show up in the Storage Explorer.
When Azurite is running the requests to the container now 404.
GET /devstoreaccount1/test/images/product-picture-system-overview-1200x900_P2Xd.height-150.png 404 0.807 ms - 131
GET /devstoreaccount1/test/images/buddy-logo.original.png 404 0.752 ms - 131
GET /devstoreaccount1/test/images/buddy_bg.original.png 404 0.526 ms - 131
The Azure Storage explorer which once worked now does not show the container.
In the folder I use to store data I can see the various json configs. as well as the __blobstorage__
folder with my files in it. The __azurite_db_blob__.json
does indeed have my container config.
{
"filename": "azureStorage/__azurite_db_blob__.json",
"collections": [{
"name": "Containers",
"data": [{
"metaProps": {},
"entityType": "Container",
"leaseState": "available",
"access": "private",
"name": "test",
"etag": "hofpfmX+/GTL3Uk/7EKWIm16ETU",
"meta": {
"revision": 0,
"created": 1519943461794,
"version": 0
},
"$loki": 1
}],
"idIndex": [1],
"binaryIndices": {},
"constraints": null,
"uniqueNames": [],
"transforms": {},
"objType": "Containers",
"dirty": false,
"cachedIndex": null,
"cachedBinaryIndex": null,
"cachedData": null,
"adaptiveBinaryIndices": true,
"transactional": false,
"cloneObjects": false,
"cloneMethod": "parse-stringify",
"asyncListeners": false,
"disableMeta": false,
"disableChangesApi": true,
"disableDeltaChangesApi": true,
"autoupdate": false,
"serializableIndices": true,
"ttl": null,
"maxId": 1,
"DynamicViews": [],
"events": {
"insert": [null],
"update": [null],
"pre-insert": [],
"pre-update": [],
"close": [],
"flushbuffer": [],
"error": [],
"delete": [null],
"warning": [null]
},
"changes": []
}, {
"name": "ServiceProperties",
"data": [],
"idIndex": [],
"binaryIndices": {},
"constraints": null,
"uniqueNames": [],
"transforms": {},
"objType": "ServiceProperties",
"dirty": false,
"cachedIndex": null,
"cachedBinaryIndex": null,
"cachedData": null,
"adaptiveBinaryIndices": true,
"transactional": false,
"cloneObjects": false,
"cloneMethod": "parse-stringify",
"asyncListeners": false,
"disableMeta": false,
"disableChangesApi": true,
"disableDeltaChangesApi": true,
"autoupdate": false,
"serializableIndices": true,
"ttl": null,
"maxId": 0,
"DynamicViews": [],
"events": {
"insert": [null],
"update": [null],
"pre-insert": [],
"pre-update": [],
"close": [],
"flushbuffer": [],
"error": [],
"delete": [null],
"warning": [null]
},
"changes": []
}, {
"name": "test",
"data": [{
"metaProps": {},
"entityType": "BlockBlob",
"leaseState": "available",
"access": "private",
"name": "original_images/bundle-enterprise.png",
"id": "QXRlc3RvcmlnaW5hbF9pbWFnZXMvYnVuZGxlLWVudGVycHJpc2UucG5n",
"uri": "azureStorage/__blobstorage__/zZYc_Gf94S_K5DBYFoUEcw9qdT8=",
"snapshot": false,
"committed": true,
"size": 28311,
"etag": "cy4zKx42DoteaPpGlJb9eQEsDO8",
"contentType": "image/png",
"meta": {
"revision": 1,
"created": 1519943666535,
"version": 0,
"updated": 1519943666538
},
"$loki": 1
},
No idea how to troubleshoot this.
I'm using azure/azure-storage-node.
This is the code that fails (in azure/azure-storage-node)
There is the check
else if (responseObject.error && responseObject.error.statusCode === Constants.HttpConstants.HttpResponseCodes.NotFound) {
responseObject.error = null;
responseObject.tableResult.exists = false;
responseObject.response.isSuccessful = true;
}
But the responseObject
is as following:
{ error:
{ StorageError: <?xml version="1.0" encoding="utf-8"?><Error><Code>TableNotFound</Code><Message>The table specified does not exist.</Message></Error>
at Function.StorageServiceClient._normalizeError (/Users/alexander/Development/Sandbox/azurite-check/node-service/node_modules/azure-storage/lib/common/services/storageserviceclient.js:1153:12)
at TableService.StorageServiceClient._processResponse (/Users/alexander/Development/Sandbox/azurite-check/node-service/node_modules/azure-storage/lib/common/services/storageserviceclient.js:738:50)
at Request.processResponseCallback [as _callback] (/Users/alexander/Development/Sandbox/azurite-check/node-service/node_modules/azure-storage/lib/common/services/storageserviceclient.js:311:37)
at Request.self.callback (/Users/alexander/Development/Sandbox/azurite-check/node-service/node_modules/request/request.js:186:22)
at emitTwo (events.js:126:13)
at Request.emit (events.js:214:7)
at Request.<anonymous> (/Users/alexander/Development/Sandbox/azurite-check/node-service/node_modules/request/request.js:1163:10)
at emitOne (events.js:116:13)
at Request.emit (events.js:211:7)
at IncomingMessage.<anonymous> (/Users/alexander/Development/Sandbox/azurite-check/node-service/node_modules/request/request.js:1085:12)
name: 'StorageError',
message: '<?xml version="1.0" encoding="utf-8"?><Error><Code>TableNotFound</Code><Message>The table specified does not exist.</Message></Error>' },
response:
{ isSuccessful: false,
statusCode: 404,
body: '<?xml version="1.0" encoding="utf-8"?><Error><Code>TableNotFound</Code><Message>The table specified does not exist.</Message></Error>',
headers:
{ 'x-powered-by': 'Express',
'content-type': 'text/html; charset=utf-8',
'content-length': '133',
etag: 'W/"85-QChnKDYjVl8qDySHFUJOL2CNEFU"',
date: 'Tue, 10 Apr 2018 08:09:42 GMT',
connection: 'keep-alive' },
md5: undefined },
contentMD5: undefined,
length: undefined,
operationEndTime: 2018-04-10T08:09:39.194Z,
targetLocation: 0,
outputStreamSent: false,
tableResult: { isSuccessful: false, statusCode: 404, TableName: 'mytable' } }
So responseObject.error.statusCode
is undefined
.
Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:236:24
blobService.createContainerIfNotExists(containerName, function (createError3) {
assert.notEqual(createError3, null);
assert.equal(createError3.code, 'ContainerBeingDeleted');
done();
});
azure-storage-node tests
base.js:266
BlobContainer
createContainerIfNotExists
should create a container if not exists:
Uncaught AssertionError [ERR_ASSERTION]: null != null
+ expected - actual
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:236:24
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:642:9
at finalCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:568:7)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\filters\retrypolicyfilter.js:189:13
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:801:17
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:1014:11
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:800:15
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:571:5)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
Rather than having to recreate the wheel with Http requests, why not use
https://github.com/Azure/azure-storage-node ?
Hi,
I am trying to connect azurite using azure javascript library.
When I make a get request to any container, the browser tries to send preflight request which is failed with error code 501-not implemented. Even when I make a options request using fiddler, it fails with same error.
What configuration I need to do to make it work.
I was wondering that how others have been solved high availability on their Azurite deployments?
Old skool way mapping storage from NAS to two nodes most probably works but then storage system it selves is SPOF.
On Azure they are using data replication but which is good/tested option for it wilt Azurite?
There seems to be an inconsistency in how this emulator handles non-existent rows compared to how "real" Azure Table Storage handles them. I discovered this while using Streamstone, which builds event streaming capabilities on top of Azure Table Storage, and while I haven't been able to discern exactly what operations they're doing on the table storage, I have been able to build a minimal example that reproduces the problem.
To reproduce, make sure you have the dotnet
CLI installed, and run
mkdir repro && cd repro
dotnet new console
dotnet add package Streamstone
Then replace the contents of Program.cs
with this:
using System;
using System.Threading.Tasks;
using Microsoft.WindowsAzure.Storage;
using Streamstone;
namespace AzuritePanicOnStreamstoneTryOpenStream
{
class Program
{
public static async Task Main(string[] args)
{
// Replace with CloudStorageAccount.Parse("your connection string"); for using a real Azure Storage Account
var account = CloudStorageAccount.DevelopmentStorageAccount;
var table = account
.CreateCloudTableClient()
.GetTableReference("Events");
await table.CreateIfNotExistsAsync();
var partition = new Partition(table, "foo");
await Stream.TryOpenAsync(partition);
}
}
}
Finally, run dotnet run
to see the program fail with
Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: Unexpected response code, Expected:OK or NotFound, Received:InternalServerError
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Core\Executor\Executor.cs:line 315
at Streamstone.Stream.OpenStreamOperation.ExecuteAsync()
at AzuritePanicOnStreamstoneTryOpenStream.Program.Main(String[] args) in C:\Work\OSS\Throwaway repros\AzuritePanicOnStreamstoneTryOpenStream\Program.cs:line 17
at AzuritePanicOnStreamstoneTryOpenStream.Program.<Main>(String[] args)
If I look at the logs for my docker container running Azurite, I see
GET /devstoreaccount1/Events(PartitionKey='foo',RowKey='SS-HEAD') 500 0.388 ms - 27
**PANIC** Something unexpected happened! Table Storage Emulator may be in an inconsistent state!
ReferenceError: partitionKey is not defined
at TableStorageManager.queryEntities (/opt/azurite/lib/core/table/TableStorageManager.js:104:40)
at QueryEntities.process (/opt/azurite/lib/actions/table/QueryEntities.js:12:29)
at Object.actions.(anonymous function) [as QueryEntity] (/opt/azurite/lib/middleware/table/actions.js:52:19)
at BbPromise.try (/opt/azurite/lib/middleware/table/actions.js:18:38)
at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
at Function.Promise.attempt.Promise.try (/opt/azurite/node_modules/bluebird/js/release/method.js:39:29)
at module.exports (/opt/azurite/lib/middleware/table/actions.js:17:18)
at Layer.handle [as handle_request] (/opt/azurite/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/opt/azurite/node_modules/express/lib/router/index.js:317:13)
at /opt/azurite/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/opt/azurite/node_modules/express/lib/router/index.js:335:12)
at next (/opt/azurite/node_modules/express/lib/router/index.js:275:10)
at BbPromise.try (/opt/azurite/lib/middleware/table/validation.js:32:9)
at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
at Function.Promise.attempt.Promise.try (/opt/azurite/node_modules/bluebird/js/release/method.js:39:29)
at module.exports (/opt/azurite/lib/middleware/table/validation.js:18:18)
However, if I connect to a real Azure Storage Account instead of the emulator, the program runs without error.
Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:317:20
blobService.getContainerProperties(containerName, function (getError, container2, getResponse) {
assert.equal(getError, null);
assert.notEqual(container2, null);
if (container2) {
assert.equal('unlocked', container2.lease.status);
assert.equal('available', container2.lease.state);
assert.equal(null, container2.lease.duration);
assert.notEqual(null, container2.requestId);
assert.notEqual(container2.hasImmutabilityPolicy, null);
assert.deepStrictEqual(typeof container2.hasImmutabilityPolicy, 'boolean');
assert.notEqual(container2.hasLegalHold, null);
assert.deepStrictEqual(typeof container2.hasLegalHold, 'boolean');
azure-storage-node tests
BlobContainer
getContainerProperties
should work:
Uncaught AssertionError [ERR_ASSERTION]: undefined != null
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:317:20
at finalCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:698:7)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\filters\retrypolicyfilter.js:189:13
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:801:17
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:1014:11
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:800:15
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:701:5)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request.<anonymous> (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage.<anonymous> (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
Add validation rules according to https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-the-table-service-data-model
Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:516:18
var options = {publicAccessLevel: BlobUtilities.BlobContainerPublicAccessType.BLOB};
blobService.setContainerAcl(containerName, null, options, function (setAclError, setAclContainer1, setResponse1) {
assert.equal(setAclError, null);
assert.notEqual(setAclContainer1, null);
assert.ok(setResponse1.isSuccessful);
azure-storage-node tests
base.js:266
BlobContainer
setContainerAcl
should work:
Uncaught AssertionError [ERR_ASSERTION]: { Error
at Function.StorageServiceClient._normalizeError (E:\repo\azurite\Azurite\externaltests\azure-storage-node\ == null
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:516:18
at finalCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:949:7)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\filters\retrypolicyfilter.js:189:13
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:801:17
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:1014:11
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:800:15
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:952:5)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
queue.ApproximateMessageCount
always returns null
Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js
// creating again will result in a duplicate error
blobService.createContainer(containerName, function (createError2, container2) {
assert.equal(createError2.code, Constants.BlobErrorCodeStrings.CONTAINER_ALREADY_EXISTS);
assert.equal(container2, null);
azure-storage-node tests
base.js:266
BlobContainer
createContainer
should work:
Uncaught AssertionError [ERR_ASSERTION]: undefined == 'ContainerAlreadyExists'
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\test\services\blob\blobservice-container-tests.js:166:18
at finalCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:568:7)
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\filters\retrypolicyfilter.js:189:13
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:801:17
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:1014:11
at E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:800:15
at processResponseCallback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:571:5)
at Request.processResponseCallback [as _callback] (E:\repo\azurite\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
See https://docs.microsoft.com/en-us/rest/api/storageservices/constructing-an-account-sas for specification.
After copying a blob, the copy has a content type of "application/octet-stream" instead of whatever was on the source blob.
Example powershell code to reproduce below.
When it runs against a real Azure storage account, the copy's resultant content type is "image/png" as expected.
New-Item "emptyfile.zzz"
#Upload as blob
$ctx = New-AzureStorageContext -Local
$container = New-AzureStorageContainer "mime-bug-repro" -Context $ctx
$originalBlob = Set-AzureStorageBlobContent -Blob "emptyfile.zzz" -Container $container.Name -File "emptyfile.zzz" -BlobType Block -Properties @{"ContentType" = "image/png"} -Context $ctx
#Initiate copy
$copyJob = Start-AzureStorageBlobCopy -SrcBlob $originalBlob.Name -SrcContainer $container.Name -DestBlob "copy-of-emptyfile.zzz" -DestContainer $container.Name -Context $ctx -DestContext $ctx
#Wait for copy.
while(($copyJob | Get-AzureStorageBlobCopyState).Status -ne "Success")
{
sleep 5
}
#Get copy
$copiedBlob = Get-AzureStorageBlob -Container $container.Name -Blob "copy-of-emptyfile.zzz" -Context $ctx
#Check ContentType
"The content type of the copy is: $($copiedBlob.ContentType)"
#Clean up file and container
$container | Remove-AzureStorageContainer -Force
Remove-Item "emptyfile.zzz"
Currently (working with 0.9.7), if I open Docker container logs in a text reader I see something like
�[0mGET /account/customer?restype=container&comp=list �[32m200 �[0m19.399 ms - 255�[0m
Escape chars are so annoying :(
Could we turn them off?
Also, would be really helpful to print timestamp in some way, e.g. HH:mm:ss.SSS
Thanks a lot!
While Get Blob
requests seem to work perfectly fine from the az
CLI, 501
errors are systematically returned when using the Blob SDK for Go.
$ az storage blob download \
-f /tmp/test.zip -n test.zip --container-name tests \
--connection-string 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;'
Finished[#############################################################] 100.0000%
{
"content": null,
"deleted": false,
"metadata": {},
"name": "test.zip",
"properties": {
"appendBlobCommittedBlockCount": null,
"blobTier": null,
"blobTierChangeTime": null,
"blobTierInferred": false,
"blobType": "BlockBlob",
"contentLength": 3108,
"contentRange": "bytes 0-3107/3108",
"contentSettings": {
"cacheControl": null,
"contentDisposition": null,
"contentEncoding": null,
"contentLanguage": null,
"contentMd5": "undefined",
"contentType": "application/zip"
},
"copy": {
"completionTime": null,
"id": null,
"progress": null,
"source": null,
"status": null,
"statusDescription": null
},
"deletedTime": null,
"etag": "\"5y1Za9ku1PTKog6mbmce+DM9xLQ\"",
"lastModified": "2018-06-22T18:26:16+00:00",
"lease": {
"duration": null,
"state": null,
"status": null
},
"pageBlobSequenceNumber": null,
"remainingRetentionDays": null,
"serverEncrypted": null
},
"snapshot": null
}
package main
import (
"context"
"fmt"
"net/url"
"github.com/Azure/azure-storage-blob-go/2017-07-29/azblob"
)
const (
accountName = "devstoreaccount1"
accountKey = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
blobEndpoint = "127.0.0.1:10000"
blobEndpointScheme = "http"
blobContainer = "tests"
blobName = "test.zip"
)
func main() {
credential := azblob.NewSharedKeyCredential(accountName, accountKey)
pipeline := azblob.NewPipeline(credential, azblob.PipelineOptions{})
containerURL := azblob.NewContainerURL(url.URL{
Scheme: blobEndpointScheme,
Host: blobEndpoint,
Path: "/" + blobContainer,
}, pipeline)
blobURL := containerURL.NewBlockBlobURL(blobName)
ctx := context.Background()
resp, err := blobURL.Download(ctx, 0, 0, azblob.BlobAccessConditions{}, false)
if err != nil {
fmt.Print(err)
return
}
fmt.Print(resp.ContentType())
}
$ go run main.go
-> blob/vendor/github.com/Azure/azure-storage-blob-go/2017-07-29/azblob.NewResponseError, /go/src/blob/vendor/github.com/Azure/azure-storage-blob-go/2017-07-29/azblob/zz_generated_response_error.go:29
===== RESPONSE ERROR (ServiceCode=) =====
Description=failed to unmarshal response body, Details: (none)
GET http://127.0.0.1:10000/tests/test.zip?timeout=61
Authorization: REDACTED
User-Agent: [Azure-Storage/0.1 (go1.10.3; darwin)]
X-Ms-Client-Request-Id: [e574d5cf-4592-443b-517c-13decb79d178]
X-Ms-Date: [Fri, 22 Jun 2018 18:36:59 GMT]
X-Ms-Version: [2017-07-29]
--------------------------------------------------------------------------------
RESPONSE Status: 501 Not Implemented
Connection: [keep-alive]
Content-Length: [229]
Content-Type: [text/html; charset=utf-8]
Date: [Fri, 22 Jun 2018 18:36:59 GMT]
Etag: [W/"e5-ZoqHfPrsJlaK3wwTnhkcerEarRI"]
X-Powered-By: [Express]
EOF
Tested using the Docker image:
$ docker run --rm -e executable=blob -p 10000:10000 arafato/azurite
$ export AZURE_STORAGE_CONNECTION_STRING='DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;'
$ az storage container create -n tests
$ az storage blob upload -c tests -f test.zip -n test.zip
Based on the test case for issue #12 we should ensure that Azurite properly responds to cases where an invalid XML is submitted,
i.e. AcessPolicy tag is missing.
I already looked into the previous issue related to storage explorer. Is it possible to connect when running on Mac OSX? I tried to connect using the connection string provided in the Readme (below) but I get an unhelpful error from explorer.
DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;
I'm running Azurite via Docker and what I've found is even though I'm storing the blob & table data next to my project the queue data isn't stored there as well. If I then stop & start the container any queues I had are lost.
This behavior doesn't match what the Storage Emulator does since that persists queue data when restarting. It'd be great if Azurite did this as well. I'd prefer the queues stored alongside my project with the other data, but even persisting them on the container would be better than losing them.
I am playing with the emulator. Seems like it does not support authentication,
https://docs.microsoft.com/en-us/rest/api/storageservices/authentication-for-the-azure-storage-services
The emulator provides static account and key.
AccountName =:: devstoreaccount1
AccountKey ::= Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
My expectation that it should fail incoming API request with a wrong account and/or key, but it seems azurite does not care much about authentication. Perhaps I am missing some configuration option?
Hi,
Is it possible to persist the state of the queues, like the blob containers?
I don't need to persist messages, but persisting queues, policies and metadata could be helpful.
I would like to create a docker image for our test environment, where blob containers and queues are pre initialize.
Currently I'm using a custom start script to run azurit and create containers and queue, but this might not be the way to go.
Best
nobiehl
Strategy:
Each step should be merged separately, so that the git history does not look like a mess and it will be possible to track back cleanly.
This approach will also allow for "easier" PR reviews and put the codebase in a position where by we can start to nicer typescript features such as async await
I use AppVeyor as my build server and they only support Windows images (this week they soft released a paid upgrade to support Windows and Linux, but I don't want to pay extra just to use Azurite for my tests).
Would it be possible to publish both a Linux and Windows based docker image?
For now my build scripts check to see if they're running on AppVeyor and if so start up the Windows storage emulator instead of setting up Azurite. I'd prefer to run the same for dev & ci though.
I tried using the node package instead of docker but when you run that it blocks and the build never completes. Docker seems to be the best solution for both local dev & ci.
It would be cool if there was a way to seed azurite with data using docker volumes, (rather than having to use creation scripts)
Hi,
I'm using azurite version: 2.6.5 in a docker container on Windows 10 and docker-ce in WSL bash.
When I try to create a storage container policy with:
export AZURE_STORAGE_CONNECTION_STRING=BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;
docker run --rm -d -p 10000:10000 arafato/azurite:2.6.5
az storage container create --name test
az storage container policy create --name read --permission rl --container-name test
I'm getting the following error:
PUT /devstoreaccount1/test?restype=container&timeout=30 201 9.546 ms - 0
GET /devstoreaccount1/test?comp=acl&restype=container 200 6.874 ms - 59
PANIC Something unexpected happened! Blob Storage Emulator may be in an inconsistent state!
TypeError: Cannot read property '0' of undefined
at parseStringAsync.then (/opt/azurite/lib/xml/Serializers.js:19:81)
at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/opt/azurite/node_modules/bluebird/js/release/promise.js:512:31)
at Promise._settlePromise (/opt/azurite/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseCtx (/opt/azurite/node_modules/bluebird/js/release/promise.js:606:10)
at Async._drainQueue (/opt/azurite/node_modules/bluebird/js/release/async.js:138:12)
at Async._drainQueues (/opt/azurite/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:763:18)
at tryOnImmediate (timers.js:734:5)
at processImmediate (timers.js:716:5)
PANIC Something unexpected happened! Blob Storage Emulator may be in an inconsistent state!
TypeError: Cannot read property '0' of undefined
at parseStringAsync.then (/opt/azurite/lib/xml/Serializers.js:19:81)
at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/opt/azurite/node_modules/bluebird/js/release/promise.js:512:31)
at Promise._settlePromise (/opt/azurite/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseCtx (/opt/azurite/node_modules/bluebird/js/release/promise.js:606:10)
at Async._drainQueue (/opt/azurite/node_modules/bluebird/js/release/async.js:138:12)
at Async._drainQueues (/opt/azurite/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:763:18)
at tryOnImmediate (timers.js:734:5)
at processImmediate (timers.js:716:5)
PANIC Something unexpected happened! Blob Storage Emulator may be in an inconsistent state!
TypeError: Cannot read property '0' of undefined
at parseStringAsync.then (/opt/azurite/lib/xml/Serializers.js:19:81)
at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/opt/azurite/node_modules/bluebird/js/release/promise.js:512:31)
at Promise._settlePromise (/opt/azurite/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseCtx (/opt/azurite/node_modules/bluebird/js/release/promise.js:606:10)
at Async._drainQueue (/opt/azurite/node_modules/bluebird/js/release/async.js:138:12)
at Async._drainQueues (/opt/azurite/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async.js:17:14)js:17:1
4)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async
.js:17
:14)
at runCallback (timers.js:763:18)
at tryOnImmediate (timers.js:734:5)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async
.UT /devstoreaccount1/test?comp=acl&restype=container - - ms - -
js:17:14)
at runCallback (timers.js:763:18) ) at tryOnImmediate (timers.js:734:5)
at processImmediate (timers.js:716:5)
PUT /devstoreaccount1/test?comp=acl&restype=container - - ms - - )
PANIC Something unexpected happened! Blob Storage Emulator may be in an inconsistent state! js:17:14TypeError: Cannot read property '0' of undefined .
at parseStringAsync.then (/opt/azurite/lib/xml/Serializers.js:19:81)
at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/opt/azurite/node_modules/bluebird/js/release/promise.js:512:31
) at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async
at Promise._settlePromise (/opt/azurite/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseFromHandler (/opt/azurite/node_modules/bluebird/js/release/promise.js:512:31) at Promise._settlePromise (/opt/azurite/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseCtx (/opt/azurite/node_modules/bluebird/js/release/promise.js:606:10)
at Async._drainQueue (/opt/azurite/node_modules/bluebird/js/release/async.js:138:12) at Async._drainQueues (/opt/azurite/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:763:18) at tryOnImmediate (timers.js:734:5)
at processImmediate (timers.js:716:5)
PANIC Something unexpected happened! Blob Storage Emulator may be in an inconsistent state!
TypeError: Cannot read property '0' of undefined
at parseStringAsync.then (/opt/azurite/lib/xml/Serializers.js:19:81)
at tryCatcher (/opt/azurite/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/opt/azurite/node_modules/bluebird/js/release/promise.js:512:31)
at Promise._settlePromise (/opt/azurite/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseCtx (/opt/azurite/node_modules/bluebird/js/release/promise.js:606:10) at Async._drainQueue (/opt/azurite/node_modules/bluebird/js/release/async.js:138:12)
at Async._drainQueues (/opt/azurite/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues [as _onImmediate] (/opt/azurite/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:763:18)
at tryOnImmediate (timers.js:734:5)
at processImmediate (timers.js:716:5)PUT /devstoreaccount1/test?comp=acl&restype=container - - ms - -
PUT /devstoreaccount1/test?comp=acl&restype=container - - ms - -
PUT /devstoreaccount1/test?comp=acl&restype=container - - ms - -
PUT /devstoreaccount1/test?comp=acl&restype=container - - ms - -
I am doing a simple search on partitionkey and rowkey -> no data is being returned. If I remove all filters from the filter I get the data back.
I have tried this using WindowsAzure.Storage
nuget package lib and also the Microsoft Azure Storage Explorer - both produce the same result
In the Nuget package: "Azurite is packaged into a single self-contained executable"
(Side note: Opening the .nupkg as an archive reveals it's actually 3 executables: blob.exe, queue.exe, and table.exe)
Is there any reason that those aren't being published outside of the Nuget package? It seems more convenient to me than having to install it to a particular VS solution. (And easier than messing with Node.js. And less memory intensive than using Docker.)
When I attempt to connect to console, I get this error message:
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"bash\": executable file not found in $PATH": unknown
Maybe that's deliberate?
Thanks!
In order to push to a storage queue from another origin, i need to add the appropriate CORS rule. At start up i try to set this programatically for the queue service....
const properties = {
Cors: {
CorsRule: [{
AllowedOrigins: ['*'],
AllowedMethods: ['POST', 'GET', 'HEAD', 'PUT'],
AllowedHeaders: ['*'],
ExposedHeaders: ['*'],
MaxAgeInSeconds: 3600
}]
}
}
qService.setServiceProperties(properties, function (error) {
if (error) {
throw error
}
})
This fails with the following error....
StorageError: Not Implemented yet.
at Function.StorageServiceClient._normalizeError (~/dev/_spikes-poc/sas/server/node_modules/azure-storage/lib/common/services/storageserviceclient.js:1160:12)
at QueueService.StorageServiceClient._processResponse (~/dev/_spikes-poc/sas/server/node_modules/azure-storage/lib/common/services/storageserviceclient.js:744:50)
at Request.processResponseCallback [as _callback] (~/dev/_spikes-poc/sas/server/node_modules/azure-storage/lib/common/services/storageserviceclient.js:317:37)
at Request.self.callback (~/dev/_spikes-poc/sas/server/node_modules/request/request.js:185:22)
at emitTwo (events.js:126:13)
at Request.emit (events.js:214:7)
at Request.<anonymous> (~/dev/_spikes-poc/sas/server/node_modules/request/request.js:1157:10)
at emitOne (events.js:116:13)
at Request.emit (events.js:211:7)
at IncomingMessage.<anonymous> (~/dev/_spikes-poc/sas/server/node_modules/request/request.js:1079:12)
I also tried hitting the associated rest endpoints, but get 501 not implemented...
GET /devstoreaccount1?restype=service&comp=properties HTTP/1.1
501: Not Implemented
PUT /devstoreaccount1?restype=service&comp=properties HTTP/1.1
501: Not Implemented
Is there another convention for setting CORS rules in Azurite?
Failing test case:
Under : ./externaltests/azure-storage-node/test/services/blob/blobservice-container-tests.js:634:01
Assuming issue is generated here, either in how Azurite writes or return the data:
var options = {publicAccessLevel: BlobUtilities.BlobContainerPublicAccessType.OFF};
blobService.setContainerAcl(containerName, signedIdentifiers, options, function (setAclError, setAclContainer, setAclResponse) {
assert.equal(setAclError, null);
assert.notEqual(setAclContainer, null);
assert.ok(setAclResponse.isSuccessful);
setTimeout(function () {
blobService.getContainerAcl(containerName, function (getAclError, containerAcl, getAclResponse) {
assert.equal(getAclError, null);
assert.notEqual(containerAcl, null);
assert.notEqual(getAclResponse, null);
if (getAclResponse) {
assert.equal(getAclResponse.isSuccessful, true);
}
assert.equal(containerAcl.signedIdentifiers.id1.Start.getTime(), new Date('2009-10-10T00:00:00.123Z').getTime());
azure-storage-node tests
base.js:266
BlobContainer
setContainerAcl
should work with signed identifiers:
Uncaught TypeError: Cannot read property 'split' of undefined
at Object.exports.parse (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\lib\common\util\iso8061date.js:48:22)
at E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\lib\common\models\aclresult.js:105:44
at Array.forEach ()
at Object.exports.parse (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\lib\common\models\aclresult.js:101:26)
at processResponseCallback (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\lib\services\blob\blobservice.core.js:874:68)
at Request.processResponseCallback [as _callback] (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\lib\common\services\storageserviceclient.js:329:13)
at Request.self.callback (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:185:22)
at Request. (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1161:10)
at IncomingMessage. (E:\repo\azurite_pdressel\Azurite\externaltests\azure-storage-node\node_modules\request\request.js:1083:12)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
Hi there,
I have an Azure durable function running locally through Azure core function tools. The main entry function is running fine, but it has problem of triggering a orchestration function through orchestration client. From the container log, I can see requests to PUT /devstoreaccount1/myfunc-leases/default/myfunc-control-00?comp=lease failed with 400 error. It seems to have problem with x-ms-proposed-lease-id
.
Is there anyway I can see the full request in log?
I have no problem of running this against a real storage account.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.