Comments (6)
@kylepope thanks for the detailed investigation. Sorry for the late response, I'm quite busy these weeks.
We are working on a new architecture of Azurite, lease is an important part in the implementation. For current Azurite implementation, @edwin-huber any quick thoughs?
from azurite.
@XiaoningLiu Thanks for the response -- that's news to me but good to hear! Just curious -- any clues what a ballpark timeline on a preliminary implementation of the new architecture? The whole lease handing described in the docs is definitely complicated so I'm sure it'll take time to get that right.
A few more details on things I looked into with my use-case described above:
- I believe MD5 hash error is due to the fact that the GetBlob implementation will calculate the hash from the the raw blob from disk to the response body. I thought I might be able to remove new lease properties I added to the persisted blob prior to hash generation (and add them to response headers instead for example), but I think this would require a bigger change and I'm not sure what the ramifications might be.
- I was specifically looking in general in the code for how some pieces of blob data are converted from raw blob properties to response headers, like lease state (which should be a "x-ms-lease-state" header in a get blob response), but it looks like we're not doing that in a general way right now.
- I also tried to adding a function to
lib/core/env.js
to generate a new ID for lease information to perhaps store that separately (e.g. prefix with a "D" character, following what was done for block ID and snapshot ID) and then making the assignment to request.id specifically for lease operations. However, this doesn't end up working because some of that information needs to be saved as part of the blob I think (I ran into more different errors). I'm sure I'd need to make changes to other pieces of the implementation, but I'd need to figure out which.
Anyway, I'm pretty much out of free cycles at the moment. That said, I'd love to hear more about your future plans.
from azurite.
I've been encountering this problem while trying to get Azure Durable Functions to work on a mac using Azurite for the storage emulation. The specific problem here is due to the fact Azurite validation isn't quite right. If you look at lib/validation/blob/LeaseId.js
, there are two places where a isUUID
is used. This check isn't to spec since the definition of a Guid is a little more flexible than what the validation is checking. I'd recommend loosening the restriction, though I'm not sure what is most appropriate here (thoughts?).
While getting around this issue in my own local testing, I noticed a second problem with how leases work. The Durable Functions extension seems to go through the following flow in my testing:
- Get lease properties via a
HEAD /devstoreaccount1/durablefunctionshub-leases/default/durablefunctionshub-control-02
- Acquire a lease on the resource via a
PUT /devstoreaccount1/durablefunctionshub-leases/default/durablefunctionshub-control-02?comp=lease
- Perform a actual update to the resource via a
PUT /devstoreaccount1/durablefunctionshub-leases/default/durablefunctionshub-control-02
with a body like{ "PartitionId": "durablefunctionshub-control-02", "Owner": "<my-name>", "Token": "df7b79142c0b4b219d259bc67b19f641", "Epoch": 2 }
However, what happens here is that the acquired lease ID, expiration and other information written to the blob store in step 2 is overwritten with information provided in step 3. The reason for that is that step 3 routes to the _createOrUpdateBlob
function in lib/core/blob/StoreManager.js
, which will blow away previous contents and update them with new ones.
I toyed with the idea to propagate the lease properties to the newly created/saved object, but ran into separate issues because the MD5 hash of the blob's value no longer matches what was saved by call 3 (I think). I may not be understanding how this stuff works, but it seems like we should instead be saving lease information to a separate document/place in the store so it remains persisted.
Anyone have thoughts on this?
from azurite.
@kylepope Thanks! For the new architecture, we have an initial design specification. In the new design, we have several layers, the protocol layer will auto generated by API swagger definition, and persistence layer will be manually maintained. Lease information needs to be persisted.
There are still several months before we can publish the new architecture of Azurite, that means for critical issues or bugs, we should fix in current Azurite architecture. As you can see in the Azurite issues, there are plenty of items to be fixed in current architecture, we are kind of lacking resources to fix them in a short time. Contributions are very welcome.
It's very difficulty for an emulator implements 100% features or align with the real Azure Storage services. For example, the API versions supported. But we can define a matrix make the differences more clear.
For the lease issue, I don't understand about the md5 issue. The MD5 hash is the blob's content md5? As far as I know, the blob content will be persisted into disk, and the properties will be persisted into database object. The hash generation shouldn't be impacted by the blob properties in the database.
from azurite.
Hi Can you try latest released 3.3.0-preview? We did some refactor for lease implementation for blobs and containers. If the issue still happens for 3.3.0-preview, please share your environments, example code, and steps for the reproduce. @blueww is helping with Azure Function comptiability invesitgation with Azurite, and we'd like more information.
from azurite.
Azure Functions Core Tools now working with latest release.
Closing this issue.
from azurite.
Related Issues (20)
- Update mysql2 version HOT 5
- Metadata Vanishing HOT 12
- UserDelegationKey fails with 500 HOT 2
- Azurite HTTP Interface Impossible to Use from the Command Line. HOT 1
- BeginCopyAsync is throwing wrong exception when source blob is not found. HOT 1
- Upload from URL not working with a SAS URL HOT 1
- Not being able to launch Azurite from DockerHub. HOT 4
- The value for one of the HTTP headers is not in the correct format HOT 2
- Please Document Azurite Logging in VS CODE - not obvious how to see logs HOT 1
- Cannot add --skipApiVersionCheck when using testcontainers HOT 2
- Docker container is not up-to-date HOT 2
- Azurite returns 500 for all PUT requests after an unexpected client restart HOT 6
- Docker image won't stop restarting HOT 4
- Port specification not working (docker, --blobPort) HOT 1
- Block blob StageBlockWithURL isn't implemented HOT 6
- Azurite will deprecate supporting Node.js 14 in a future release.
- Azurite not recognizing accounts other than default even though prerequisites are met. HOT 2
- Azure Shared Access Signature - Signature did not match using ingress nginx HOT 3
- `Access-Control-Allow-Origin` header not set in Azurite response
- Aspire default table HealthCheck fails with 400 bad request
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azurite.