Comments (15)
Could provide code that reproduces this error along with information on the length of the file you're attempting to upload?
Also, I see you responded with this trace on another issue but it looks like from this post this is a separate issue. Would you mind if I delete the other post?
from azure-storage-android.
the length of the file i am attempting to upload is 45M
from azure-storage-android.
I guess i use uploadBlock(final String blockId, final InputStream sourceStream, final long length) may resolve this issue,but ,i do not know how to use this method, can you give me a example?Thank you!
from azure-storage-android.
I'd love to try to figure out with you what's going wrong with uploadFile, but I need the code you're using as well as information on the device to be able to debug further. Could you provide this?
My initial guess is that your device may have relatively low memory or other apps are using a lot of memory. This might cause BufferedInputStream to not be able to allocate the size of the buffer we are requesting. You can reduce the buffer size by setting setStreamWriteSizeInBytes on the CloudBlockBlob. Note that reducing this will cause more, but smaller, calls to the storage service. Again, this is just a guess on my part but might be worth a try.
uploadBlock requires somewhat more knowledge of the storage service to use than the other upload methods. It is what we use underneath the covers in uploadFromFile. The basic idea with uploadBlock is that you upload your data in chunks (blocks) and then commit the list of blocks using commitBlockList. As uploadBlock and commitBlockList are simple wrappers around basic REST calls, you can see the REST documentation for PutBlock and PutBlockList for more information.
from azure-storage-android.
you can see https://github.com/leleliu008/Newton_for_Android/blob/master/src/com/leleliu008/newton/framework/upload/service/UploadBinder.java, thank you!
from azure-storage-android.
I can see you do AzureStorage.uploadFile but this is not actually our API - you must have wrapped our code. Could you please provide a minimum code example that reproduces the problem?
Did the suggestion I gave above (setStreamWriteSizeInBytes) work for you?
from azure-storage-android.
I ran into the similar issue with OutOfMemoryError and the solution with setting setStreamWriteSizeInBytes on the CloudBlockBlob worked for me.
In my code I'm using CloudBlockBlob.openOutputStream to retrieve the output stream and then copy data to it using simple write(byte[] buffer, int offset, int count) method.
Here is the stack trace of my exception before setting setStreamWriteSizeInBytes:
java.lang.OutOfMemoryError
at java.io.ByteArrayOutputStream.expand(ByteArrayOutputStream.java:91)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:201)
at com.microsoft.azure.storage.blob.BlobOutputStream.writeInternal(BlobOutputStream.java:698)
at com.microsoft.azure.storage.blob.BlobOutputStream.write(BlobOutputStream.java:624)
from azure-storage-android.
Glad to get confirmation. We'd love to fix this library side but I haven't found a good way yet. It's simply that some phones do not have the ability to allocate a large enough buffer period, or they normally would but have other processes running which consume memory, etc. I could reduce the default for setStreamWriteSize but then phones with more memory (or less memory usage) would lose some upload perf. Out of curiosity
- What did you end up setting setStreamWriteSizeInBytes to?
- What kind of device are you running on? (curious to see hardware specs)
from azure-storage-android.
- It's pretty strange, but reducing size down to 3 MB is enough (as I understand the default size is 4 MB and it's just for 1 MB larger)
- The device is Samsung Galaxy S2 with 1 GB of RAM. And the uploading process is the most memory consuming. The size of the only additionally allocated buffer during upload is just 128 KB.
I've checked memory consumption in DDMS and didn't found any noticeable memory leaks. I may be wrong, but I guess the issue in my case is caused by ByteArrayOutputStreams.toByteArray call which makes the copy of stream's internal buffer and it happens too often.
It might be not the best solution, but there is a way to get ByteArrayOutputStream's content without copying. The internal buffer is a protected field and it can be made accessible without reflection just by making a derieved class with getter, so "ByteArrayInputStream bufferRef" used in BlobOutputStream.dispatchWrite method can be created without calling toByteArray and making copy of outBuffer's content.
from azure-storage-android.
We did something similar and also have not seen leaks. ByteArrayOutputStream.toByteArray is one of the potential issues and I'd guess the issue in your case with the Galaxy S2. More profiling is needed on our end to find other potential culprits though.
Some smaller phones running more processes might also be having heap size issues. In addition to ByteArrayOutputStream issues, I'm also looking at android:largeHeap (which seems potentially abusive, so leaning on no) and http://developer.android.com/intl/ja/reference/android/app/ActivityManager.html#getMemoryClass() to help determine how much memory the phone has at the moment and optimize stream write size.
If you have additional ideas to explore, I'd love to know.
More advanced info, just if you're interested: We do actually need multiple byte arrays (so potentially copies). The reason for this is that we're spinning up an async thread to write to the service which has to access the buffer, but you should still be able to continue writing to the output stream without blocking. This requires fundamentally two buffers, even with parallelism at 1. If you increase parallelism, the number of buffers needed increases. Copying into a buffer from a buffer pool could help, but this is a bit less trivial than simply extending the class. These are the kind of optimizations we are also exploring.
from azure-storage-android.
Any news on this ?
I still have issues with 1.0.0 :
Caused by java.lang.OutOfMemoryError: Failed to allocate a 33554444 byte allocation with 16777216 free bytes and 29MB until OOM
java.io.BufferedInputStream.fillbuf (BufferedInputStream.java:163)
java.io.BufferedInputStream.read (BufferedInputStream.java:295)
com.microsoft.azure.storage.core.Utility.writeToOutputStream (Utility.java:1178)
com.microsoft.azure.storage.core.Utility.writeToOutputStream (Utility.java:1090)
com.microsoft.azure.storage.blob.BlobOutputStream.write (BlobOutputStream.java:645)
com.microsoft.azure.storage.blob.CloudBlockBlob.upload (CloudBlockBlob.java:683)
com.microsoft.azure.storage.blob.CloudBlob.uploadFromFile (CloudBlob.java:1665)
This happens very often to my users, any idea ?
from azure-storage-android.
Hi @victorleduc,
Have you tried the suggested approach of reducing the buffer size by setting setStreamWriteSizeInBytes on the CloudBlockBlob as a workaround? We will use this issue to track this item internally for a possible solution in the library as well.
cc:/ @jofriedm-msft
from azure-storage-android.
I tried this without success, I finally fixed it by forking your library and changing this 3 constants :
- In BlobConstants.java :
- MAX_SINGLE_UPLOAD_BLOB_SIZE_IN_BYTES from 64Mo to 4Mo
- DEFAULT_SINGLE_BLOB_PUT_THRESHOLD_IN_BYTES from 32Mo to 4Mo
- In Constants.java :
- MAX_MARK_LENGTH from 64Mo to 4Mo
The issue is that in the writeToOutputStream method, the sourceStream mark size is at his maximum (64Mo), so the read fails on low memory devices (I tested on Galaxy S4 mini).
It is set to the maximum here in the CloudBlockBlob.java file line 649 :
if (sourceStream.markSupported()) {
// Mark sourceStream for current position.
sourceStream.mark(Constants.MAX_MARK_LENGTH);
}
from azure-storage-android.
Hi @victorleduc,
Please try updating the SingleUploadThresholdInBytes value on the BlobRequestOptions per API or on your BlobClient.
We will update if there are any decisions to modify some of the constants.
Thanks,
Elham
from azure-storage-android.
Hi,
I don't remember precisely because I did this a month ago, but I remember trying to change the SingleUploadThresholdInBytes in the options, and it didn't fix it. I had to to what I said to you to make it work.
Victor
from azure-storage-android.
Related Issues (20)
- BlobListingDetails missing enum ALL?
- Problem uploading a file
- Unable to resolve hos
- Slow speed download
- CloudBlockBlob generated a wrong SAS token
- Random Non-Fatal Exceptions
- Outdated Sample
- Can there be a kotlin version of the sample compatible with latest azure storage version. HOT 1
- java.lang.ClassNotFoundException: Didn't find class "com.fasterxml.jackson.core.JsonFactory" on path: DexPathList
- SAS token blob container Shared Key authentication HOT 4
- Unexpected attempt to get register for a value without a register HOT 8
- Not found any example to upload a video to Blob.
- Update the Library and Simples
- CloudBlobContainer authentication error when repeating listBlobs() HOT 1
- Closeable resource has leak
- Unavailable ".withClientClaims()" method.
- Azure blob storage setup query
- Changing service endpoint for China
- This repo is missing important files
- why weak crypto ("MD5") using in this library
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-storage-android.