Comments (11)
Ok I figured out the issue, resumable uploads is fully supported without companion - apologies for my false claims! Even more amazingly, it is supported exactly how I wanted it! (we locally store the successfully uploaded chunks within a file descriptor object and call listParts to compare!)
The issue was from my end, for my listParts endpoint I was returning a object containing the array! I went so far as to locally develop uppy haha - this was where I saw that yes, the companion code from above was being run but it was not the source of my error.
After two companion code blocks, it would go on to the listParts I mentioned earlier, it expects an array for the .find method to work and hence this was the falling point. Will update the code so it can be a good starting point for people in the future!
Thanks for your patience and apologies for any confusion! 🙏
from uppy.
This has been fixed: #4526
When using tus, you may not need Golden Retriever. The difference is after a browser refresh the user as to select the file again with tus and if the file fits in storage, it will be automatically loaded by Golden Retriever.
from uppy.
Checkout this option, it will likely fix your issue: https://uppy.io/docs/aws-s3-multipart/#shouldusemultipartfile
from uppy.
I'm not sure this is the place for Next.js specific issues. But if you have a reproducible example in codesandbox/stackblitz, then we may be able to help.
from uppy.
@uppy/golden-retriever
and uppy/aws-s3
(-multipart
) can work together yes.
By this I mean, will uppy golden retriever not store the file but store successfully uploaded chunks for me?
It stores the files, it's up to the uploader to decide what needs to be uploaded.
from uppy.
Apologies didn't see #4269 (comment)
Any plans to implement S3 support? Would be helpful if docs were clearer on which uploader golden retriever is actually intended for?
From what I just read, TUS innately supports resumable uploads, so if we were using TUS would golden retriever even be needed?
from uppy.
Thanks for the speedy response, appreciate the help!
My example wasn't great, my project is using S3-multipart and I tried integrating Golden Retriever last night. I will update the repo to test out with dashboard shortly!
I would like to achieve Resumable Uploads with S3 Multipart and looking at #2121 it looks like there has been some progress.
I was looking at the source code of multipart uploader to see why it may not be working in my project, and this line (201) of code stood out to me:
} else if (this.#isRestoring) { this.options.companionComm.restoreUploadFile(this.#file, { uploadId: this.options.uploadId, key: this.options.key }) this.#resumeUpload() }
Is companion required to support resumable S3 multipart uploads?
As users may be uploading large files, I do not want them to be stored with GR, what I would prefer is that successfully uploaded chunk metadata is stored instead, so if the same file is uploaded, we can pick up chunk metadata from where we left.
from uppy.
Ok I just updated the example code to include multipart and golden retriever, made into a hook like in my project.
The issue persists, something is going wrong, even when I try to pause and resume a upload - I think nextjs is bundling stuff funny.
![image](https://private-user-images.githubusercontent.com/135124328/311793474-3fdbe100-7c0e-42be-b5ee-6f5ba874ede2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE4MzU3MTAsIm5iZiI6MTcyMTgzNTQxMCwicGF0aCI6Ii8xMzUxMjQzMjgvMzExNzkzNDc0LTNmZGJlMTAwLTdjMGUtNDJiZS1iNWVlLTZmNWJhODc0ZWRlMi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyNFQxNTM2NTBaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT01NDdiOTEzODQ5YTIyZWJjZGFmN2U1ZDhkYmViMjY0YzkxYmMwMDllNzUyOTdiZTRmZWZjZTRmNTkwZWUwNjgxJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.so_o3IcvsYaOZVdi42m4tsPAKB1FgP0_vLAh-yplmSw)
![image](https://private-user-images.githubusercontent.com/135124328/311793598-a2639cd1-323d-4c38-addf-9c35a9d45ada.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE4MzU3MTAsIm5iZiI6MTcyMTgzNTQxMCwicGF0aCI6Ii8xMzUxMjQzMjgvMzExNzkzNTk4LWEyNjM5Y2QxLTMyM2QtNGMzOC1hZGRmLTljMzVhOWQ0NWFkYS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyNFQxNTM2NTBaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zZTNmMGNlZWYyYmQ3NmE2ZDY4NGFhYmFjZGI2MGU2MGJhYzE3YmE4MjNhODQxNTY5ZDM4ZmYwZWYxZjlmMTVlJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.9HlGRxw7jA9VAVlQq84zcn0rRjFoPuD2FpIbKJW6uJI)
You can test the example with 20mb files as I set the chunk size as 5mb.
There are two scenarios when we refresh the page 50% through upload:
-
The ghost file appears, if we try to continue, there is a crash - I don't think this is from uppy though, I think its next 14 "webpack" / turbopack
-
We try to a fresh upload with the same file, the upload starts from the beginning!
I have put the sw.js in my next public folder, so this should bypass webpack bundling. I have also done a dynamic import of Golden retriever so it should be available as required in the sw.js client?
from uppy.
Ofcourse, I just wanted to know if resumable multipart s3 uploads is supported, before I bother tackling the nextjs specific issues?!
By this I mean, will uppy golden retriever not store the file but store successfully uploaded chunks for me?
from uppy.
Ok I made some good progress!
I ran a react & express server with Golden Retriever, without the service worker. For me not to store the files on client, this is necessary.
This time the ghost file appeared and then I had a listParts problem, so when using golden retriever to reupload a previously uploaded file, listParts is called, this is good!
When I fixed this, I got back to this
This error comes from here, and because my companion is undefined, everything following is undefined. So a companion is required to support resumable uploads!
function _resumeUpload2() {
this.options.companionComm.resumeUploadFile(_classPrivateFieldLooseBase(this, _file)[_file], _classPrivateFieldLooseBase(this, _chunks)[_chunks], _classPrivateFieldLooseBase(this, _abortController)[_abortController].signal).then(_classPrivateFieldLooseBase(this, _onSuccess)[_onSuccess], _classPrivateFieldLooseBase(this, _onReject)[_onReject]);
}
Will try with a companion next haha! thanks, so I don't think its a next issue! but a companion problem! Will learn more about this!
from uppy.
I would just like to add that for me to resume a multipart s3 upload, I don't believe a fully fledged companion server should be a necessity!
Surely it should be possible to continue an upload from storing client side which chunks were successfully sent? We could call listParts to verify it is a valid upload!
Many thanks to the Uppy team regardless! I appreciate what you have built!!
from uppy.
Related Issues (20)
- SyntaxError in Safari on macOS when loading uppy.min.js (v3.22.2 and later) HOT 3
- Please add documentation on how to use uppy with an existing s3 client HOT 1
- Tus endpoint not being respected after initial request behind reverse proxy HOT 9
- i18n "Shared with me" etc HOT 1
- Allow streaming upload also for unknown length streams HOT 1
- @uppy/companion: include origin in dynamic customer credentials
- Using Vue2 to integrate uppy caused a strange mistake HOT 2
- aws-nodejs-example | Multipart complete api | Throws 400 bad requests. HOT 5
- title.value is not accessible inside uppy.upload in vue HOT 1
- I get this error "ResizeObserver loop completed with undelivered notifications" when I set width and height 100% in Angular HOT 5
- Uncaught TypeError: Cannot read properties of null (reading 'postMessage') in Companion's Callback Endpoint HOT 10
- Fix TypeScript sins HOT 5
- v4: typescript error for `uppy.setFileState()`
- Typescript error for 'new Uppy().use(AwsS3, {})' HOT 2
- XHR plugin expects upload response to be a valid JSON HOT 8
- companionAllowedHosts is not implemented in RemoteSources preset
- Uppy companion server error HOT 4
- Uppy 4.0.5 error HOT 5
- getTemporarySecurityCredentials() docs for S3 missing
- `Complete` event is not fired if one or more files are removed before they are uploaded using XHRUpload
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from uppy.