Comments (3)
I am experiencing the exact same problem with implementing uppy with the s3 multipart upload + golden retriever in an attempt to get it to resume the file upload where it left off after cutting the connection via a browser page refresh.
(I have been successful if i leave out the {serviceWorker: true} from the config, however that is not suitable for my requirements as it restarts the upload from the beginning if i lose the connection. )
Main difference between your implementation and mine is that i did not use companion or Tus because i really do not want to setup a server just to transfer the file to s3 in the end. I think it is more efficient and not overly complicated to achieve resumable uploads without the need of another server for the file to flow through before it goes to s3.
Thinking that this may not be resolved and may have to build a solution myself that mimics this library.
from uppy.
Thinking that this may not be resolved and may have to build a solution myself that mimics this library.
Probably better for the ecosystem and to save yourself some effort if you are willing to contribute rather than building it again. Would you be interested in taking a look?
from uppy.
Wanted to post an update about this issue here. My co-workers have been able to take a crack at the implementation of Uppy with the golden retriever plugin without the flag for the webWorker and have been successful.
The main issue they were able to solve was an error regarding an invalid signature (s3 signing) when attempting to resume the upload from where it was left off. After this was resolved, it seems that the solution works the way we need it (resumable uploads without the need of another server and without the need of Companion or Tus)
My suspicion is that the invalid signature error was caused by a malformed listParts command on the server which is used when attempting to resume the upload.
Note: The Error posted at the beginning of this thread I believe is still valid.
The good thing is that it seems seems that the webWorker may not be needed in order to implement a client -> s3 upload solution with support for resumable uploading.
For reference, I had used the example for the nodejs-aws server endpoints that are in this file: https://github.com/transloadit/uppy/blob/main/examples/aws-nodejs/index.js
The problem code is in this part:
app.get('/s3/multipart/:uploadId', (req, res, next) => {
const client = getS3Client()
const { uploadId } = req.params
const { key } = req.query
if (typeof key !== 'string') {
res.status(400).json({ error: 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"' })
return
}
const parts = []
function listPartsPage (startAt) {
client.send(new ListPartsCommand({
Bucket: process.env.COMPANION_AWS_BUCKET,
Key: key,
UploadId: uploadId,
PartNumberMarker: startAt,
}), (err, data) => {
if (err) {
next(err)
return
}
parts.push(...data.Parts)
if (data.IsTruncated) {
// Get the next page.
listPartsPage(data.NextPartNumberMarker)
} else {
res.json(parts)
}
})
}
listPartsPage(0)
})
The solution that seems to work well is changing how the listParts is called, this is what I have come up with that seems to work (need to change the endpoint handler to be asynchronous) :
let data = await s3Client.send(
new ListPartsCommand({
Bucket: AWS_BUCKET,
Key: key,
UploadId: uploadId,
MaxParts: 1000,
})
);
parts.push(...data.Parts);
// continue to get list of all uploaded parts until the IsTruncated flag is false
while (data.IsTruncated) {
data = await s3Client.send(
new ListPartsCommand({
Bucket: AWS_BUCKET,
Key: key,
UploadId: uploadId,
MaxParts: 1000,
PartNumberMarker: data.NextPartNumberMarker,
})
);
parts.push(...data.Parts);
}
res.json(parts);
I have submitted a PR to fix that issue in the aws-nodejs example to hopefully help others avoid the same error.
I hope this helps others who are trying to setup a similar solution using Uppy
*EDIT: I have realized that the solution I have explained here does not need to change the code so much, instead it is just a matter of conditionally passing in the PartNumberMarker into the config of the ListPartsCommand that is sent to s3.
I have also updated the PR to reflect this. here is what that endpoint handler looks like now with the update.
app.get('/s3/multipart/:uploadId', (req, res, next) => {
const client = getS3Client()
const { uploadId } = req.params
const { key } = req.query
if (typeof key !== 'string') {
res.status(400).json({ error: 's3: the object key must be passed as a query parameter. For example: "?key=abc.jpg"' })
return
}
const parts = [];
function listPartsPage(startsAt = undefined) {
let config = {
Bucket: process.env.COMPANION_AWS_BUCKET,
Key: key,
UploadId: uploadId,
};
if (startsAt) config.PartNumberMarker = startsAt;
client.send(new ListPartsCommand(config), (err, data) => {
if (err) {
next(err);
return;
}
parts.push(...data.Parts);
// continue to get list of all uploaded parts until the IsTruncated flag is false
if (data.IsTruncated) {
listPartsPage(data.NextPartNumberMarker);
} else {
res.json(parts);
}
});
}
listPartsPage();
})
from uppy.
Related Issues (20)
- `Complete` event is not fired if one or more files are removed before they are uploaded using XHRUpload
- ThumbnailGenerator: Failed to execute 'createObjectURL' on 'URL': Overload resolution failed.
- Paste from clipboard support on Drag & Drop module HOT 2
- New plugin: AI Image Generator HOT 3
- Composable, headless Uppy UI components HOT 1
- Implement Google Picker API
- Recent commit caused custom providers to break HOT 3
- Getting error in companion server while upgraded NuGet Packages HOT 4
- Tus uploader `onShouldRetry` has incorrect type signature
- upload-success not setting uploadURL correctly HOT 4
- Thumbnails from cloud companion uploaders (Google Drive, Dropbox, etc) sometimes do not display before upload HOT 1
- Dashboard briefly flashes `isDraggingOver` off while dragging over drop zone. HOT 2
- Error [ERR_MODULE_NOT_FOUND] in useUppyState.js in Vite Remix Application HOT 11
- Uploading additional batches with Transloadit is broken `allowMultipleUploadBatches: true` HOT 1
- Plugin link is broken in Readme HOT 2
- Add custom button to add bufferArray or file from external source HOT 1
- Uppy (4.1) does not work on a new Sveltekit (2.5.22) project HOT 1
- Update the Image Editor UI
- Removed the resume button for the Golden Retriever Plugin in Uppy
- Indicate that @uppy/react exports DashboardModal in the docs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from uppy.