Tus resumes unrelated uploads, causing conflicts or lost metadata

[defnull]

Tus resumes unrelated uploads, causing conflicts or lost metadata

[Acconut]

In an ideal world, duplicate uploads of the same file would be avoided, but in practice this is not always easily possible, so I think this should be addressed. The easiest (and sensible) solution seems to be adding the uppy ID into the file ID. tus-js-client puts the fingerprints into the global localStorage, where collisions can appear leading to the described behavior. If we can embed information about the corresponding uppy instance into the file ID and thus fingerprint, this should work.

[Murderlon]

Hi, did you try to simply pass your own fingerprint option? From the docs:

All options are passed to tus-js-client and we document the ones here that are required, added, or changed. This means you can also pass functions like onAfterResponse.
We recommended taking a look at the API reference from tus-js-client to know what is supported.

---

This breaks applications that need to support duplicate uploads, for example to allow the same file being uploaded to different folders or with different metadata.

To me it sounds a lot better to handle duplicate uploads on the server and prevent all your users from having to upload everything twice. That's double their bandwidth and more chances of it going wrong or out of sync. With tus this should be trivial with hooks and events, which our official servers have.

[defnull]

Hi, did you try to simply pass your own fingerprint option?

Yes. It is unconditionally overwritten by the Uppy Tus uploader plugin, so setting that option for the plugin has no effect. See

uppy/packages/@uppy/tus/src/index.ts

Line 249 in 68af8a3

uploadOptions.fingerprint = getFingerprint(file)

This breaks applications that need to support duplicate uploads, for example to allow the same file being uploaded to different folders or with different metadata.

To me it sounds a lot better to handle duplicate uploads on the server and prevent all your users from having to upload everything twice. That's double their bandwidth and more chances of it going wrong or out of sync. With tus this should be trivial with hooks and events, which our official servers have.

Currently the back-end controls uploads via tusd emitted events and trusts those events. Metadata is used to decide in which 'bucket' a file is uploaded, and this has to happen before the first byte is transferred. So, two uploads with different 'bucket' metadata absolutely have to trigger two Tus uploads. This is why I would like to add this information to File.id (or at least Tus fingerprint).

[defnull]

This is not just about this particular use-case by the way, it may affect other applications that depend on metadata in Tus uploads as well. Tus supports metadata natively and accepts and persists it along with the upload binary data. Metadata is part of the upload artifact. Files with different metadata should not be considered equal and silently skipped, or there should at least be a way to control which metadata is part of an upload identity (e.g. by overriding Tus fingerprints or the generation of File.id).

[defnull]

I experimented a bit and think that it would not be enough to be able to control the Tus fingerprint, because other components (most notably GoldenRetriever) also assume that File.id is a globally unique identifier. So, to avoid conflicts when using multiple Uppy instances on the same domain with different configuration, the getSafeFileId function should take the Uppy instance ID into account by default. To fully support duplicate uploads, users should be able to replace or extend the ID generation on a per-Instance basis.

I solved this for myself by replacing a single line:

diff --git a/packages/@uppy/core/src/Uppy.ts b/packages/@uppy/core/src/Uppy.ts
index f813b2242..502ba08fb 100644
--- a/packages/@uppy/core/src/Uppy.ts
+++ b/packages/@uppy/core/src/Uppy.ts
@@ -911,7 +911,7 @@ export class Uppy<M extends Meta, B extends Body> {
     const fileType = getFileType(file)
     const fileName = getFileName(fileType, file)
     const fileExtension = getFileNameAndExtension(fileName).extension
-    const id = getSafeFileId(file)
+    const id = getSafeFileId(file).replace("uppy-", `uppy-${this.getID()}`)
 
     const meta = file.meta || {}
     meta.name = fileName

This is obviously not a real solution, just a hotfix. A better way would be to have a configurable hook that either adds to, or completely replaces the ID generated by getSafeFileId.

[Murderlon]

@Acconut what do you think of this?