Add option to delete or move any files or directories that cause errors

[eharris]

Add option to delete or move any files or directories that cause errors

[nielash]

Even better would be to make rclone sync or other operations where it is the obvious intent actually do this cleanup automatically. For reference, see #6437 which was closed as completed even though the problem remains.

I tend to agree with you that deleting would be the "correct" behavior when the undecryptable file is on the dest during a sync. I didn't go that far with the PR for a few reasons:

1. Concerns raised here about people intentionally mixing crypt with non-crypt, despite docs advising against this
2. Concerns raised here about this catching users by surprise, and deleting their files before they have a chance to investigate the problem
3. The crypt backend is not aware of when it is acting as a source vs. a dest, so it would actually be pretty tricky to make it delete files only when acting as a dest. It would be easy to make it delete them always, but then that verges on a different kind of incorrect, as sync is not supposed to delete anything from the source. Likewise, copy is not supposed to delete anything at all, and crypt is blind to whether the caller is sync vs. copy.

I like your idea of a --delete-errored flag that deletes garbage whenever it's encountered. It wouldn't be terribly difficult to implement that for crypt.

[eharris]

1. Concerns raised here about people intentionally mixing crypt with non-crypt, despite docs advising against this

In my opinion, this is a concern of no importance, at least as far as errored files on the destination. A sync operation is clearly intended to make the destination match the source, and removing any spurious destination files (including errored ones) is clearly within that intent. If the operation is a sync, and the file/dir doesn't exist on the source, then it should get removed on the destination, full stop. If someone wants to embed non-crypt data within a crypt sync destination, then it's reasonable to expect to have to to add --exclude rules to deal with that (although I admit I'm not exactly sure how that would/should work).

2. Concerns raised here about this catching users by surprise, and deleting their files before they have a chance to investigate the problem

I also don't think this is a valid concern, as the default for deletions is --delete-after, so no data will be lost since any potentially errored files that exist on the source would already have been replaced before deletions occur. If the user has chosen a different deletion behavior, then they got what they asked for. And if they want to be super safe, they can always use --max-delete 0, or they should do the normal (and often recommended) precaution of running with --dry-run before they actually run their command to make sure it's doing what they expect.

3. The crypt backend is not aware of when it is acting as a source vs. a dest, so it would actually be pretty tricky to make it delete files only when acting as a dest. It would be easy to make it delete them always, but then that verges on a different kind of incorrect, as sync is not supposed to delete anything from the source. Likewise, copy is not supposed to delete anything at all, and crypt is blind to whether the caller is sync vs. copy.

I totally agree that deleting errored files always (regardless of whether they are on the source or destination) is problematic, and I would not support such a solution unless it is protected by a separate and non-default flag to enable it.

However, the difficulty of implementation doesn't change the fact that for a sync operation performed on a destination with extraneous files (including errored ones), the only proper behavior is that they should be removed.

I like your idea of a --delete-errored flag that deletes garbage whenever it's encountered. It wouldn't be terribly difficult to implement that for crypt.

I did not intend for my suggestion of --delete-errored to apply to the source, since that would be unexpected and potentially quite dangerous. As an example, would a file that is unreadable be considered an error? While it isn't necessarily common to have files that are able to be deleted by a user that cannot read them, this is certainly possible (via acl's, SELinux policy, etc), and this could easily cause unexpected data loss if applied to the source.

I would never expect that the source of an operation would be modified under any circumstances unless the clear intent of the operation is to do so (e.g. bisync).

[nielash]

You make good points here, and it's hard to argue with them because in principle I agree with them.

I think ultimately it is @ncw's decision, and if @ncw thinks it's a good idea, I am happy to work on a PR that treats undecryptable dest files like any other missing-on-src file. I have some ideas about how it could be done -- it would be tricky, but not impossible.