OOM: `mdedup` hangs then exits with message `Killed`

[danielhatton]

OOM: `mdedup` hangs then exits with message `Killed`

[danielhatton]

Thanks. Being a bit of a kludger, I might just write a wrapper script that takes my list of n maildir folders, and invokes mdedup the requisite n(n-1)/2 times to compare them all pairwise. Although n(n-1)/2 for my dataset is about 500, so it'll take a while to run.

[danielhatton]

Thinking a bit further about those numbers, it looks like mdedup is trying to hold the entire contents of the maildir tree in RAM at once, which is surprising given that what it's trying to do is compare hashes of certain headers.

[kdeldycke]

Yes, mail-deduplicate implementation is quite naive and choke on a non-trivial size of mails. The goal of this CLI was first to have it work before making it performant. We haven't reach that stage yet that's why implementing a cache has been proposed for several years, see: #87.

I do not have any time to work on mail-deduplicate right now. But feel free to propose PRs! :)

[kdeldycke]

When in doubt, brute force it. If it works, it's not a kludge. And machine time is cheaper than developer time. 😁

[kdeldycke]

Still, the commit history of that project indicate there's a non-null chance of me refreshing the code base once a year. So if your patient you might see a new release of mail-deduplucate in a couple of months.

[kdeldycke]

@shirosaki just proposed PR #562 to reduce the memory usage of mail-deduplicate. I just merged it upstream and try to cut a release today.

[kdeldycke]

Just released mail-deduplicate 7.3.0, with performance enhancements from @shirosaki .

I will close this issue for now on then.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.