Backups much slower (5 mins compared to 0.3 secs) than reported "Duration" -- any way to speed it up?

[QUser534]

Backups much slower (5 mins compared to 0.3 secs) than reported "Duration" -- any way to speed it up?

[ThomasWaldmann]

After the files processing is when it does the stats stuff. Does it get faster without --stats?

As the stats in borg 1.x include overall repo / all archives stats, this might take longer if there are a lot of archives and/or if the repo is big.

[QUser534]

Thanks @ThomasWaldmann --stats was the culprit. Backup time went from 300 seconds to 9 seconds!

[infectormp]

Can you reproduce this behavior using a local repository, i.e., not using the network?

[ThomasWaldmann]

Possible explanations (IIRC):

• the loading and saving time of the repo and chunks index is generally not included in the backup time (but just the time needed to process files from the backup set).
• the time needed for a chunks cache resync is also not included (this happens e.g. if 2 clients back up to the same repo)

If you add the --list option, it might get clearer: the time while it lists files is the time that counts for the backup time.

[ThomasWaldmann]

Also, --stats might make it slower.

[QUser534]

Thanks for the quick response. I have added the --list option and I have also reviewed the FAQ. Neither has indicated what this 5 minute overhead could be especially considering the amount of data being backed up in this run is < 1 kilobyte and the total universe of files is 106 and only 39MB.

There is only one machine making the backup. Considering the entire dataset can be transferred several times over SSH within the 5 minutes why does is it taking 5 minutes to transfer less than the one kilobyte incremental change?

Results of --list command:

196 files / directories listed relatively quickly. Almost all had the flags d/U. 3 had the flag "s". 1 had the flag "A". 5 had the flag "x"

The delay seems to be after the files list.

The following environment variables are set which guarantee that there is no file cache issue in addition these tests are run directly after each other so no TTL issue:

BORG_FILES_CACHE_SUFFIX="segment-a"
BORG_FILES_CACHE_TTL="3"

I assume that borg must be downloading something from the repository and this occurs after the file list is generated because that happens relatively quickly. What could borg be doing to the repository that takes 5 minutes when the dataset is only 106 files and 39MB?

I have verified that within the borg directory exists a chunk directory, so that should have a local cache. Are there any other ways to see where borg is spending it's time?

EDIT: total non-incremental dataset size is actually < 39MB because that includes excluded data. Probably more like 20MB.