Still noticing intermittent freezing and "extended" launch time with 04-DEC Build

[DeathStalker77]

Still noticing intermittent freezing and "extended" launch time with 04-DEC Build

[mathiascode]

I think that there is also merit to ensuring that the architecture can effectively SCALE! Because you will always have users who queue large amounts, not only in total, but per user. And what you might consider a "large amount" might not be what I consider to be a large amount!

Considering how much of my free time I've spent in the past few months, with no compensation whatsoever, to make the transfer backend scale according to your needs specifically, it's not really motivating to keep reading comments like this.

The remaining freezes I know of are on the GUI side, which will be addressed when we port the treeviews from GtkTreeView to GtkColumnView.

Nothing personal, the last week just hasn't been great in general. I think I'll get the first release candidate out, and hopefully take a break for a while.

[stefantalpalaru]

~110k files in DL queue

I also noticed a drastic drop in download speed when trying to download thousands of files (tens of GB) from the same user, plus a high rate of connection timeouts while downloading.

Maybe we flood the other user with slskmessages.QueueUpload messages because we don't have a limit to our download queue and, when some of those messages don't get a response, we mark the connection as timed out and stop all downloads.

Ideally, we would be able to configure a limit to the download queue just like we do for the upload one, but a sensible hard-coded value would also work.

[slook]

Unknown if this has anything to do with share size, scanning, or DL queue (or any combination thereof).

It will not be possible to work on this bug issue until you are able to establish by further testing what exactly is the cause.

Perhaps you can make backups of downloads.json and uploads.json then clear a list to observe what difference it makes.

[mathiascode]

~110k files in DL queue

I also noticed a drastic drop in download speed when trying to download thousands of files (tens of GB) from the same user, plus a high rate of connection timeouts while downloading.

Maybe we flood the other user with slskmessages.QueueUpload messages because we don't have a limit to our download queue and, when some of those messages don't get a response, we mark the connection as timed out and stop all downloads.

Ideally, we would be able to configure a limit to the download queue just like we do for the upload one, but a sensible hard-coded value would also work.

Downloads currently work as follows:

1. Send QueueUpload message for every file we want to download once
2. If a download fails with "Connection timeout", retry it after 3 minutes. Otherwise download stays queued without further messages being sent.
3. If we receive a "Too many files" or "Too many megabytes" status for a download, remember how many (n) downloads are currently queued. When all successfully queued downloads have finished transferring, retry n more rejected downloads.

Are queued downloads timing out, or ones that are about to start transferring?

[stefantalpalaru]

Just the errors:

2023-12-05 16:55:20 [Conn] Direct connection of type P to user some_user failed. Error: Timed out
2023-12-05 16:56:13 [Conn] Cannot respond to indirect connection request from user some_user. Error: Timed out
2023-12-05 16:57:43 [Conn] Cannot respond to indirect connection request from user some_user. Error: Timed out
2023-12-05 16:58:19 [Conn] Direct connection of type P to user some_user failed. Error: Timed out
2023-12-05 16:59:13 [Conn] Cannot respond to indirect connection request from user some_user. Error: Timed out
2023-12-05 16:59:23 [Conn] Direct connection of type P to user some_user failed. Error: Timed out
2023-12-05 16:59:41 [Transfer] Download attempt for file @@foo/bar/baz.flac from user some_user failed with status Connection timeout

I'll email you a longer log.

[mathiascode]

I'll email you a longer log.

If I understand the logs correctly, the user sends a request to start uploading a file to you, it starts "Transferring", but eventually times out because no file data is sent for a long time. Can you confirm if this is what you're seeing in the GUI?

The "Timed out" errors in the logs usually indicate that the user's port is closed, but they should still be able to connect to your open port (i.e. a connection should be established regardless of these errors).

Does the transfer timeout issue happen with other users, or just this specific one?

[stefantalpalaru]

the user sends a request to start uploading a file to you

No. I'm downloading from that user. I think the protocol's message name is confusing.

eventually times out because no file data is sent for a long time

Or because some of this flurry of queueing messages timeout, so the whole socket is considered timed out, even if file transfers keep working. Also, it looks like all those messages are sent at the same time, in parallel, with no limits.

Does the transfer timeout issue happen with other users, or just this specific one?

I don't try to download 200+ GB (17K+ files) from other users. Anyway, I don't think it's user-specific because, if I keep my download queue under certain limits, the freeze does not appear.

On a positive note, those limits have increased on recent Nicotine+ commits. I can now queue 150GB and not worry about freezes.

[slook]

So, the performance is satisfactory enough in most normal type of usage scenarios?

[stefantalpalaru]

So, the performance is satisfactory enough in most normal type of usage scenarios?

Yes, but corner cases are useful for showing architectural flaws.

Queueing messages need to be serialised and throttled, timed out messages need to be retried a few times and the socket should not be marked as timed out if it's still carrying packets. A timed out socket also needs to be recreated, every minute or so.

[DeathStalker77]

I think that there is also merit to ensuring that the architecture can effectively SCALE! Because you will always have users who queue large amounts, not only in total, but per user. And what you might consider a "large amount" might not be what I consider to be a large amount!

I can tell you that if I'm browsing a user's files, I'm going to queue EVERYTHING that I want from that user at that time. I am NOT going to go to the "hassle" of doing it "piecemeal" or going back & forth. Just how large that user's queue turns out to be, is entirely dependent on what I am looking to get from them. NO ONE should CARE how much someone DLs (or queues) from them, as long as the DL'r has satisfied any minimum requirements a user might have asked (ex, Please share X number of files)

That would be the equivalent of you buying a car, but the manufacturer (in this case you all as the devs) have "decided" that they don't think you should be driving more than 100 miles per day, simply because it is not what they do.

[slook]

Thank you both, these are good observations.

Further testing and development is warranted to improve stability during bulk operations and faster startup under heavy load conditions.

Perhaps we might work more on this throughout the revisions of the 3.3.x branch, but for now as there doesn't seem to be any egregious bugs, I shall remove the Bug label of this issue in favour of Enhancement.

[mathiascode]

Closing in favor of #2910. It's the same issue (slow treeview additions/updates), but less severe due to fewer files.