Unbounded growth in transmit buffer with WebSockets

[EndlessDex]

Unbounded growth in transmit buffer with WebSockets

[scaprile]

I don't really think I have to clone your repo and spend my time browsing your code. Sorry about that.
Why don't you please trim this down to the simplest minimum piece of code reproducing your issue, post it here, and I'll gladly check it, and if it is our fault, fix it.
Please reopen this issue when you've done that

[EndlessDex]

    mg_timer_add(&mgr, WEBSOCKET_READ_INTERVAL, MG_TIMER_REPEAT, WebSocketSend, &mgr);

static void WebSocketSend(void *arg) {
    static unsigned char message[BUFSIZE] = {0};
    static char message_enc[BUFSIZE] = {0};

    int len = snprintf(reinterpret_cast<char *>(message), BUFSIZE, "%d: %s", packet_counter++, nonsense);

    // Encode the data
    len = mg_base64_encode(message, len, message_enc, BUFSIZE);

    for (auto it = websocket_map.begin(); it != websocket_map.end(); ++it) {
        mg_ws_send(it->first, message_enc, len, WEBSOCKET_OP_TEXT);
        MG_DEBUG(("WebSocketSend: %s", it->second.c_str()));
    }

    struct mg_mgr *mgr = (struct mg_mgr *)arg;
    struct mg_connection *c;
    for (c = mgr->conns; c != NULL; c = c->next) {
        if (c->is_websocket) {
            MG_DEBUG(("%p %lu : txBuf %p/%u/%u/%u    rxBuf %p/%u/%u/%u", c, c->id,
                      c->send.buf, c->send.len, c->send.size, c->send.align,
                      c->recv.buf, c->recv.len, c->recv.size, c->recv.align));
        }
    }
}

[EndlessDex]

As I state in my original comment, the repo is already in the minimal state. Please take the time to actually read the full content of the issue before closing it next time.

[scaprile]

Sorry, I fail to see how that gets to run. I can't compile that.
I've already been to your repo and had to navigate to find what is supposed to be some file that actually might have some reference to how you might perhaps be calling mongoose library functions. In that file I had to scroll several pages. We seem to differ in the definition of "minimal".
If you are a paying customer or have a support contract, please contact [email protected] and we'll gladly serve you, otherwise, please be kind to us and submit something small and simple that we can build and run to reproduce your issue.

[EndlessDex]

The issue has been sent to the support team. I AM a paying customer and look forward to the response.

First you ask for less code and then you ask for more. Both times without addressing the mentioned issue at all. I still do not believe you have actually read my post.

The source you need is in the linked repo at the mentioned filepath. It it a waste of MY time to continue discussing this with you when you have put in seemingly zero effort.

[scaprile]

I did read your post, I did visit your repo. I need to see how you call our functions, and your last post does not show that.
Do you call mg_timer_add() only once ?
How do you call the event manager Data is not sent when you call mg_ws_send(), it is sent when you call the manager.
Are you keeping everything in a single thread ?
Quite likely you already know that, but since you did not identify yourself, I need to ask.
Our support team will see your message tomorrow, I don't run Edge and I can't build a small app to try what you say. Unfortunately, we'll have to wait until tomorrow. I'll reopen and pin this issue for you.

[cpq]

@EndlessDex thanks for the report!
@scaprile thanks for looking into this!

Mongoose uses poll/select/epoll to figure out, can it drain the c->send output IO buffer to the peer, or not. Apparently when a browser stalls network IO, the TCP/IP stack on Mongoose side fills its socket buffer, and poll/select/epoll reports that this connection cannot drain, thus c->send keep the data and grows as you write more to it. The connection does not close, it just stalls.

It looks like that the question here is - how can we detect this situation? I believe it should be handled in the following way the WebSocketSend() function can respect the size of the output buffer. If it is larger than a certain threshold that you can tune, don't write to the connection:

  for (auto it = websocket_map.begin(); it != websocket_map.end(); ++it) {
    struct mg_connection *c = it->first;
    if (c->send.len > TOO_BIG) {
      MG_INFO(("Connection %lu is stalled, not sending anything to it"));
      continue;
    }
    ...

Would that be a solution to the problem?

Note: Mongoose uses exactly this technique internally, to shape the traffic when serving large files. When an HTTP server gets a request to a huge file, it does not mg_send() the whole file immediately, cause that could just kill low resource device. Instead, it fills the output buffer gradually, respecting the c->send.len. See this:

mongoose/src/http.c

Line 456 in 68e2cd9

if (c->send.len >= c->send.size) return; // Rate limit

This is a built-in HTTP event handler function for the static files. It preallocates the c->send IO buffer, and refuses to write more data if it was not drained yet. Hence the check against c->send.len. Your code could do the same.

A nitpick: Mongoose's printf functions support %m/%M non-standard modifiers that can base64-encode, see API Rerefence. You can create your own custom printer that e.g. prints your structures as binary or JSON. Your snippet may look like this:

static void WebSocketSend(void *arg) {
    char buf[BUFSIZE];
    size_t len = mg_snprintf(buf, sizeof(buf), "%d: %M", packet_counter++, mg_print_base64, 5, "hello");
    // Send buf to all WS clients

Or even without an intermediate buffer - base64 encode directly to the c->send IO buffer. E.g. send a JSON message with the connection ID:

static void WebSocketSend(void *arg) {
    for (auto it = websocket_map.begin(); it != websocket_map.end(); ++it) {
        mg_ws_printf(it->first, WEBSOCKET_OP_TEXT, "{%m: %lu}", MG_ESC("id"), it->first->id);
        MG_DEBUG(("WebSocketSend: %s", it->second.c_str()));
    }
   packet_counter++;

[cpq]

Closing this. Feel free to reopen if there is more data on the issue.