bufferevent_write does not immediately write data to the socket

[IronsDu]

bufferevent_write does not immediately write data to the socket

[azat]

Hello.

bufferevent_write as asynchronous write, it only adds to the evbuffer, and later when write is possible it will be written.

Also the time may depends on the some ratelimiting and stacking various bufferevents (though unlikely you have this).

What is the delay? Maybe EV_WRITE is not enabled for this bufferevent? (but then it will not be written at all). Can you provide a 100 line reproducer?

[IronsDu]

Hi, thanks for the reply. The issue has suddenly ceased to exist and I'm still not sure exactly what caused it. When the problem occurred before, the delay could reach 10ms+.

Also, I found that the behavior of the max_read buffer differs between version 2.1.12 and 2.2.1.alpha: in 2.1.12 I set the length beyond 16K via bufferevent_set_max_single_read, but it still occurs in the callbacks for processing the data that no message of the fulfilled size is read at once ( I realize that this can happen for real), however in 2.2.1.alpha this problem does not occur.

[IronsDu]

@azat

Hello.
Some code is:

static int sess_response(t_sess* sess)
{
    if (sess->bev != NULL && bufferevent_write(sess->bev, sess->msg_out_buf, sess->msg_out_len) < 0)
    {
        LOG_REPORT_ERROR("bufferevent_write failed");
    }
    return 0;
}

static void conn_readcb(struct bufferevent* bev, void* user_data)
{
    t_sess* sess = (t_sess*)user_data;
    while (sess_nonblock_parse_request(sess))
    {
        // process one request, ant then call sess_response send response to client.
        sess_handle_one_request(sess);
    }
}

static void
listener_cb(struct evconnlistener* listener, evutil_socket_t fd, struct sockaddr* sa, int socklen, void* user_data)
{
    t_uint32 network_event_base_index = os_interlock_inc_fetch(&next_network_event_base_index);

    struct event_base*  network_event_base = NULL;
    struct bufferevent* bev;
    t_sess*             sess = sess_create(NULL, FALSE);

    network_event_base_index = network_event_base_index % network_event_base_num;
    network_event_base       = network_event_base_array[network_event_base_index];

    bev = bufferevent_socket_new(network_event_base, fd, BEV_OPT_CLOSE_ON_FREE);
    if (!bev)
    {
        event_base_loopbreak(network_event_base);
        return;
    }

    sess->bev           = bev;

    bufferevent_set_max_single_read(bev, 16 * 1024);
    bufferevent_set_max_single_write(bev, 16 * 1024);

    bufferevent_setcb(bev, conn_readcb, NULL, conn_eventcb, sess);
    bufferevent_enable(bev, EV_WRITE | EV_READ);
}

listener = evconnlistener_new_bind(listen_event_base,
                                   listener_cb,
                                   (void*)listen_event_base,
                                   LEV_OPT_REUSEABLE | LEV_OPT_CLOSE_ON_FREE,
                                   -1,
                                   (struct sockaddr*)&sin,
                                   sizeof(sin));

Benchmark result:

use libevent 2.2.1.alpha. This above code cost 3m2s complete my job, bug If we direct use send function write data to session, it cost 2m44s .

[IronsDu]

Through my test and analysis I feel this is caused to calling epoll _ wait too many times.