{error, overload} observed in ranch_acceptor

[ninenines]

{error, overload} observed in ranch_acceptor

[essen]

Yeah, we should probably just drop these connections indeed.

[essen]

Hm where did you see an {error, overload}? Also I'm not sure I can do anything about it? This is the supervisor doing that. Could you provide me with more details?

[essen]

@jinnipark I need more details. Please see previous comment. Thanks!

[jinnipark]

Please find the patch below:

--- src/ranch_acceptor.erl
+++ src/ranch_acceptor.erl
@@ -52,14 +52,20 @@ loop(LSocket, Transport, Protocol, MaxConns, Opts, ListenerPid, ConnsSup) ->
                MaxConns, Opts, ListenerPid, ConnsSup);
        %% Found my sockets!
        {accept, CSocket} ->
-           {ok, ConnPid} = supervisor:start_child(ConnsSup,
-               [ListenerPid, CSocket, Transport, Protocol, Opts]),
-           Transport:controlling_process(CSocket, ConnPid),
-           ConnPid ! {shoot, ListenerPid},
-           NbConns = ranch_listener:add_connection(ListenerPid, ConnPid),
-           {ok, MaxConns2} = maybe_wait(ListenerPid, MaxConns, NbConns),
-           ?MODULE:init(LSocket, Transport, Protocol,
-               MaxConns2, Opts, ListenerPid, ConnsSup);
+           case supervisor:start_child(ConnsSup,
+                                       [ListenerPid, CSocket, Transport, Protocol, Opts]) of
+               {ok, ConnPid} ->
+                   Transport:controlling_process(CSocket, ConnPid),
+                   ConnPid ! {shoot, ListenerPid},
+                   NbConns = ranch_listener:add_connection(ListenerPid, ConnPid),
+                   {ok, MaxConns2} = maybe_wait(ListenerPid, MaxConns, NbConns),
+                   ?MODULE:init(LSocket, Transport, Protocol,
+                                MaxConns2, Opts, ListenerPid, ConnsSup);
+               Error ->
+                   fubar_log:error(?MODULE, [Error]),
+                   ?MODULE:loop(LSocket, Transport, Protocol,
+                                MaxConns, Opts, ListenerPid, ConnsSup)
+           end;
        %% Upgrade the max number of connections allowed concurrently.
        {set_max_conns, MaxConns2} ->
            ?MODULE:loop(LSocket, Transport, Protocol,

I took this from version 9fd9294.

[essen]

Yes, I get that, but the only errors supervisor:start_child/2 may return in the case of a simple_one_for_one supervisor is process start failure which is due to the protocol process failing to start. I'm not sure why it would? Can you explain?

[jinnipark]

What I observed is not well documented in the erlang reference. The child process itself is ok as it starts and runs ok usually. {error, overload} is an error from the supervisor itself, so it must be handled I guess. When I run my server without this patch and apply very severe load - I mean connect/disconnect/reconnect -, after some time there is no listener any more.

[essen]

I looked at the source and there is no mention of any {error, overload} anywhere in the Erlang source. What version are you using? Are you sure you are not using for example supervisor2.erl which may have added this "feature"?

[jinnipark]

I'm using R15B01 and without any suvervisor2. I don't even know what it is.

1. Just downloaded otp source and followed 'configure; make; make install'.
2. Started a tcp echo server in ranch with a little bit of logging code.
3. Ran a load generator from a different machine that just repeated connect/disconnect simultaneous 50k clients as best as it could.
4. After 20 minutes or so, the load generator couldn't connect any more.
5. Couldn't find any listener with netstat -lt.
6. Did rb to check error logs stored with log_mf_h and found {error, overload} with the source code reference which was ranch_acceptor line 55.
7. Still reproducible. Never survives longer than 1 hour.
8. No such failure after the patch.

[essen]

Alright. Let me ask on the mailing list where this error could come from. I need to understand where it's coming from. I would match on {error, overload} directly however, to avoid hiding user errors in the protocol start_link function.

[essen]

But before I do that, can I see the source for the protocol you are using?

[jinnipark]

I do only server programming and the load generator is not mine but a QA agent's that I don't have the code myself.
I'll prepare my own and try reproducing the error with it.

[essen]

I am talking about the server not the client. What's the module that Ranch uses for the protocol?

[jinnipark]

Oh, my...
You're right.
I've been returning {error, overload} in one of my protocol handlers actually.
Sorry.
But still I don't want the error return in handler to stop the listener though.

[essen]

This could be a feature added to Ranch, to avoid creating a protocol process if the service is overloaded. The above patch would be good except instead of Error we want to match on {error, overload} and it needs to be added to the guide and have a test.

[essen]

Hm the above patch isn't good actually, this doesn't close the socket. Needs to close the socket.

[jinnipark]

Thank you. I'll wait for your update.

[essen]

Would you like to do it?

[essen]

In the current code from last night, if you return anything other than {ok, Pid} then the supervisor ignores what you returned and no process is created. I believe that fixes this!

Closing, thanks.