upstream_check_module with NGINX 1.7.6 segfaults when reloading configuration

[yaoweibin]

upstream_check_module with NGINX 1.7.6 segfaults when reloading configuration

[anprevosto]

Upstream configuration is very simple :

upstream t7 {
    server 10.70.8.20:8200;
    server 10.70.8.20:8204;
    check interval=10000 rise=2 fall=2 timeout=5000 type=http;
}

[yaoweibin]

OK. I will check this problem.

[dmitry-saprykin]

Hello,

I have also had the same problem and fixed it with following patch:

#48

Kind regards,
Dmitry Saprykin

On 5 December 2014 at 08:39, Weibin Yao(姚伟斌) [email protected]
wrote:

OK. I will check this problem.

—
Reply to this email directly or view it on GitHub
#46 (comment)
.

[JonathanSerafini]

Same issue here, with nginx-1.7.4, nginx_upstream_check_module-0.3.0, and the check_1.7.2+.patch patch applied. Patching with diff generated from PR48 does not seem to have resolved the issue.

Nginx starts up correctly, however a few reloads will cause it to segfault.
Error log seems to print out : segfault at 11 ip 00007f976562ffd0 sp 00007fffc59b8d78 error 4 in libc-2.19.so[7f97654ee000+1bb000]

[JonathanSerafini]

Just tested nginx-1.7.8, nginx_upstream_check_module-master, check_1.7.5+ as well as PR48 combo and that seems to have resolved the issue on this end.

[yaoweibin]

I have merged PR48. It seems OK for me. I'm traveling out these days. I will test the issue you said. Than k you.

[JonathanSerafini]

Awesome, thanks a bunch !

[yaoweibin]

I will close this issue. Keep using the newest nginx version.

Thank you all.

[gstaples]

This is still an issue here. I'm getting that exact same backtrace as anprevosto with nginx-1.7.9 and this module updated today. I double and triple checked that I was actually using check-1.7.5+.patch.

I've pasted a full backtrace: http://pastebin.com/DrK52J2A

Can you take another look?

[gstaples]

Just a tad more info:
(gdb) print peer_shm->upstream_name->data
$6 = (u_char *) 0x1000 <Address 0x1000 out of bounds>

That's in ngx_http_upstream_check_find_shm_peer(), one frame up from the strncpy that segfaults.

[gstaples]

In that same area:

    if (ngx_memcmp(addr->sockaddr, peer_shm->sockaddr, addr->socklen) == 0
        && upstream_name->len == peer_shm->upstream_name->len
        && ngx_strncmp(upstream_name->data, peer_shm->upstream_name->data, upstream_name->len) == 0) {
        return peer_shm;
    }

(gdb) print upstream_name->len
$23 = 8
(gdb) print peer_shm->upstream_name->len
$24 = 8

So the (upstream_name->len == peer_shm->upstream_name->len) requirement is satisfied.

(gdb) print upstream_name->data
$25 = (u_char *) 0x153b421 "ucr-farm"
(gdb) print peer_shm->upstream_name->data
$26 = (u_char *) 0x1000 <Address 0x1000 out of bounds>

Woops.

[yaoweibin]

Could you show me your full config?

Thank you.

[gstaples]

I'll see what I can do. Our config is pretty big.
On Jan 30, 2015 8:02 AM, "Weibin Yao(姚伟斌)" [email protected]
wrote:

Could you show me your full config?

Thank you.

—
Reply to this email directly or view it on GitHub
#46 (comment)
.

[gstaples]

Basically, no, I can't send you the full config. We have over 370 files
included. 8000 lines. Keys. ACLs. My boss would kill me.

Our upstream configs look like a few dozen of these:

upstream mpp-farm {
least_conn;

server server1.example.com:50001; server server2.example.com:50001

check interval=5000 fall=3 rise=2 timeout=2000 default_down=false

type=http;
check_http_send 'GET /internal/health HTTP/1.0\r\nHost:
mpp-farm\r\nUser-Agent: nginx_upstream_check_module/1.2\r\n\r\n';
check_http_expect_alive http_2xx;

keepalive 6;  # per worker pool across all upstream servers

}

On Fri, Jan 30, 2015 at 10:09 AM, Garrick Staples <[email protected]

wrote:

I'll see what I can do. Our config is pretty big.
On Jan 30, 2015 8:02 AM, "Weibin Yao(姚伟斌)" [email protected]
wrote:

Could you show me your full config?

Thank you.

—
Reply to this email directly or view it on GitHub
#46 (comment)
.

[gstaples]

I just tried a build with nginx-1.7.5 and had the same segfault. After the second HUP, peer_shm is trashed.

[grahamhar]

I also see this issue, I think it is related to number of configured upstreams. I have one config that works OK and this only has 4 upstreams configured. I will try to do a little more to pin down if it is a specific number when this starts to fail.

[grahamhar]

I built with your latest master and nginx 1.7.9 and this seems to stop this problem happening for me.

[gstaples]

I suspect the number of configs didn't actually have anything to do with
it. It was just non-deterministic. There was a double free() or something
going on.

Our setup with its giant config seems nice and stable now.

On Mon, Feb 9, 2015 at 8:50 AM, grahamhar [email protected] wrote:

I built with your latest master and nginx 1.7.9 and this seems to stop
this problem happening for me.

—
Reply to this email directly or view it on GitHub
#46 (comment)
.

[MattDevUK]

Can I ask what the "fix" was? We're experiencing the same Segfault when reloading (Always on the second reload). Using Nginx v1.7.2 with the 1.7.2 patch and version 0.3.0 of the module. :S

Ah, just saw the pull Request #48 was merged after 0.3.0. Is there any info on when the next release will be? Not too keen on working with a master.

EDIT: Used Master branch and it fixed the segfaults. So now just want to wait for the next release if possible :D