corrupt server response with brotli_comp_level 1 and lower

[eustas]

corrupt server response with brotli_comp_level 1 and lower

[eustas]

I've found that everything works fine, if nginx runs in daemon off mode. In daemon on it pretty quickly enters infinite loop.
The worst thing is that it could be fixed by adding ngx_log_error(..., ALERT ...) right after the start of the start of the main loop (line 334).

So it looks like a nasty race condition inside nginx itself :-(

Unfortunately I have low chances installing gdb on my Mac this week (almost no Internet there). Going to publish this commit that pollutes the error log, so you could check if it helps in your environment/setup as well.

[eustas]

Finally, I've got time to finish a patch.
Still have to add integration tests... But you can already try it: https://github.com/eustas/ngx_brotli/tree/rewrite

It would be nice, if you check if it fixes your problem (and does not cause new issues).

[eustas]

Hi. Thanks for reporting back. Fixed the problem with "notfound" - the problem is that next filter in the chain does not use input on the first call and returns NGX_OK. Logically, this status was forwarded to previous filter... And previous filter, never called brotli filter again =(
Looks like filter stack is a can of worms, overall...

Haven't tested 403 status and large lorem... Internet is very thin there...

Would appreciate very much if you rerun your tests with the latest commit to rewrite. Thank you.

[eustas]

Pushed aforementioned workaround. Going to debug further as soon as I can get gdb.

[eustas]

No response from nginx community =(

Tuned workflow,... seems to work correctly now =)

As always, would appreciate very much if you rerun your tests with the latest commit to rewrite. Thank you!

[eustas]

Good. Then I'm going t merge this branch to master soon. Thank you!

[eustas]

Pinpointed the logical problem. Going to fix tomorrow morning.
(Or this night, if got enough free time)

[eustas]

It seems that it will take more time than I've expected. Going to comb the whole module code, so it won't hurt to read...

[eustas]

Still working on the issue, fix is 80% ready, but we all know how deceptive the last 20% are.
In the worst case it will be ready in the middle of the next week.

[eustas]

Btw: in rewrite version of filter brotli_buffers are ignored. Zero output copy concept is used instead, i.e. internal encoder copy of compressed data is forwarded.

[eustas]

Latest findings: at some moment socket denies to send more data (see ngx_writev) and returns NGX_AGAIN to ngx_darwin_sendfile_chain. As a reaction it sets wev->ready = 0.
After this all successive calls to ngx_darwin_sendfile_chain just return, because !wev->ready.

Can we find any nginx experts who could explain how wev->ready is could / should be rearmed?

[khavishbhundoo]

@eustas Try to post on https://forum.nginx.org/list.php?29 for more attention

Really appreciate your work:)

[xabbu]

@eustas again thank you for your work on this.

I tried the new commit 0052669 and it seems everything works now. All brotli compressed responds are valid and nginx didn't hang at any time. I also tried not only my constructed cases, I didn't have any problems on real websites. =)

[eustas]

Completed adding basic integration tests. Will merge in few hours

[eustas]

Thank you for the report. If it is easy to reproduce - it will be easy to fix =)
Going to dig into it soon.

[eustas]

Was able to reproduce something like this. It, indeed reports 200, most of the content is delivered, except for a tiny tail. Going to debug.

[eustas]

Looks like the "tail" is the block pushed to encoder with "finish" operation.
Going to investigate further.

[xabbu]

Thanks for your work on this, I really appreciate it.

[xabbu]

I tested your rewrite branch and the result is a not entirely successful.

I used the same configure options for the build and nearly the same nginx.conf for this test. I did however disabled http2 and added a option to get a 403 status code.

With a brotli_comp_level higher than 1 everything works as expected. With brotli_comp_level 1 and 0 not everything works.

First I tried the standard nginx index.html and the Lorem5000.txt. I received this files without a problem. The bortli archive was not corrupt. Then I tried notfound to get an 404 responds.

$ curl -H 'Accept-encoding: br' http://debian-test.local/notfound | brotli -cd
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:01:00 --:--:--     0
curl: (52) Empty reply from server
corrupt input [con]

This time the server did not send any data. So the "corrupt input" can be ignored. The access.log file shows this entry

[07/Mar/2018:23:44:46 +0100] "GET /notfound HTTP/1.1" 404 0 gzip - br -

But nginx should send the standard 404 page. Without compression or with gzip, it works a expected.

The same happens when I try to access a restricted folder.

$ curl -H 'Accept-encoding: br' http://debian-test.local/config | brotli -cd
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:01:00 --:--:--     0
curl: (52) Empty reply from server
corrupt input [con]

After the 1 minute time-out, curl just stops. The log entry:

[07/Mar/2018:23:51:16 +0100] "GET /config HTTP/1.1" 403 0 gzip - br -

A Browser (Chrome) shows an error page with the error ERR_EMPTY_RESPONSE.

After that, I tried to get bigger text files ( larger than 2 MiB uncompressed) form this server. But now it gets a little bit weird. I tested it with Lorembigger.txt, which has a size of 2.4 MiB.

$ curl -H 'Accept-encoding: br' http://debian-test.local/Lorembigger.txt -o Lorembigger.txt.br
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  382k    0  382k    0     0   1237      0 --:--:--  0:05:01 --:--:--     ^C

I stopped it after 5 minutes. Every time I get a different size. The brotli compressed file is corrupt. And on the server I can see with top

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND  
 8560 nobody    20   0   50564   3544   2156 R 100.0  0.1   1:13.03 nginx 
 ...

At this point, I can't establish a new connection. It doesn't matter if the connection is http or https or uses compression. There is always a time-out. I need to kill and restart nginx. The log files shows nothing. Without compression or with gzip, it works a expected.

This happens also in a browser. The first part is visible but it is not the whole file. And the nginx process uses one core on the server and stops responding.

[centminmod]

@xabbu maybe related to default brotli_buffers size set as to how you handle larger txt file compression ?

But I'm also testing the rewrite branch fixes and same result as @xabbu = not fixed yet at least for not found files but do get 404 status but large file works

before fixes nginx 1.13.9 + ngx_brotli

curl -H 'Accept-encoding: br' https://domain.com/notfound | brotli -cd
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   162  100   162    0     0    439      0 --:--:-- --:--:-- --:--:--   439
corrupt input [con]

after fixes from your 2 commits in https://github.com/eustas/ngx_brotli/blob/9fa33e998b6c44b1becbdef807c8af15c30c1df6/src/ngx_http_brotli_filter_module.c

curl -H 'Accept-encoding: br' https://domain.com/notfound | brotli -cd
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   162  100   162    0     0    560      0 --:--:-- --:--:-- --:--:--   562
corrupt input [con]

curl -I https://domain.com/notfound
HTTP/2 404 
date: Wed, 07 Mar 2018 23:49:26 GMT
content-type: text/html; charset=utf-8
content-length: 162
server: nginx centminmod
x-powered-by: centminmod

using settings

brotli on;
brotli_static on;
brotli_min_length 1000;
brotli_buffers 1024 8k;
brotli_comp_level 5;
brotli_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;

nginx config

nginx -V
nginx version: nginx/1.13.9
built by gcc 8.0.1 20180304 (experimental) (GCC) 
built with OpenSSL 1.1.1-pre2 (alpha) 27 Feb 2018
TLS SNI support enabled

configure arguments: --with-ld-opt='-L/usr/local/lib -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-I/usr/local/include -m64 -march=native -DTCP_FASTOPEN=23 -g -O3 -Wno-error=strict-aliasing -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wno-cast-function-type -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-compat --with-http_stub_status_module --with-http_secure_link_module --add-dynamic-module=../nginx-module-vts --with-libatomic --with-http_gzip_static_module --add-dynamic-module=../ngx_brotli --add-dynamic-module=../incubator-pagespeed-ngx-1.13.35.2-stable --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.4.2 --add-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.31 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.14 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre=../pcre-8.41 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-http_v2_hpack_enc --with-openssl=../openssl-1.1.1-pre2 --with-openssl-opt='enable-ec_nistp_64_gcc_128 enable-tls1_3'

larger lorem.txt

With larger ~2144KB lorem.txt file after fixes from your 2 commits in https://github.com/eustas/ngx_brotli/blob/9fa33e998b6c44b1becbdef807c8af15c30c1df6/src/ngx_http_brotli_filter_module.c

using settings

brotli on;
brotli_static on;
brotli_min_length 1000;
brotli_buffers 1024 8k;
brotli_comp_level 5;
brotli_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;

curl -H 'Accept-encoding: br' https://domain.com/lorem.txt | brotli -cd | head -n10
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6036    0  6036    0     0  52034      0 --:--:-- --:--:-- --:--:-- 52034
Semper velit facilisi nullam litora dui habitasse porta montes et pellentesque, amet nunc elementum neque. Felis molestie blandit platea tempor vivamus nisi, dolor praesent hac accumsan gravida. Integer luctus dolor varius a.

Morbi varius. Lorem in a dictumst venenatis quam convallis. Lobortis tellus netus dictumst ligula et dictumst suspendisse lobortis nisl torquent Purus iaculis tortor ornare aliquet aliquam praesent platea ac integer ullamcorper mauris.

Primis sagittis. Malesuada faucibus id eget habitant faucibus. In varius per auctor elementum egestas. Nonummy sapien.

Fermentum primis ullamcorper enim vel nunc aenean ornare penatibus dui pretium, lacinia fusce diam eleifend vestibulum nonummy interdum pede magnis luctus cras At mattis suscipit.

Posuere augue. Parturient potenti diam, lacinia nisl quis. Nisi ipsum luctus fringilla lacus est vivamus posuere magnis dapibus curae; feugiat elementum mollis lorem torquent ridiculus luctus mauris. Penatibus. Posuere cras taciti.

curl -I https://domain.com/lorem.txt
HTTP/2 200 
date: Thu, 08 Mar 2018 00:03:41 GMT
content-type: text/plain; charset=utf-8
content-length: 2194948
last-modified: Thu, 08 Mar 2018 00:01:44 GMT
vary: Accept-Encoding
etag: "5aa07d68-217e04"
server: nginx centminmod
x-powered-by: centminmod
expires: Fri, 09 Mar 2018 00:03:41 GMT
cache-control: max-age=86400
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
accept-ranges: bytes

curl -H 'Accept-encoding: br' -I https://domain.com/lorem.txt
HTTP/2 200 
date: Thu, 08 Mar 2018 00:04:17 GMT
content-type: text/plain; charset=utf-8
last-modified: Thu, 08 Mar 2018 00:01:44 GMT
vary: Accept-Encoding
etag: W/"5aa07d68-217e04"
server: nginx centminmod
x-powered-by: centminmod
expires: Fri, 09 Mar 2018 00:04:17 GMT
cache-control: max-age=86400
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
content-encoding: br

curl -H 'Accept-encoding: gzip' -I https://domain.com/lorem.txt  
HTTP/2 200 
date: Thu, 08 Mar 2018 00:04:41 GMT
content-type: text/plain; charset=utf-8
last-modified: Thu, 08 Mar 2018 00:01:44 GMT
vary: Accept-Encoding
etag: W/"5aa07d68-217e04"
server: nginx centminmod
x-powered-by: centminmod
expires: Fri, 09 Mar 2018 00:04:41 GMT
cache-control: max-age=86400
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
content-encoding: gzip

curl tests for sizes with curltest

brotli

curltest br https://domain.com/lorem.txt
URI: https://domain.com/lorem.txt (br)
Uncompressed size : 2.09 MiB
Compressed size   : 5.89 KiB

gzip

curltest gzip https://domain.com/lorem.txt
URI: https://domain.com/lorem.txt (gzip)
Uncompressed size : 2.09 MiB
Compressed size   : 23.00 KiB

large file brotli test

ok here

curl -o lorem.txt -H 'Accept-encoding: br' https://domain.com/lorem.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6036    0  6036    0     0  52486      0 --:--:-- --:--:-- --:--:-- 52486

brotli -t lorem.txt

brotli -V
brotli 1.0.2

[centminmod]

ok retest with latest rewrite branch file https://github.com/eustas/ngx_brotli/blob/83a3066181bc8e86ea0581b6f5b561aefb004d3c/src/ngx_http_brotli_filter_module.c

curl -H 'Accept-encoding: br' https://domain.com/notfound | brotli -cd
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   162  100   162    0     0   1557      0 --:--:-- --:--:-- --:--:--  1557
corrupt input [con]

looks good in logging

tail -1 /var/log/nginx/localhost.access.log 
xxx.xxx.xxx.xxx - - [08/Mar/2018:23:17:41 +0000] "GET /notfound HTTP/2.0" 404 162 "-" "curl/7.58.0" "-" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=-

tail -1 /var/log/nginx/localhost.error.log  
2018/03/08 23:17:41 [error] 14028#14028: *1 open() "/usr/local/nginx/html/notfound" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: domain.com, request: "GET /notfound HTTP/2.0", host: "domain.com"

[xabbu]

@centminmod I noticed in your setting, you use brotli_min_length 1000;. The 404 replay is only +- 162 bytes big, so nginx should not use the brotli compression on this. But a plain/html response can't be decompressed by the brotli binary. Which results in a corrupt input message. Can you try the curl command without | brotli -cd, you should be able to read the curl output.

@eustas I tried the new commit and it fixes for me the nginx http status response problem. I didn't see any "Empty reply from server" message. All responses are valid brotli compressed files.

I also tried to get my big file problem a little bit more reproducible.
This time I wanted a text file that is available for everybody. I looked for one of the bigger books and then used "War and Peace" by Tolstoy (English translation), it can be found via Project Gutenberg.
https://www.gutenberg.org/files/2600/2600-0.txt

I used a QEMU virtualized Debian stretch. This is my config
https://gist.github.com/xabbu/38dac1d8d1794a7eb7ea571b2248ef6a

I noticed sometimes I can download the brotli compressed 2600-0.txt 10 times before nginx stops responding and sometimes it happens on the first try.

Then I used a little bash script with a while loop to download the compressed file as often as possible until nginx stops responding.

#!/bin/bash
i=1
while [ true ]; do
  curl -s -H 'Accept-encoding: br' -o /tmp/2600-0.txt.br http://debian-test.local/2600-0.txt
  echo "$i"
  sleep 2
  i=`expr $i + 1`
done

I'm the only one who connects to this nginx server and I only made one http connection a time. At some point nginx just stops responding and I can see with top something like this

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND 
  540 www-data  20   0   39092   4192   2932 R 100.0  0.1   0:17.00 nginx 

With brotli_comp_level 1; it took on average 12 downloads until nginx hangs. When nginx hangs, I rebooted the instance and run the script again. With brotli_comp_level 2; it took on average 23 downloads until nginx hangs. I also tried brotli_comp_level 3;, it was on average 24 downloads until nginx hangs. The highest download count was 87 times. But my sample runs per compression level are not that often, only 20 times.

Tomorrow, I will try it on a real vServer. I probably use a simple Wordpress installation. The target file will be probably wp-includes/js/tinymce/tinymce.min.js or wp-includes/js/plupload/moxie.js

[centminmod]

cheers @xabbu without brotli -cd command

curl -H 'Accept-encoding: br' https://domain.com/notfound | tail -10
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   162  100   162    0     0   1542      0 --:--:-- --:--:-- --:--:--  1542
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

Tested your script out on my HTTP/2 HTTPS domain with ngx_brotli and my default brotli_comp_level 5 and no hangs at all my OVH MC-32 i7 4790K Haswell with 32Gb ram and 2x240GB SSD raid 1 CentOS 7.4. My nginx server is built using jemalloc memory allocator and not system glibc malloc though.

pidstat

pidstat -durh -C nginx 1

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1520563289  1000      2806    0.00    0.00    0.00    0.00     3      0.00      0.00  178416  56408   0.17      0.00      4.00      0.00  nginx

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1520563290  1000      2802   10.00    0.00    0.00   10.00     0   2792.00      0.00  178416  56636   0.17      0.00      0.00      0.00  nginx

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1520563292  1000      2790   11.00    0.00    0.00   11.00     3   2794.00      0.00  178416  56628   0.17      0.00      0.00      0.00  nginx

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1520563294  1000      2784   10.00    1.00    0.00   11.00     2   2791.00      0.00  178416  56724   0.17      0.00      0.00      0.00  nginx

top

top -c
top - 02:35:02 up  1:07,  2 users,  load average: 0.09, 0.06, 0.07
Tasks: 208 total,   1 running, 207 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  3.3 us,  0.0 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  :  2.7 us,  1.3 sy,  0.0 ni, 96.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32742492 total, 29520288 free,  2490012 used,   732192 buff/cache
KiB Swap:  2095096 total,  2095096 free,        0 used. 29758216 avail Mem 
                                                                                                                                                                                                                                                         
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                              
 2797 nginx     10 -10  178416  56660   3008 S   3.7  0.2   0:02.60 nginx

nginx -V
nginx version: nginx/1.13.9
built by gcc 8.0.1 20180304 (experimental) (GCC) 
built with OpenSSL 1.1.1-pre2 (alpha) 27 Feb 2018
TLS SNI support enabled

configure arguments: --with-ld-opt='-L/usr/local/lib -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-I/usr/local/include -m64 -march=native -DTCP_FASTOPEN=23 -g -O3 -Wno-error=strict-aliasing -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wno-cast-function-type -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-compat --with-http_stub_status_module --with-http_secure_link_module --add-dynamic-module=../nginx-module-vts --with-libatomic --with-http_gzip_static_module --add-dynamic-module=../ngx_brotli --add-dynamic-module=../incubator-pagespeed-ngx-1.13.35.2-stable --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.4.2 --add-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.31 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.14 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre=../pcre-8.41 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-http_v2_hpack_enc --with-openssl=../openssl-1.1.1-pre2 --with-openssl-opt='enable-ec_nistp_64_gcc_128 enable-tls1_3'

curl -I -H 'Accept-encoding: br' https://domain.com/2600-0.txt
HTTP/2 200 
date: Fri, 09 Mar 2018 02:29:03 GMT
content-type: text/plain; charset=utf-8
last-modified: Sat, 17 Dec 2016 11:20:56 GMT
vary: Accept-Encoding
etag: W/"58551f98-33433e"
server: nginx centminmod
x-powered-by: centminmod
expires: Sat, 10 Mar 2018 02:29:03 GMT
cache-control: max-age=86400
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
content-encoding: br

curltest br https://domain.com/2600-0.txt
URI: https://domain.com/2600-0.txt (br)
Uncompressed size : 3.20 MiB
Compressed size   : 1.09 MiB

curltest gzip https://domain.com/2600-0.txt
URI: https://domain.com/2600-0.txt (gzip)
Uncompressed size : 3.20 MiB
Compressed size   : 1.19 MiB

[eustas]

Going to try to investigate today

[centminmod]

thanks @eustas

[eustas]

Can't reproduce on my desktop. On the other side, found that lldb works better than gdb =)
Going to dig further to see why nginx stucks...

[eustas]

Btw this explains why adding more logging helps - socket happens to send all the data before more bytes are pushed...