download via nginx proxy max 1 GB

[eikek]

download via nginx proxy max 1 GB

[eikek]

Thank you!

I'm not so experienced with nginx. I just tried with my setup and it worked for me downloading a 2.2G file (tried with Chromium 80.0.3987.163 and wget 1.20 on nginx 1.16.1). My config is this

location / {
    proxy_pass http://sharry-intern:9090;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    client_max_body_size 105M;
    proxy_send_timeout   300s;
    proxy_read_timeout   300s;
    send_timeout         300s;
}

Do you have any idea why this can work here?

The doc says "The zero value disables buffering of responses to temporary files." Do you know if this means that the response must fit in memory (if buffering is enabled)?

Maybe it's even better to disable buffering responses from the body, wdyt?

[hoedlmoser]

here its Chromium 80.0.3987.163, nginx 1.14.2 for sharry and an other instance for https-offloading (on an other virtual machine) of same version.

Do you have any idea why this can work here?

maybe you set proxy_buffering off somewhere globally? thats what I've done on my https-offloading nginx. check with nginx -T and search for this parameter.

they are describing here https://trac.nginx.org/nginx/ticket/1472 kind of timeout by backend: "if the limit is reached, nginx will stop reading from the backend till all disk-buffered data are sent to the client. This in turn can result in a send timeout on the backend side."
my nginx error log reports such a backend timeout.
2020/04/11 16:31:59 [error] 791#791: *176597 readv() failed (104: Connection reset by peer) while reading upstream, client: 10.21.11.66, server: sharry.rekmp.net, request: "GET /api/v2/sec/share/CeR3a5qjEBd-v3pJ7hnEW14-1inSKT76Nqp-8S7nsVQxbzX/file/HHyNz1JLBjx-9SDGQmirhze-VXSRWKw8ufZ-SLBN6xK7Sa7 HTTP/1.0", upstream: "http://127.0.0.1:9090/api/v2/sec/share/CeR3a5qjEBd-v3pJ7hnEW14-1inSKT76Nqp-8S7nsVQxbzX/file/HHyNz1JLBjx-9SDGQmirhze-VXSRWKw8ufZ-SLBN6xK7Sa7", host: "sharry.rekmp.net"
maybe your downlink is just that fast that you will not experience that backend timeout.

Do you know if this means that the response must fit in memory (if buffering is enabled)?

if I do not misunderstand http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering and https://www.scalescale.com/tips/nginx/optimizing-nginx-for-serving-files-bigger-than-1gb/ I would clearly say: no, need not fit into memory.

Maybe it's even better to disable buffering responses from the body, wdyt?

I would propose disabling buffering in any case. buffering means loosing bandwith you later do not have.

this is my config

  location / {
    proxy_pass http://localhost:9090;
    include /etc/nginx/proxy_params;

    # https://eikek.github.io/sharry/doc/reverseproxy#nginx
    proxy_http_version 1.1;
    #proxy_set_header Upgrade $http_upgrade;
    #proxy_set_header Connection $connection_upgrade;
    client_max_body_size 105M;
    proxy_max_temp_file_size 0;
    proxy_send_timeout   300s;
    proxy_read_timeout   300s;
    send_timeout         300s;
    location = / {
      return 302 /app;
    }
  }

[eikek]

I see, thank you for all the details. Yes, I then also think it is better to disable buffering in nginx. This only introduces backpressure problems.

And yes, I usually test with a too good connection. I haven't tested with slow connections yet, which I should do more seriously. It also means to provide more options to configure the sharry http server, for example to set custom timeouts. Currently for slow connections, one can only use a lower chunk size (not 100M). I will invest into these things for the next release.