502 with high request rate

[erikvl87]

502 with high request rate

[calumgoodchild]

@Erikvl87 apologies for not getting back to you, we found it to be an nginx issue.

Running it through an nginx lb resulted in the 502 errors regardless of the containers being on separate machines or not. So, we created a piece of code to manually load balance the requests, this increased the rate significantly, and with 3 machines running 1 container per machine, we were able to reach around 800 rps.

I would like to try running this through an external lb rather than nginx, but this may come in the future. Or, maybe a distributed lb over multiple machines.

I tried a load of configs in nginx to increase rps/ performance, but, to no avail.

Thanks for your help and I hope someone else can find this insightful.

[Erikvl87]

I haven't had the time yet to create a proper reproduceable case, but I guess the HTTP 502 response is coming from the LanguageTool HTTPServer. If you agree with that I would advice you to open up an issue at the official LanguageTool repository first: https://github.com/languagetool-org/languagetool/issues

I am not sure about your setup but have you looked at configuring a load balancer, e.g. using Docker Swarm?

[calumgoodchild]

Strange, because I've tested it behind an nginx load balancer with multiple containers but the rps does not increase. It's not nginx throttling the requests because I've also tested direct to the container.

I'll open a ticked on the LanguageTool repo.

Cheers for coming back.

[Erikvl87]

If I have time next weekend I'll try to reproduce to check if it has something to do with the Dockerfile or with LanguageTool itself. If you happen to have some easy steps to reproduce, please add them in both this ticket and the one you are creating in the LanguageTool repo.

[Erikvl87]

Did you find the cause of the HTTP 502 errors? I did a small test with the configuration below, but sending out lots of requests did not result in HTTP 502 errors.

I did had the same issue as you where scaling the app (on the same machine) didn't increase the RPM (sometimes the RPM even lowered instead of increased when scaling to multiple containers) but that could be due to network-, memory- or CPU limitations or improper LanguageTool configuration (I have been playing with the settings pipelinePrewarming, pipelineCaching and maxCheckThreads and memory settings with little or no avail). Nor do I know if running multiple virtual images on the same physical machine should increase the RPM or that a single instance with different configuration would be more appropriate. I think those are great questions to ask at the official LanguageTool repository. Did you by any chance already opened up a ticket there?

My tests were on a laptop (Intel® Core™ i7-10710U with 16GB ram) running Docker on Windows 10 using the WSL 2 based engine.

docker-compose.yml

version: "3.8"

services: 
    langtool:
        image: erikvl87/languagetool
        environment:
            - langtool_pipelinePrewarming=true
            - langtool_pipelineCaching=true
            - Java_Xms=2g
            - Java_Xmx=2g
        networks: 
            - "langtool_net"

    ngnix:
        image: nginx:1.19.1
        ports: 
            - "8080:80"
        volumes: 
            - './nginx.conf:/etc/nginx/nginx.conf:ro'
        networks: 
            - "langtool_net"
        depends_on: 
            - langtool

networks: 
    langtool_net:

nginx.config

events {
  worker_connections 4096;  ## Default: 1024
}

http {
    upstream app {
        server langtool:8010; ## Docker will redistribute the requests over the scale set
    }

    server {
        listen 80;

        location / {
            proxy_set_header HOST $host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass http://app;
        }
    }
}

startup cmd: docker-compose up --scale langtool=3

[Erikvl87]

@calumgoodchild, since I am not yet convinced your issue has anything to do with the Docker setup from this repository and I haven't heard from you for a while, I will be closing this issue. Feel free to leave a comment with details if you think otherwise and I will reopen this issue.

[alexanderad]

Hey @calumgoodchild, sorry for bringing this thread from ages ago, but would you mind sharing your final config for the LT?

And approximate specs of the machines you were running? Config is of interest to me to understand if fasttext / ngrams influence the performance. And finally the relation between threads and CPUs of the machines you had on what's going on there with the checkers threadpools.

Thanks! 🌟