[BUG] scheduler crash after renew letsencrypt

[thelittlefireman]

[BUG] scheduler crash after renew letsencrypt

[TheophileDiot]

Hi @thelittlefireman, could you provide your configuration please ?

[TheophileDiot]

@thelittlefireman bump

[thelittlefireman]

[2023-10-27 22:59:25] - API - ❌ - Can't send API request to http://nginx:5100/reload : Request failed: HTTPConnectionPool(host='nginx', port=5100): Max retries exceeded with url: /reload (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd160a1b3e0>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2023-10-27 22:59:25] - SCHEDULER - ❌ - Error while reloading nginx
[2023-10-27 22:59:26] - SCHEDULER - ℹ️  - Executing job bunkernet-register from plugin bunkernet ...
[2023-10-27 22:59:27] - BUNKERNET - ℹ️  - BunkerNet is not activated, skipping registration...
[2023-10-27 22:59:27] - SCHEDULER - ℹ️  - Executing job greylist-download from plugin greylist ...
[2023-10-27 22:59:27] - SCHEDULER - ℹ️  - Successfully updated database for the job bunkernet-register from plugin bunkernet
[2023-10-27 22:59:28] - GREYLIST - ℹ️  - Greylist is not activated, skipping downloads...
[2023-10-27 22:59:28] - SCHEDULER - ℹ️  - Successfully updated database for the job greylist-download from plugin greylist
[2023-10-27 22:59:29] - SCHEDULER - ℹ️  - Executing job realip-download from plugin realip ...
[2023-10-27 22:59:30] - REALIP - ℹ️  - RealIP is not activated, skipping download...
[2023-10-27 22:59:30] - SCHEDULER - ℹ️  - Successfully updated database for the job realip-download from plugin realip
[2023-10-27 22:59:31] - SCHEDULER - ℹ️  - Executing job whitelist-download from plugin whitelist ...
[2023-10-27 22:59:33] - WHITELIST - ℹ️  - ✅ Database connection established
[2023-10-27 22:59:33] - WHITELIST - ℹ️  - Whitelist for IP is not cached, processing downloads..
[2023-10-27 22:59:33] - WHITELIST - ℹ️  - Whitelist for RDNS is not cached, processing downloads..
[2023-10-27 22:59:33] - WHITELIST - ℹ️  - Whitelist for ASN is not cached, processing downloads..
[2023-10-27 22:59:33] - WHITELIST - ℹ️  - Whitelist for USER_AGENT is not cached, processing downloads..
[2023-10-27 22:59:33] - WHITELIST - ℹ️  - Whitelist for URI is not cached, processing downloads..
[2023-10-27 22:59:33] - SCHEDULER - ℹ️  - Successfully updated database for the job whitelist-download from plugin whitelist
[2023-10-27 23:57:07] - SCHEDULER - ❌ - An error occurred when checking for changes in the database : Traceback (most recent call last):
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
    self.dialect.do_execute(
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/default.py", line 922, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/share/bunkerweb/deps/python/pymysql/cursors.py", line 153, in execute
    result = self._query(query)
             ^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/pymysql/cursors.py", line 322, in _query
    conn.query(q)
  File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 558, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 822, in _read_query_result
    result.read()
  File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 1200, in read
    first_packet = self.connection._read_packet()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 739, in _read_packet
    packet_header = self._read_bytes(4)
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 795, in _read_bytes
    raise err.OperationalError(
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/share/bunkerweb/db/Database.py", line 281, in check_changes
    .first()
     ^^^^^^^
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/query.py", line 2748, in first
    return self.limit(1)._iter().first()  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/query.py", line 2847, in _iter
    result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
                                                  ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 2306, in execute
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 2188, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
    result = conn.execute(
             ^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1416, in execute
    return meth(
           ^^^^^
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1848, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
    self._handle_dbapi_exception(
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
    self.dialect.do_execute(
  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/default.py", line 922, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/share/bunkerweb/deps/python/pymysql/cursors.py", line 153, in execute
    result = self._query(query)
             ^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/pymysql/cursors.py", line 322, in _query
    conn.query(q)
  File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 558, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 822, in _read_query_result
    result.read()
  File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 1200, in read
    first_packet = self.connection._read_packet()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 739, in _read_packet
    packet_header = self._read_bytes(4)
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 795, in _read_bytes
    raise err.OperationalError(
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
[SQL: SELECT bw_metadata.custom_configs_changed AS bw_metadata_custom_configs_changed, bw_metadata.external_plugins_changed AS bw_metadata_external_plugins_changed, bw_metadata.config_changed AS bw_metadata_config_changed, bw_metadata.instances_changed AS bw_metadata_instances_changed 
FROM bw_metadata 
WHERE bw_metadata.id = %(id_1)s 
 LIMIT %(param_1)s]
[parameters: {'id_1': 1, 'param_1': 1}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

[2023-10-27 23:57:07] - ENTRYPOINT - ℹ️  - Scheduler stopped

[thelittlefireman]

I'm also losing connection from scheduler and nginx really often. (not stable)

[thelittlefireman]

version: '3.5'

networks:
  bw-universe:
    name: bw-universe
    external: true

  bw-docker:
    name: bw-docker
    external: true

  backend-net:
    external: true
    name: backend-net

services:
  authelia:
    image: authelia/authelia
    container_name: authelia
    user: "1024:101"
    environment:
      - PUID=1024
      - PGID=101
    volumes:
      - ./authelia:/config
      - "/etc/TZ:/etc/timezone:ro"
      - "/etc/localtime:/etc/localtime:ro"
      - "./nginx/letsencrypt:/certs:ro"
    restart: unless-stopped
    ports:
      - 127.0.0.1:9091:9091
    environment:
      - TZ=France/Paris
    networks:
      - backend-net

  bw-syslog-ng:
    image: lscr.io/linuxserver/syslog-ng:latest
    container_name: bw-syslog-ng
    environment:
      - PUID=1024
      - PGID=101
    ports:
      - 514:5114/udp
    volumes:
      - ./bw-syslog-ng/config:/config
      - ./bw-syslog-ng/log:/var/log
    networks:
      - backend-net

  bw-docker-proxy:
    image: tecnativa/docker-socket-proxy:nightly
    container_name: bw-docker-proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - CONTAINERS=1
      - LOG_LEVEL=warning
    networks:
      - bw-docker

  bw-scheduler:
    image: bunkerity/bunkerweb-scheduler:dev
    container_name: bw-scheduler
    depends_on:
      - bw-docker-proxy
      - bw-db
      - bw-syslog-ng
    logging:
      driver: syslog
      options:
        syslog-address: "udp://localhost:514"
    volumes:
      - ./nginx/letsencrypt:/var/cache/bunkerweb/letsencrypt/etc:mode=0770,uid=101,gid=101
      - ./nginx/config:/data
    environment:
      - DOCKER_HOST=tcp://bw-docker-proxy:2375
      - API_HTTP_PORT=5100
      - API_SERVER_NAME=nginx
      - LOG_LEVEL=notice
      - DATABASE_URI=mysql+pymysql://bunkerweb:XXXXXXXXXXX@bw-db:3306/db
    networks:
      - bw-universe
      - bw-docker
# Fix nginx running in host network
    extra_hosts:
      - "nginx:192.168.1.150"

  bw-db:
    image: mysql:latest
    container_name: bw-db
    user: "1024:101"
    environment:
      - MYSQL_RANDOM_ROOT_PASSWORD=yes
      - MYSQL_DATABASE=db
      - MYSQL_USER=bunkerweb
      - MYSQL_PASSWORD=XXXXXXXXXXX
    expose:
      - 3306
    volumes:
      - "./nginx/db:/var/lib/mysql"
    networks:
      - bw-universe

  nginx:
    image: bunkerity/bunkerweb:dev
    container_name: nginx
    restart: always
    network_mode: "host"
    labels:
      - "bunkerweb.INSTANCE" # required for the scheduler to recognize the container
    volumes:
      # Set timezone
      - "/etc/TZ:/etc/timezone:ro"
      - "/etc/localtime:/etc/localtime:ro"
    # dropping all capabilities
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges
    depends_on:
      - authelia
      - bw-scheduler
      - bw-docker-proxy
      - bw-db
      - bw-syslog-ng
    logging:
      driver: syslog
      options:
        syslog-address: "udp://localhost:514"
    environment:
      - EXTERNAL_PLUGIN_URLS=https://github.com/bunkerity/bunkerweb-plugins/archive/refs/tags/v1.1.zip
      - DATABASE_URI=mysql+pymysql://bunkerweb:XXXXXXXXXXX@bw-db:3306/db
      - MULTISITE=yes
      - LOG_LEVEL=notice
      - API_WHITELIST_IP=127.0.0.0/8 172.16.3.0/24 172.16.4.0/24
      - API_HTTP_PORT=5100
      - API_SERVER_NAME=nginx
      - USE_LETS_ENCRYPT_STAGING=no
      - AUTO_LETS_ENCRYPT=yes
      - [email protected]
      - SERVER_NAME=auth.test.te syno.test.te nextcloud.test.te
      - SERVE_FILES=no
      - REDIRECT_HTTP_TO_HTTPS=yes
      - DISABLE_DEFAULT_SERVER=yes
      - DNS_RESOLVERS=1.1.1.1 1.0.0.1
      - HTTP_PORT=10080
      - HTTPS_PORT=10443
      - HTTP2=yes
      - SSL_PROTOCOLS=TLSv1.2 TLSv1.3
      - USE_GZIP=yes
      - GZIP_COMP_LEVEL=5
      - GZIP_MIN_LENGTH=500
      - USE_BROTLI=yes
      - BROTLI_COMP_LEVEL=5
      - BROTLI_MIN_LENGTH=500
      - STRICT_TRANSPORT_SECURITY=max-age=31536000; includeSubDomains; preload
      - USE_BAD_BEHAVIOR=no
      - BAD_BEHAVIOR_STATUS_CODES=400 401 403 405 444 429
      - USE_DNSBL=yes
      - USE_BLACKLIST=yes
      - USE_WHITELIST=yes
      - WHITELIST_IP=127.0.0.1/8 192.168.0.0/16
      - GREYLIST_USER_AGENT=okhttp* WebDAV DAVx5* axios* sindresorhus* *ggpht.com GoogleImageProxy Python* aiohttp* Bitwarden_Mobile*
      - USE_REVERSE_PROXY=yes
      - REVERSE_PROXY_CONNECT_TIMEOUT=180s
      - REVERSE_PROXY_SEND_TIMEOUT=180s
      - REVERSE_PROXY_READ_TIMEOUT=180s
      - REVERSE_PROXY_URL_999=/authelia
      - REVERSE_PROXY_HOST_999=https://127.0.0.1:9091/api/verify
      - REVERSE_PROXY_HEADERS_999=X-Original-URL $$scheme://$$http_host$$request_uri;Content-Length ""
      - auth.test.te_USE_REVERSE_PROXY=yes
      - auth.test.te_REVERSE_PROXY_URL=/
      - auth.test.te_REVERSE_PROXY_HOST=https://127.0.0.1:9091
      - auth.test.te_REVERSE_PROXY_INTERCEPT_ERRORS=no
      - |
        auth.test.te_CUSTOM_CONF_SERVER_HTTP_auto-custom-auth=
        proxy_busy_buffers_size 256k;
        proxy_buffers 8 128k;
        proxy_buffer_size 128k;
      - nextcloud.test.te_REVERSE_PROXY_URL=/
      - nextcloud.test.te_REVERSE_PROXY_HOST=http://127.0.0.1:32680
      - nextcloud.test.te_REVERSE_PROXY_KEEPALIVE=yes
      - nextcloud.test.te_REVERSE_PROXY_BUFFERING=no
      - nextcloud.test.te_ALLOWED_METHODS=GET|POST|HEAD|COPY|DELETE|LOCK|MKCOL|MOVE|PROPFIND|PROPPATCH|PUT|UNLOCK|OPTIONS|REPORT|SEARCH
      - nextcloud.test.te_MAX_CLIENT_SIZE=2G
      - nextcloud.test.te_CONTENT_SECURITY_POLICY=object-src 'none'; frame-ancestors 'self'; form-action 'self'; block-all-mixed-content; sandbox allow-forms allow-same-origin allow-scripts allow-popups allow-downloads; base-uri 'self';
      - nextcloud.test.te_BAD_BEHAVIOR_STATUS_CODES=400 401 403 405 444
      - nextcloud.test.te_X_CONTENT_TYPE_OPTIONS=
      - nextcloud.test.te_FEATURE_POLICY=accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'none'; battery 'none'; camera 'none'; display-capture 'none'; document-domain 'none'; encrypted-media 'none'; execution-while-not-rendered 'none'; execution-while-out-of-viewport 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; layout-animation 'none'; legacy-image-formats 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; navigation-override 'none'; payment 'none'; picture-in-picture 'self'; publickey-credentials-get 'none'; speaker-selection 'none'; sync-xhr 'none'; unoptimized-images 'none'; unsized-media 'none'; usb 'none'; screen-wake-lock 'none'; web-share 'none'; xr-spatial-tracking 'none';
      - nextcloud.test.te_PERMISSIONS_POLICY=accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), hid=(), idle-detection=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(self), publickey-credentials-get=(), screen-wake-lock=(), serial=(), usb=(), web-share=(), xr-spatial-tracking=()
      - nextcloud.test.te_REMOVE_HEADERS=x-content-type-options
[...]
      - USE_MODSECURITY=yes
      - USE_MODSECURITY_CRS=yes

      - USE_BUNKERNET=no
      - USE_API=no
      
      - USE_LIMIT_REQ=yes
      - LIMIT_REQ_RATE=200r/s

      - USE_LIMIT_CONN=yes
      - LIMIT_CONN_MAX_HTTP1=20
      - LIMIT_CONN_MAX_HTTP2=200
      
      - BLACKLIST_COUNTRY=CN RU
      
      - USE_OPEN_FILE_CACHE=yes
      - OPEN_FILE_CACHE=max=1000 inactive=60s
      - OPEN_FILE_CACHE_ERRORS=yes
      - OPEN_FILE_CACHE_MIN_USES=2
      - OPEN_FILE_CACHE_VALID=30s
      
        ### PLUGINS ###
      - USE_CROWDSEC=no
      - CROWDSEC_API=http://127.0.0.1:48080
      - CROWDSEC_API_KEY=XXXXXXXXXXXXXXXXX
      - USE_VIRUSTOTAL=no
      - USE_DISCORD=no
      - USE_SLACK=no
      - USE_CLAMAV=no
      - USE_CORAZA=no

[thelittlefireman]

I don't know if it' related but i've got watchtower container which update my scheduler, nginx etc containers.

maybe bw-scheduler and idkw it looses connection to nginx container.

[TheophileDiot]

Hi @thelittlefireman, thank you for your configuration. By the way you don't need to provide the settings in the scheduler container:

- API_HTTP_PORT=5100
- API_SERVER_NAME=nginx

[TheophileDiot]

Do you have issues sending the configuration from the scheduler to BunkerWeb with the setting:

- USE_API=no

?

[TheophileDiot]

This can be because of your Docker environment having unstable networks for some reason 🤔

[fl0ppy-d1sk]

Hello @thelittlefireman,

Can you try to disable watchtower and tell us if does anything ?

[thelittlefireman]

I switch to docker version v1.5.3 (vs dev). So watchtower doesn't update it every day and I don't new crash.

But I'm still loosing connection from scheduler to nginx after 1 day, which is quite annoying:

[2023-11-10 13:46:25] - API - ❌ - Can't send API request to http://nginx:5100/cache : Request failed: HTTPConnectionPool(host='nginx', port=5100): Max retries exceeded with url: /cache (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6aa3bfc920>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2023-11-10 13:46:25] - SCHEDULER - ❌ - Error while sending /var/cache/bunkerweb folder
[2023-11-10 13:46:25] - SCHEDULER - ℹ️  - Reloading nginx ...
[2023-11-10 13:46:25] - API - ❌ - Can't send API request to http://nginx:5100/reload : Request failed: HTTPConnectionPool(host='nginx', port=5100): Max retries exceeded with url: /reload (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6aa3bfe150>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2023-11-10 13:46:25] - SCHEDULER - ❌ - Error while reloading nginx

Nginx container seems to lose API_PORT (5100) after a will.

 XXXX@XXXX:~$ docker exec -it -u 0 nginx netstat -planet | grep 5100

Any clue ?

[fl0ppy-d1sk]

Hello @thelittlefireman,

Can you try with new 1.5.5 version ?

[TheophileDiot]

@thelittlefireman bump