Error When trying to Renew SSL Cert

[fleetssl]

Error When trying to Renew SSL Cert

[alexzorin]

I can't reproduce this using our servers - seems that even http://ns3.djpnetworks.com.au/.well-known/pki-validation/test produces the HTTP 500.

Does anything show up in /usr/local/cpanel/logs/error_log when you make such a request?

[IT-Dave]

[2020-03-07 18:30:05 +1100] info [cphttpd] Internal Server Error: "GET /.well-known/acme-challenge/3zKigH9BDiygXss1UsEk1olMYf8BltPAr7yGSQbZapA HTTP/1.1" 500 Error ID 1526b5f107c38
cpsrvd 1526b5f107c38: Cpanel::Exception::IO::FileOpenError/(XID 8xvcuz) The system failed to open the file “/usr/local/apache/htdocs/.well-known/acme-challenge/3zKigH9BDiygXss1UsEk1olMYf8BltPAr7yGSQbZapA” for reading because of an error: Permission denied
at /usr/local/cpanel/Cpanel/Server/Handlers/Httpd/Static.pm line 159, line 2.
Cpanel::Server::Handlers::Httpd::Static::_handle_path_if_exists("server_obj", Cpanel::Server=HASH(0x1fb4608), "setuid", "nobody", "path", "/usr/local/apache/htdocs/.well-known/acme-challenge/3zKigH9BD"..., "size_limit", 32768) called at /usr/local/cpanel/Cpanel/Server/Handlers/Httpd/Static.pm line 87
Cpanel::Server::Handlers::Httpd::Static::handle("server_obj", Cpanel::Server=HASH(0x1fb4608), "setuid", "nobody", "path", "/usr/local/apache/htdocs/.well-known/acme-challenge/3zKigH9BD"..., "size_limit", 32768) called at /usr/local/cpanel/Cpanel/Server/Handlers/Httpd.pm line 249
Cpanel::Server::Handlers::Httpd::_serve_static(Cpanel::Server::Handlers::Httpd=HASH(0x226f6f0), "ns3.djpnetworks.com.au", "/.well-known/acme-challenge/3zKigH9BDiygXss1UsEk1olMYf8BltPAr"...) called at /usr/local/cpanel/Cpanel/Server/Handlers/Httpd.pm line 95
Cpanel::Server::Handlers::Httpd::handler(Cpanel::Server::Handlers::Httpd=HASH(0x226f6f0)) called at cpsrvd.pl line 8686
cpanel::cpsrvd::_cphttpd_main_loop() called at cpsrvd.pl line 1284
cpanel::cpsrvd::handle_one_connection(13) called at cpsrvd.pl line 1109
cpanel::cpsrvd::script() called at cpsrvd.pl line 429

[2020-03-07 18:30:05 +1100] info [cphttpd] Internal Server Error: "GET /.well-known/acme-challenge/3zKigH9BDiygXss1UsEk1olMYf8BltPAr7yGSQbZapA HTTP/1.1" 500 Error ID 1526b5f107c38
cpsrvd 1526b5f107c38: Cpanel::Exception::IO::FileOpenError/(XID 6hqgsf) The system failed to open the file “/usr/local/apache/htdocs/.well-known/acme-challenge/3zKigH9BDiygXss1UsEk1olMYf8BltPAr7yGSQbZapA” for reading because of an error: Permission denied
at /usr/local/cpanel/Cpanel/Server/Handlers/Httpd/Static.pm line 159, line 2.
Cpanel::Server::Handlers::Httpd::Static::_handle_path_if_exists("server_obj", Cpanel::Server=HASH(0x1fb4608), "setuid", "nobody", "path", "/usr/local/apache/htdocs/.well-known/acme-challenge/3zKigH9BD"..., "size_limit", 32768) called at /usr/local/cpanel/Cpanel/Server/Handlers/Httpd/Static.pm line 87
Cpanel::Server::Handlers::Httpd::Static::handle("server_obj", Cpanel::Server=HASH(0x1fb4608), "setuid", "nobody", "path", "/usr/local/apache/htdocs/.well-known/acme-challenge/3zKigH9BD"..., "size_limit", 32768) called at /usr/local/cpanel/Cpanel/Server/Handlers/Httpd.pm line 249
Cpanel::Server::Handlers::Httpd::_serve_static(Cpanel::Server::Handlers::Httpd=HASH(0x226f738), "ns3.djpnetworks.com.au", "/.well-known/acme-challenge/3zKigH9BDiygXss1UsEk1olMYf8BltPAr"...) called at /usr/local/cpanel/Cpanel/Server/Handlers/Httpd.pm line 95
Cpanel::Server::Handlers::Httpd::handler(Cpanel::Server::Handlers::Httpd=HASH(0x226f738)) called at cpsrvd.pl line 8686
cpanel::cpsrvd::_cphttpd_main_loop() called at cpsrvd.pl line 1284
cpanel::cpsrvd::handle_one_connection(13) called at cpsrvd.pl line 1109
cpanel::cpsrvd::script() called at cpsrvd.pl line 429

[2020-03-07 18:30:05 +1100] info [cphttpd] Internal Server Error: "GET /.well-known/acme-challenge/3zKigH9BDiygXss1UsEk1olMYf8BltPAr7yGSQbZapA HTTP/1.1" 500 Error ID 1526b5f107c38
cpsrvd 1526b5f107c38: Cpanel::Exception::IO::FileOpenError/(XID 5jpvw7) The system failed to open the file “/usr/local/apache/htdocs/.well-known/acme-challenge/3zKigH9BDiygXss1UsEk1olMYf8BltPAr7yGSQbZapA” for reading because of an error: Permission denied
at /usr/local/cpanel/Cpanel/Server/Handlers/Httpd/Static.pm line 159, line 2.
Cpanel::Server::Handlers::Httpd::Static::_handle_path_if_exists("server_obj", Cpanel::Server=HASH(0x1fb4608), "setuid", "nobody", "path", "/usr/local/apache/htdocs/.well-known/acme-challenge/3zKigH9BD"..., "size_limit", 32768) called at /usr/local/cpanel/Cpanel/Server/Handlers/Httpd/Static.pm line 87
Cpanel::Server::Handlers::Httpd::Static::handle("server_obj", Cpanel::Server=HASH(0x1fb4608), "setuid", "nobody", "path", "/usr/local/apache/htdocs/.well-known/acme-challenge/3zKigH9BD"..., "size_limit", 32768) called at /usr/local/cpanel/Cpanel/Server/Handlers/Httpd.pm line 249
Cpanel::Server::Handlers::Httpd::_serve_static(Cpanel::Server::Handlers::Httpd=HASH(0x226f6f0), "ns3.djpnetworks.com.au", "/.well-known/acme-challenge/3zKigH9BDiygXss1UsEk1olMYf8BltPAr"...) called at /usr/local/cpanel/Cpanel/Server/Handlers/Httpd.pm line 95
Cpanel::Server::Handlers::Httpd::handler(Cpanel::Server::Handlers::Httpd=HASH(0x226f6f0)) called at cpsrvd.pl line 8686
cpanel::cpsrvd::_cphttpd_main_loop() called at cpsrvd.pl line 1284
cpanel::cpsrvd::handle_one_connection(13) called at cpsrvd.pl line 1109
cpanel::cpsrvd::script() called at cpsrvd.pl line 429

[2020-03-07 18:30:06 +1100] info [cphttpd] Internal Server Error: "GET /.well-known/acme-challenge/3zKigH9BDiygXss1UsEk1olMYf8BltPAr7yGSQbZapA HTTP/1.1" 500 Error ID 1526b5f107c38

[alexzorin]

Great, that reveals the reason (the nobody user doesn't seem to be able to access files in that directory). I don't understand why it is happening, though.

The entire directory tree should be 0755 and the file itself is created as 0644 - world readable. What are the permissions on this directory:

stat /usr/local/apache/htdocs/.well-known/

[IT-Dave]

[root@ns3 ~]# stat /usr/local/apache/htdocs/.well-known/
File: ‘/usr/local/apache/htdocs/.well-known/’
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd01h/64769d Inode: 785814 Links: 4
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-02-21 18:00:19.842819363 +1100
Modify: 2020-03-06 02:30:22.035701825 +1100
Change: 2020-03-06 02:30:22.035701825 +1100
Birth: -

[alexzorin]

😖 . While I am testing some things, could you please also try these:

namei -l /usr/local/apache/htdocs/.well-known/acme-challenge/

and

sudo -u nobody namei -l /usr/local/apache/htdocs/.well-known/acme-challenge/

I appreciate your quick replies.

[IT-Dave]

[root@ns3 ~]# namei -l /usr/local/apache/htdocs/.well-known/acme-challenge/
f: /usr/local/apache/htdocs/.well-known/acme-challenge/
dr-xr-xr-x root root /
drwxr-xr-x root root usr
drwxr-xr-x root root local
drwx------ root root apache
drwxr-xr-x root root htdocs
drwxr-xr-x root root .well-known
drwxr-xr-x root root acme-challenge

[IT-Dave]

[root@ns3 ~]# sudo -u nobody namei -l /usr/local/apache/htdocs/.well-known/acme-challenge/
f: /usr/local/apache/htdocs/.well-known/acme-challenge/
dr-xr-xr-x root root /
drwxr-xr-x root root usr
drwxr-xr-x root root local
drwx------ root root apache
htdocs - No such file or directory

[alexzorin]

Great, we found the problem. The permissions on /usr/local/apache/ are 0700 for some reason - should be 0755.

I have installed DNSONLY 3 times since starting to investigate this ticket, and each time the permissions have been correct:

[root@epoxy-truck ~]# sudo -u nobody namei -l /usr/local/apache/htdocs/.well-known/acme-challenge/
f: /usr/local/apache/htdocs/.well-known/acme-challenge/
drwxr-xr-x root root /
drwxr-xr-x root root usr
drwxr-xr-x root root local
drwxr-xr-x root root apache
drwxr-xr-x root root htdocs
drwxr-xr-x root root .well-known
drwxr-xr-x root root acme-challenge

You could of course chmod it back to the intended 0755, but the fact that it happened on two of your servers kind of concerns me.

I do not think that cPanel intends it to be 0700 at any time (because it breaks really basic functionality that cPanel relies on) ... so it must be something else that is doing it. What that thing is though, I have no idea. Other third party packages? Automation scripts? 🤷‍♂️

[IT-Dave]

Just running PowerDNS, NSCD, MariaDB 10.3, Config Server Friewall, Munin Service Monitor and thats it.

[IT-Dave]

I also had issues in the past with the SSL Renewing itself so i have just setup a cronjob i tried to look into the issue previously but never could work out really where things where going wrong.

0 3 29 */2 * /usr/local/bin/fleetssl-dnsonly > /dev/null 2>&1

[PudottaPommin]

I had same problem as a result ... I have 2 VPS instances installed year apart ( 2-3 years ago first and 2nd 1-2 years ago ). I wasn't able to get the certificate working. Both of them had same settings, but errors differed. After fresh installation of DNSONLY no issues with certificate and it worked perfectly. My take is that probably some problems happened when DNSONLY updated to newer versions ( I think there was at some point update from EasyApache 3 to EasyApache 4 ).

[alexzorin]

@PudottaPommin thanks, that's some pretty useful information. Unfortunately I've suffered an injury which prevents me typing easily so there might be a bit of a delay on investigating.