NODE_EXTRA_CA_CERTS with "unable to verify the first certificate"

[travisghansen]

NODE_EXTRA_CA_CERTS with "unable to verify the first certificate"

[weishiuchang]

Just to show the cert from dex matches what's in NODE_EXTRA_CA_CERTS
(Commands performed from the eas pod as kubectl exec)

eas@external-auth-server-5ddd6ffdb7-xkdkc:~/app$ echo | openssl s_client -connect dex.auth-system.svc.cluster.local:443 2>/dev/null | openssl x509 -subject -nocert
subject=C = US, ST = Colorado, L = Boulder, O = Drolrevo, CN = *.drolrevo.com

eas@external-auth-server-5ddd6ffdb7-xkdkc:~/app$ openssl x509 -in $NODE_EXTRA_CA_CERTS -subject -nocert
subject=C = US, ST = Colorado, L = Boulder, O = Drolrevo, CN = *.drolrevo.com

[weishiuchang]

To verify it is the self-signed cert, I ran eas with NODE_TLS_REJECT_UNAUTHORIZED=0 and it works flawlessly (much kudos to you, @travisghansen, I finally got a working oauth proxy that works with dex/traefik/kubernetes-dashboard), but while that's fine as a tech demo, it's not safe.

[travisghansen]

@weishiuchang thanks for the interest in the project and kind words! I'm not an expert on what uses that setting honestly. Are you using oidc or oauth2?

Is there any chance you need a chain of certs?

[weishiuchang]

@travisghansen I am using oidc

plugins: [
  {
    discover_url: "https://dex.auth-system.svc.cluster.local/.well-known/openid-configuration",
    ...
  },
]

I'm not sure if it needs to be a chain of certs, the self-signed cert is a pretty standard one generated from openssl req -newkey rsa:2048 -config /tmp/san.conf -extensions v3_req -nodes -keyout /data/drolrevo.com.key -x509 -days 3650 -out /data/drolrevo.com.crt

openssl x509 -in $NODE_EXTRA_CA_CERTS -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
86:3c:d0:12:6f:7a:96:f4
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, ST = Colorado, L = Boulder, O = Drolrevo, CN = *.drolrevo.com
Validity
Not Before: Apr 1 17:01:16 2020 GMT
Not After : Mar 30 17:01:16 2030 GMT
Subject: C = US, ST = Colorado, L = Boulder, O = Drolrevo, CN = *.drolrevo.com
...
X509v3 extensions:
X509v3 Key Usage:
Key Encipherment, Data Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:drolrevo.com, DNS:dex.auth-system.svc.cluster.local, DNS:dex.drolrevo.com

(drolrevo.com is a fake/example domain for testing)

I'm digging through nodejs documents to see how I can help troubleshoot this, but since I know almost nothing about nodejs this is a lot to dig through...

[travisghansen]

I'm not a pki expert, but I don't know if you can do what you're describing...or at least what I understand you're doing.

If I'm reading through the lines appropriately it appears you're using the exact same certificate to both serve up the website and as the CA file in eas. That's not really what you want.

You want to create your own CA and then issue a cert. The issued cert will be what you push to dex and the CA gets pushed to eas.

Something along the lines here: https://medium.com/@tbusser/creating-a-browser-trusted-self-signed-ssl-certificate-2709ce43fd15

rootCA.pem is what you'd give to eas and then the server.key/server.crt is what would get loaded into dex.

[weishiuchang]

Interesting. https://mattcbaker.com/posts/developing-https-node-local/ seems to imply that self-signed certs should work with NODE_EXTRA_CA_CERTS. I will try generating a self-signed CA and signing a csr with that and see if it addresses the issue.

[travisghansen]

Maybe you can..

I also just noticed the names don't match...unless you added SANs it will still fail. The loaded cert is for *.drolrevo.com but you're accessing the service as dex.auth-system.svc.cluster.local so even though you trust the 'CA'/crt validation will still fail.

[weishiuchang]

I have it listed under the SANs:

...
DNS:drolrevo.com, DNS:dex.auth-system.svc.cluster.local, DNS:dex.drolrevo.com
...

[weishiuchang]

I'm generating a self-signed CA now and will use it to sign a cert and try that. I will also try creating a non-wildcard cert as well with a cn that is explicitly dex.auth-system.svc.cluster.local and report back here with what I find.

[travisghansen]

Awesome thanks!

[weishiuchang]

Alright! Got eas working now pointing at a self-signed oidc endpoint (dex).

The culprit was in actually making a real CA with keyUsage=keyCertSign as it seems nodejs purposely ignores pems in NODE_EXTRA_CA_CERTS without that fairly crucial SSL extension. Which, in retrospect, is a face-palm moment for me, as that keyUsage is what makes a self-signed cert a real CA (all root CAs are self-signed certs, btw). It's even in the variable name.

This was through many, many variations of tests with eas deployment and kubernetes dashboard to get there. Appreciate the speedy response @travisghansen as it got me down the right path to get this working.

[weishiuchang]

A slight clarification on my closing comment -

I ended up just generating a single self-signed cert with keyUsage=keyCertSign, without having to create a root CA - signed server cert chain.

[travisghansen]

@weishiuchang wow thanks for digging down into that! That's a tricky one to find for sure :(

If you don't mind maybe share a little more detail here for others stumbling upon this in the future about the self-signed cert creation process.

I'd also be interested in details of how you tied it into kubernetes-dashboard so if you write a blog entry or similar send it my way :)

[weishiuchang]

@travisghansen I will do. I'm ironing out a few things then I will do a write up on the entire setup. Right now I'm struggling with trying to run kubernetes dashboard behind traefik http reverse proxy without using http://localhost (had to put a default ingress directly to my dashboard to make that happen), as it seems kubernetes dashboard specifically prevents that in 1.7 https://github.com/kubernetes/dashboard/blob/master/docs/user/accessing-dashboard/1.7.x-and-above.md

[weishiuchang]

https://github.com/weishiuchang/writeups/tree/master/onprem-kube-dashboard

[dreami2023]

I added the script below to /etc7profile.d.
But I keep getting the same error.
npm config set cafile /temp/ca/bundled.pem --global
yarn config set cafile /temp/ca/bundled.pem --global
echo "registry=http://registry.npmjs.org/" > ~/.npmrc
echo "cafile=/temp/ca/bundled.pem" >> ~/.npmrc
export NODE_EXTRA_CA_CERTS=/temp/ca/bundled.pem

[travisghansen]

Are you using the container to deploy or some other means?