crm crash during cert renewal

[gainsley]

crm crash during cert renewal

[gainsley]

The client was nil, which is intentionally done by lb.go:GetDedicatedRootLBClients():

if err != nil {
	log.SpanLog(ctx, log.DebugLevelInfra, "failed to get rootLB client for dedicated cluster", "key", clusterInst.Key, "error", err)
	// set client as nil and continue, caller will generate alert accordingly
	client = nil
}

It became null because an openstack command failed:

2024-05-30T07:48:40.638Z        INFO    6c13b42c2f461e03        openstack/openstack-cmd.go:41   OpenStack Command Start {"name": "openstack", "parms": "subnet show -f json 49359fba-f43a-4ada-a311-8411188486d1"}
2024-05-30T07:48:46.763Z        INFO    6c13b42c2f461e03        openstack/openstack-cmd.go:49   OpenStack Command Done  {"parmstr": "subnet show -f json 49359fba-f43a-4ada-a311-8411188486d1", "elapsed time": "6.124968207s"}
2024-05-30T07:48:46.763Z        INFO    6c13b42c2f461e03        openstack/openstack-server.go:53        unable to update server IPs     {"sd": {"Addresses":null,"Networks":null,"ID":"b3aee777-ae07-47d8-afa9-3bcd3c22125c","Name":"q-b-q-dev.b-m-t.e.app.tdg.edgexr.org","Status":"ACTIVE"}, "err": "failed to look up subnet 49359fba-f43a-4ada-a311-8411188486d1 for ip 80.187.128.219, can't unmarshal subnet detail, invalid character 'F' looking for beginning of value"}
2024-05-30T07:48:46.763Z        INFO    6c13b42c2f461e03        vmlayer/lb.go:830       failed to get rootLB client for VM app instance {"key": {"name":"q-b","organization":"q-Dev","cloudlet_key":{"organization":"t","name":"b-m"}}, "error": "unable to update server IPs -- failed to look up subnet 49359fba-f43a-4ada-a311-8411188486d1 for ip 80.187.128.219, can't unmarshal subnet detail, invalid character 'F' looking for beginning of value"}

Running the same openstack command locally worked fine, although one of my openstack commands did return an extra failure message:

openstack subnet list | grep 49359fba-f43a-4ada-a311-8411188486d1
Failed to discover available identity versions when contacting https://b.t.d:5000/v3. Attempting to parse version from URL.
| 49359fba-f43a-4ada-a311-8411188486d1 | external-subnet                                                                | 22a842f3-4a1a-4c55-8e0b-f7438d045a91 | 80.187.128.0/22      |

It still eventually succeeded. Note that the json parse error "invalid character 'F'" is probably trying to unmarshal a string that starts wtih "Failed to ...".

Unforunately the caller (in this case certifications.go:refreshCerts()) did not check for a nil client.