Successful install but pods crashing.

[sassoftware]

Successful install but pods crashing.

[henmo09]

here is an output
root@henmo:/project/viya4-deployment/roles/baseline/defaults# kubectl get pods -n viya4-frg-v1 |grep sas-workflow-manager-app
sas-workflow-manager-app-6bbd577b57-ccrxq 0/1 Init:CrashLoopBackOff 74 5h54m
root@henmo:/project/viya4-deployment/roles/baseline/defaults# kubectl describe pods -n viya4-frg-v1 |grep sas-workflow-manager-app
Name: sas-workflow-manager-app-6bbd577b57-ccrxq
Labels: app=sas-workflow-manager-app
app.kubernetes.io/name=sas-workflow-manager-app
sas.com/component-name: sas-workflow-manager-app
Controlled By: ReplicaSet/sas-workflow-manager-app-6bbd577b57
KUBE_POD_NAME: sas-workflow-manager-app-6bbd577b57-ccrxq (v1:metadata.name)
sas-workflow-manager-app:
Image: cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-workflow-manager-app:2.9.0-20200929.1601422577467
Liveness: exec [/opt/sas/viya/home/bin/sas-liveness-tool -manifest-file /opt/sas/viya/home/share/sas-workflow-manager-app/run-manifest.json] delay=0s timeout=1s period=10s #success=1 #failure=3
SAS_K8S_DEPLOYMENT_NAME: sas-workflow-manager-app
root@henmo:/project/viya4-deployment/roles/baseline/defaults# kubectl get pods -n viya4-frg-v1 |grep sas-workflow-manager-app
sas-workflow-manager-app-6bbd577b57-ccrxq 0/1 Init:CrashLoopBackOff 74 5h54m
root@henmo:/project/viya4-deployment/roles/baseline/defaults#

[henmo09]

s

[henmo09]

/project/viya4-deployment/roles/baseline/defaults# kubectl get pods -n viya4-frg-v1 |grep sas-workflow-manager-app
sas-workflow-manager-app-6bbd577b57-ccrxq 0/1 Init:CrashLoopBackOff 74 5h54m

[thpang]

Have you looked at the logs of the pods to see why they have not started? I would start with consul, then look at the cache pods, then logon. Good places to start.

[henmo09]

these are one of the logs, but not much info.
4-frg-v1 sas-workflow-manager-app-6bbd577b57-ccrxq 0/1 Init:CrashLoopBackOff 231 19h
root@henmo:# kubectl logs sas-analytics-components-65cf89944f-s7fb6 -n viya4-frg-v1
Error from server (BadRequest): container "sas-analytics-components" in pod "sas-analytics-components-65cf89944f-s7fb6" is waiting to start: PodInitializing
root@henmo:# kubectl logs sas-analytics-events-5575d8c5cc-g7nsl -n viya4-frg-v1
Error from server (BadRequest): container "sas-analytics-events" in pod "sas-analytics-events-5575d8c5cc-g7nsl" is waiting to start: PodInitializing
root@henmo:# kubectl logs sas-backup-purge-job-1628727300-65st8 -n viya4-frg-v1
error: a container name must be specified for pod sas-backup-purge-job-1628727300-65st8, choose one of: [sas-backup-purge-job sas-backup-agent] or one of the init containers: [sas-certframe]
root@henmo:#

[henmo09]

see logs for logon and pod describe.

root@henmo:# kubectl logs sas-logon-app-764795d975-vf7ws -n viya4-frg-v1
Error from server (BadRequest): container "sas-logon-app" in pod "sas-logon-app-764795d975-vf7ws" is waiting to start: PodInitializing
root@henmo:# kubectl describe pod sas-logon-app-764795d975-vf7ws -n viya4-frg-v1
Name: sas-logon-app-764795d975-vf7ws
Namespace: viya4-frg-v1
Priority: 0
Node: aks-stateless-38049675-vmss000000/192.168.0.5
Start Time: Wed, 11 Aug 2021 18:47:06 +0000
Labels: app=sas-logon-app
app.kubernetes.io/name=sas-logon-app
pod-template-hash=764795d975
sas.com/deployment=sas-viya
workload.sas.com/class=stateless
Annotations: prometheus.io/path: /internal/metrics
prometheus.io/port: 10445
prometheus.io/scheme: https
prometheus.io/scrape: true
sas.com/certificate-file-format: jks
sas.com/component-name: sas-logon-app
sas.com/component-version: 2.39.9-20201002.1601645502383
sas.com/kustomize-base: spring
sas.com/pod-uses-readiness-probe: http
sas.com/tls-enabled-ports: all
sas.com/version: 2.39.9
sidecar.istio.io/proxyCPU: 15m
sidecar.istio.io/proxyMemory: 115Mi
Status: Pending
IP: 10.244.2.178
IPs:
IP: 10.244.2.178
Controlled By: ReplicaSet/sas-logon-app-764795d975
Init Containers:
sas-certframe:
Container ID: containerd://72b7afc28d8d20212a463c58b36afb0987db0173ae4b416fe93ac7695fccf4a5
Image: cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe:3.7.2-20201208.1607438373197
Image ID: cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe@sha256:2ce3072fb2693e8b89301a2bc314dcc68fc1c8fa92901e3568202df9ef995030
Port:
Host Port:
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 12 Aug 2021 14:14:00 +0000
Finished: Thu, 12 Aug 2021 14:14:00 +0000
Ready: False
Restart Count: 233
Limits:
cpu: 500m
memory: 500Mi
Requests:
cpu: 50m
memory: 50Mi
Environment Variables from:
sas-certframe-ingress-certificate-config-cmm2t44t88 ConfigMap Optional: false
sas-certframe-user-config-kfckf2c4g8 ConfigMap Optional: false
sas-certframe-java-config-gf4kt94898 ConfigMap Optional: false
Environment:
KUBE_POD_NAME: sas-logon-app-764795d975-vf7ws (v1:metadata.name)
SAS_CERTFRAME_TOKEN_DIR: /certframe-token
SAS_ADDITIONAL_CA_CERTIFICATES_DIR: /customer-provided-ca-certificates
Mounts:
/certframe-token from certframe-token (rw)
/customer-provided-ca-certificates from customer-provided-ca-certificates (rw)
/security from security (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-49tpb (ro)
Containers:
sas-logon-app:
Container ID:
Image: cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-logon-app:2.39.9-20201002.1601645502383
Image ID:
Ports: 8080/TCP, 10445/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 50m
memory: 763Mi
Readiness: http-get https://:http-internal/internal/ready delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
sas-java-config-t7t5thbgkd ConfigMap Optional: false
sas-shared-config-5m2f4bdg2c ConfigMap Optional: false
sas-consul-client Secret Optional: false
sas-spring-management-config-d64bbkhb97 ConfigMap Optional: false
sas-java-tls-config-fch4777298 ConfigMap Optional: false
Environment:
SAS_K8S_DEPLOYMENT_NAME: sas-logon-app
Mounts:
/opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts from security (rw,path="cacerts")
/opt/sas/viya/config/etc/SASSecurityCertificateFramework/private from security (rw,path="private")
/opt/sas/viya/config/tmp from tmp (rw)
/security from security (rw)
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-49tpb (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
certframe-token:
Type: Secret (a volume populated by a Secret)
SecretName: sas-certframe-token
Optional: false
security:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
customer-provided-ca-certificates:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sas-customer-provided-ca-certificates-29kdmk686c
Optional: false
tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
default-token-49tpb:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-49tpb
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
workload.sas.com/class=stateful:NoSchedule
workload.sas.com/class=stateless:NoSchedule
Events:
Type Reason Age From Message

---

Normal Pulled 43m (x222 over 19h) kubelet Container image "cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe:3.7.2-20201208.1607438373197" already present on machine
Warning BackOff 3m31s (x5327 over 19h) kubelet Back-off restarting failed container

[thpang]

At this point you'll have to figure out the best way for you to debug your environment. If the deployment ran without failure, there is more than likely a configuration issue you need to investigate. As I stated above, looking at the logs for your pods is going to be key. If you need a tool to help, use Lens. It's a great UI for helping one navigate their kubernetes cluster. If you have some specific information about the deployment you feel is causing the issue, open another issue with information there about the problem.