[Newrelic-bundle] <unable to upgrade or downgrade helm chart version getting field is immutable> when pixie is enabled with nri-bundle

[jvadera]

[Newrelic-bundle] when pixie is enabled with nri-bundle

from helm-charts.

[workato-integration]

https://issues.newrelic.com/browse/NEWRELIC-9463

[vanvanis]

We have started getting the below exceptions during the Rebuild of NewRElic Helm Chart inside our EKS Cluster in PROD environment:-

2023/06/23 01:22:43 executing task helminstallexternal...
2023/06/23 01:22:43 helm repo add newrelic https://helm-charts.newrelic.com && helm repo update
2023/06/23 01:22:44 helm upgrade esg-newrelic newrelic/nri-bundle --install --atomic --wait --wait-for-jobs --history-max 5 --timeout 20m -f /codebuild/output/src2249/src/s3/00/.helm/eks/prod/newrelic.yaml -n newrelic
2023/06/23 01:23:02 "newrelic" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "gatekeeper" chart repository
...Successfully got an update from the "splunk" chart repository
...Successfully got an update from the "newrelic" chart repository
...Successfully got an update from the "external-secrets" chart repository
...Successfully got an update from the "projectcalico" chart repository
Update Complete. ⎈Happy Helming!⎈

Error: UPGRADE FAILED: an error occurred while rolling back the release. original upgrade error: cannot patch "esg-newrelic-newrelic-pixie" with kind Job: Job.batch "esg-newrelic-newrelic-pixie" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app.kubernetes.io/name":"newrelic-pixie", "controller-uid":"d683a034-8d67-4cd9-95bd-5d61cb574ce6", "job-name":"esg-newrelic-newrelic-pixie", "release":"esg-newrelic"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container{core.Container{Name:"cluster-registration-wait", Image:"gcr.io/pixie-oss/pixie-dev-public/curl:1.0", Command:[]string{"sh", "-c", "set -x; URL="https://${SERVICE_NAME}:${SERVICE_PORT}/readyz"; until [ $(curl -m 0.5 -s -o /dev/null -w "%!{(MISSING)http_code}" -k ${URL}) -eq 200 ]; do echo "Waiting for cluster registration. If this takes too long check the vizier-cloud-connector logs." sleep 2; done; "}, Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"SERVICE_NAME", Value:"vizier-cloud-connector-svc", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"SERVICE_PORT", Value:"50800", ValueFrom:(*core.EnvVarSource)(nil)}},

Please advise the Fix for the same.

[DemonICOLI]

I can confirm that this error is also present on my deployment with helm

Error: UPGRADE FAILED: release newrelic-bundle failed, and has been rolled back due to atomic being set: cannot patch "newrelic-bundle-newrelic-pixie" with kind Job: Job.batch "newrelic-bundle-newrelic-pixie" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app.kubernetes.io/name":"newrelic-pixie", "batch.kubernetes.io/controller-uid":"806b2d5e-4e5e-4a06-a2f5-277a2bc4ddc9", "batch.kubernetes.io/job-name":"newrelic-bundle-newrelic-pixie", "controller-uid":"806b2d5e-4e5e-4a06-a2f5-277a2bc4ddc9", "job-name":"newrelic-bundle-newrelic-pixie", "release":"newrelic-bundle"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container{core.Container{Name:"cluster-registration-wait", Image:"gcr.io/pixie-oss/pixie-dev-public/curl:1.0", Command:[]string{"sh", "-c", "set -x; URL=\"https://${SERVICE_NAME}:${SERVICE_PORT}/readyz\"; until [ $(curl -m 0.5 -s -o /dev/null -w \"%!{(MISSING)http_code}\" -k ${URL}) -eq 200 ]; do echo \"Waiting for cluster registration. If this takes too long check the vizier-cloud-connector logs.\" sleep 2; done; "}, Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"SERVICE_NAME", Value:"vizier-cloud-connector-svc", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"SERVICE_PORT", Value:"50800", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, Containers:[]core.Container{core.Container{Name:"newrelic-pixie", Image:"newrelic/newrelic-pixie-integration:2.1.4", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"CLUSTER_NAME", Value:"fl-satellite-dev", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"NR_LICENSE_KEY", Value:"", ValueFrom:(*core.EnvVarSource)(0xc01aa82800)}, core.EnvVar{Name:"PIXIE_API_KEY", Value:"", ValueFrom:(*core.EnvVarSource)(0xc01aa82820)}, core.EnvVar{Name:"PIXIE_CLUSTER_ID", Value:"", ValueFrom:(*core.EnvVarSource)(0xc01aa82840)}, core.EnvVar{Name:"COLLECT_INTERVAL_SEC", Value:"15", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"HTTP_SPAN_LIMIT", Value:"750", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DB_SPAN_LIMIT", Value:"250", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList{"memory":resource.Quantity{i:resource.int64Amount{value:250, scale:6}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"250M", Format:"DecimalSI"}}, Requests:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:250, scale:6}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"250M", Format:"DecimalSI"}}, Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc0133f8858), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc018ff1e60), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable

[isaacadeleke-nr]

Hey @jvadera would you mind sharing your values.yaml file (with secrets expunged of course). I am trying to reproduce the error, but haven't been successful in doing so yet.

Also is this understanding correct: you are trying to upgrade from chart version 5.0.7 to 5.0.20 ?

[davidgit]

Hi ,

I would suggest to visit New Relic Support, where you can engage the New Relic Support Community or open a support ticket depending on your support level. The support team is best positioned to assist with your specific needs.

Please provide a link to this GitHub issue when submitting your community post or support ticket.

Thanks!

[techdragon]

@davidgit this is an issue with the helm chart? Why would we want to engage the support community or open a ticket instead of tracking the issue here? Can this please be reopened as I also have this same error. So that's 4 separate corroborating people having the exact same issue with the pixie enabled chart.