Release stuck in CannotUpdateExternalResource

[crossplane-contrib]

Release stuck in CannotUpdateExternalResource

[turkenh]

Seems to be related to the case described here: #36 (comment)

Enabling rollback could help to recover from such cases: #36 (comment)

[turkenh]

An actual fix would probably implementing something like this in helm provider controller: helm/helm#7139 (comment)

[turkenh]

@muvaf reported same issue and we don't see any pod restarts in helm provider. helm deployment probably cancelled due to context deadline in this case.

[turkenh]

In summary;

If chart contains hooks, helm deployment takes longer since helm go client waits (blocking) until hooks completed (if pre), or all other resources completed (if post), which could become more severe because of things like:

• cluster nodes not being ready (composed k8s cluster could report ready before all nodes are running, which could be the case here)
• slow network (pulling pod images)
• hook job itself taking longer

Actual helm release marked as pending-install and there is no way to continue installation or do an upgrade on top of that release, if, during deployment, deployment process cancelled somehow (similar to hitting ctrl-c in helm client):

• controller pod being restarted/killed
• context deadline exceeded

A short term solution could be increasing the context deadline which would block reconciler loop longer. This is not a perfect solution for a k8s controller, but since helm provider was already configured to run 10 workers by default, it wouldn't hurt much.

Actual fix should be to find a way to do the deployment as asynchronous instead of blocking with helm go client.

Even with a fix, I believe rollback should be enabled in helm release (esp. if there are hooks in the chart) by setting spec.rollbackLimit like this.

[turkenh]

Having short term solution merged, closing this issue in favor of #63

Please reopen, if you observe again.