ccdb-migrate job should not wait for Istio

[cloudfoundry]

ccdb-migrate job should not wait for Istio

[cwlbraa]

Maybe there is a way we can have the migrator task check if the istio-proxy sidecar container exists and iff it exists wait for it to become healthy.

Alternatively, it could retry-until-short-timeout to log in to the database, regardless of what errors it sees when it's trying to connect. That behavior would be correct and istio-agnostic.

[matt-royal]

I have a branch that replaces the istio-specific logic with generic logic that waits for a DB connection to be available before running the migrations. However, it looks like we still need to explicitly kill the istio sidecar as part of the job, as the istio-proxy is still running after the run-migrations container completed. Is this expected?

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175367511

The labels on this github issue will be updated when the story is started.

[tcdowney]

@mike1808 based on the concerns on the Istio issue around this using undocumented aspects of the Kubernetes Pod start-up process are you confident that Istio will be able to maintain this behavior?

I always considered their "fix" here to be less of a fix and more of a best-effort mitigation. I'm worried that across Istio upgrades (y'all test this so it's probably less risky) and Kubernetes versions this behavior might change and we'll have to add the check back in.

Maybe there is a way we can have the migrator task check if the istio-proxy sidecar container exists and iff it exists wait for it to become healthy.

[tcdowney]

Yeah, that's a better idea!

[mike1808]

Currently, this issues blocks @cloudfoundry/cf-for-k8s-networking team from merging Contour work to develop in cf-for-k8s.

[matt-royal]

Maybe there is a way we can have the migrator task check if the istio-proxy sidecar container exists and iff it exists wait for it to become healthy.

Alternatively, it could retry-until-short-timeout to log in to the database, regardless of what errors it sees when it's trying to connect. That behavior would be correct and istio-agnostic.

One simple way to do this: make a new rake task that spins until it can connect to the DB (and may eventually time out). The ccdb migrate job can then run rake spin-until-db-connects EXISTING_RAKE_TASKS

[tcdowney]

Yeah, unfortunately for Jobs you have to explicitly kill it with that /quitquitquit endpoint. The hope is that this can be confused if/when Kubernetes ever gets something like first-class Sidecar containers (kubernetes/enhancements#753), but right now it's kinda rough.

This Istio issue talks more about it:
istio/istio#11659

[tcdowney]

This is fixed now on the latest cf-for-k8s main. See:

• cloudfoundry/cloud_controller_ng@d7a5fe3#diff-8e48e5cccab8a9fc9d245274271c1c1f0065248b0cd44539a08dc6aeca0f39a1
• cloudfoundry/cloud_controller_ng@56ee90b#diff-8e48e5cccab8a9fc9d245274271c1c1f0065248b0cd44539a08dc6aeca0f39a1
• cloudfoundry/cloud_controller_ng@b8aa9c4#diff-8e48e5cccab8a9fc9d245274271c1c1f0065248b0cd44539a08dc6aeca0f39a1

Thanks for reporting this @mike1808 ! 🙂