cloudfoundry-incubator/configgin

CLI app to generate configurations using BOSH ERB templates

configgin's Issues

Failed to fetch properties during pod restart

Background
api pod will fetch property in cc_uploader by BOSH links, like here. https://github.com/cloudfoundry/capi-release/blob/develop/jobs/cloud_controller_ng/templates/cloud_controller_ng.yml.erb#L360
(Just take api as an example, actually this issue also happened with other components.)

Description
This works well during the first deployment, but when we inject a sidecar container into the api pod, it will trigger a pod recreate, sometimes there's one error occurs like below

+ bash /opt/fissile/startup/scripts/forward_logfiles.sh
+ configgin --jobs /opt/fissile/job_config.json --env2conf /opt/fissile/env2conf.yml
Resolving link cc_uploader via service {"role"=>"cc-uploader", "job"=>"cc_uploader"}
/usr/local/rvm/gems/ruby-2.3.5/gems/bosh-template-2.0.0/lib/bosh/template/evaluation_link.rb:30:in `p': Can't find property '["internal_hostname"]' (Bosh::Template::UnknownProperty)
	from /var/vcap/jobs-src/cloud_controller_ng/templates/cloud_controller_api.yml.erb:445:in `block in get_binding'
	from /usr/local/rvm/gems/ruby-2.3.5/gems/bosh-template-2.0.0/lib/bosh/template/evaluation_context.rb:152:in `if_link'
	from /var/vcap/jobs-src/cloud_controller_ng/templates/cloud_controller_api.yml.erb:443:in `get_binding'
	from /usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/erb.rb:864:in `eval'
	from /usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/erb.rb:864:in `result'
	from /usr/local/rvm/gems/ruby-2.3.5/gems/configgin-0.14.1/lib/job.rb:57:in `generate'
	from /usr/local/rvm/gems/ruby-2.3.5/gems/configgin-0.14.1/bin/configgin:79:in `block (2 levels) in <top (required)>'
	from /usr/local/rvm/gems/ruby-2.3.5/gems/configgin-0.14.1/bin/configgin:78:in `each'
	from /usr/local/rvm/gems/ruby-2.3.5/gems/configgin-0.14.1/bin/configgin:78:in `block in <top (required)>'
	from /usr/local/rvm/gems/ruby-2.3.5/gems/configgin-0.14.1/bin/configgin:75:in `each'
	from /usr/local/rvm/gems/ruby-2.3.5/gems/configgin-0.14.1/bin/configgin:75:in `<top (required)>'
	from /usr/local/rvm/gems/ruby-2.3.5/bin/configgin:23:in `load'
	from /usr/local/rvm/gems/ruby-2.3.5/bin/configgin:23:in `<main>'
	from /usr/local/rvm/gems/ruby-2.3.5/bin/ruby_executable_hooks:15:in `eval'
	from /usr/local/rvm/gems/ruby-2.3.5/bin/ruby_executable_hooks:15:in `<main>'

Configgin breaks with PENDING status POD

Hello,

The following scenario broke my kube cluster during an upgrade.

Taking the example of a single component(e.g. diego-access) which is a kube deployment with 2 replicas, and have default values for maxSurge(1) and maxUnavailable(1).

During the upgrade kube will bring a new POD(due to the maxSurge) and take one down(maxUnavailable). However, due to the POD affinity and NodeSelector configurations, the new POD will not find nodes available to schedule himself, so it stays under a PENDING status, which will cause configgin to break.

Configgin code will try to get the following from the POD kubectl -n cf get SOMEPOD -o json | jq .status.containerStatuses(see get_pods_per_image ), but because is empty, it will break, making the POD that was supposed to be remove(due to maxUnavailable) to get a CrashLoopBackOff status.

I have a fork branch that will check first if the pod data.status.containerStatuses is empty, before trying to access it.

Can someone take a look on this, opinions?

Regards,
Enrique Encalada

fail to update bosh-link configuration on pod-0 when pod-1 is restarted

To enable sharding mechanism , we used bosh.link feature to get each node information and write the node's ip address and port into configuration.yml.

  sorted_instances=link("metricscollector").instances.sort_by {|i|i.address}
  nodeIndex=sorted_instances.index(sorted_instances.find{|i|i.id == spec.id})
  addrs=sorted_instances.map{|i| "'#{i.address}:#{p('autoscaler.metricscollector.server.port')}'"}
  nodeAddrs="[" + addrs.join(",") +"]"  

Then, the resulted configuration looks like:

  node_addrs: ['172.30.147.84:7103','172.30.206.99:7103']
  node_index: 0

The bosh-link render works good when creating multiple pod. But ... when one pod is restarted, it will gain a new IP address. In this situation, the other un-changed pod will get the configuration refresh, then the node_addrs in the configuration is wrong ...

See below example:

At the very beginning:

NAME                            READY STATUS      RESTARTS   AGE       IP               NODE
autoscaler-metrics-0   2/2       Running   0         4s        172.30.147.88   10.73.120.54
autoscaler-metrics-1   2/2       Running   0         12s       172.30.206.101   10.73.120.28

The configuration in autoscaler-metrics-0 is:

  node_addrs: ['172.30.147.88:7103','172.30.206.101:7103']
  node_index: 0

The configuration in autoscaler-metrics-1 is:

  node_addrs: ['172.30.147.88:7103','172.30.206.101:7103']
  node_index: 1

Up to now, everything is good.

Then, I deleted autoscaler-metrics-0 , and k8s created it again.
Now,
autoscaler-metrics-0 2/2 Running 0 4s 172.30.147.84 10.73.120.54

The configuration in autoscaler-metrics-0 is correct:

  node_addrs: ['172.30.147.84:7103','172.30.206.101:7103']
  node_index: 0

But the configuration in autoscaler-metrics-1 still is:

  node_addrs: ['172.30.147.88:7103','172.30.206.101:7103']
  node_index: 1

The ip address 172.30.147.88 is totally out-of-date now.

Is that possible to fix this link issue?