Yum update -y causing disconnects on CentOS 7

[hashicorp]

Yum update -y causing disconnects on CentOS 7

[jasonmcintosh]

Can work on this as a later update. We have a very custom packer json file so not a huge deal at this point. We can either close the issue or leave it for documentation/future enhancement.

COULD be the specific version of CentOS or something else. We do a CIS Hardened image with quite a few customizations to really lock down our OS's... could be something in there (wouldn't figure but who knows).

[brikis98]

I'm a bit confused. install-consul should only be run as part of the base install: e.g., in a Packer template that creates the AMI.

Are you using the script at runtime?

[jasonmcintosh]

It's run as part of the base install - NOT at runtime. BUT because yum update sometimes restarts services (e.g. systemd/etc), it disconnects packer terminating the script BEFORE it gets to installing Consul and similar. As such, such updates should NOT be part of an application install but part of standard builds for AMI's - doing a yum update -y is out of scope of what consul should be doing as part of it's install.

[brikis98]

Not sure I agree. It seems important to call update to ensure we get the latest version of, for example, the AWS CLI. If yum install works, I don't see any reason yum update would fail...

[jasonmcintosh]

yum update itself doesn't fail. However, yum update can restart SSHD. Which disconnects packer. Which causes the REST of the install to fail. Even with expects_disconnect set to true, packer sees this, then immediately says "Oh I'm done" and stops the instance and creates an AMI. This results in an AMI without consul installed. We finally traced it back to the yum update causing this as a patch updated some configuration (e.g. selinux)

[brikis98]

Oh, interesting. I've never seen yum update force sshd to restart, but I suppose it's possible. Perhaps we could add a flag to make the update step optional? Or is there some flag to skip service restart?

[jasonmcintosh]

Our fix was to move it to the consul.json packer script. IF you do this as a one step, then next step is the install. That way you can add "expect_disconnect" flag, as well as put a delay for any other updates to finish processing. Note, not an array item but a separate installer block. If you want, we can do a PR For that :)

[brikis98]

Hm, that's quite a nasty workaround just to run yum update! We use yum update often and have never seen this. This must be unique to CentOS / SELinux, which we haven't officially supported/tested with these modules yet.

If there's no other way from preventing yum update from restarting services, then your workaround sounds like the best bet. That said, I'm a bit concerned that this breaks backwards compatibility, so a flag to disable update in the script itself may be a safer route.