VGM complaining about task running too long

[nemosupremo]

VGM complaining about task running too long

[sdwr98]

Update: The "task running too long" error message in the VGM error log is incorrect (maybe a cut-and-paste issue) but the real error is returned in the JSON body of the /token request. We're seeing this:

Response from VGM: {"status":"Unsealed","ok":false,"error":"400: permission denied"}

And in the Vault logs a second or two before, this:

2016/10/20 21:05:07 [INFO] expire: revoked 'auth/token/create/7e416e9759a9a353685eccf099b9b33375bcd4fe'

EDIT:

One more thing. We're running two VGM instances for HA. One of them (the one that is handing out tokens successfully) has this in the logs every hour:

2016/10/20 20:44:54 Renewing token with ttl of 1h0m0s.
2016/10/20 20:44:54 Renewed token with ttl of 1h0m0s.

But the one that is returning errors has token renewal for the first 2 hours of running, but then stops - and exactly one hour after the last token renewal, the errors start showing up.

[nemosupremo]

My initial guess is the token you provided to vault doesn't have the right permissions in order to create tokens. Are you providing the same token to both instances? I'm not 100% sure, but it could be that a renewal on one machine is invalidating the token on another. (Sorry for the slow response)

[sdwr98]

We've been running it in our environment with just one VGM instance (to rule out the very scenario that you mentioned). Just a few minutes ago, I saw this error in the VGM log:

2016/10/25 19:57:39 Failed to lookup token. Not starting renewal watcher. Error: Get https://vault-sandbox.XXX.com/v1/auth/token/lookup-self: dial tcp: i/o timeout 

I haven't seen that every time, but I'm wondering if there's some sort of issue with our networking environment.

Having said that, do you think it would be worth sealing the VGM instance in a case like this? Since not restarting the renewal watcher will guarantee an unusable instance.

[nemosupremo]

That would make sense. If I have this right

• We try to renew a token
• It fails due to a network error
• We miss the window for renewing the token
• We should seal the instance here as the token could not be renewed.

[sdwr98]

That would be great.