Instance with no outbound connectivity seems to suffer from startup delay

[aws]

Instance with no outbound connectivity seems to suffer from startup delay

[LordAlfredo]

Thank you for bringing this to our attention. What's happening is the eic_harvest_hostkeys script is bound to a systemd unit set to run before sshd (basically as soon as hostkey generation/configuration is finished) and, as part of the script, make a call to ec2-instance-connect.[region].amazonaws.com. Of course, if the security group is blocking all outbound traffic then it wouldn't be able to reach that (even if it is an Amazon endpoint), fail, and delay systemd starting sshd.

There's already some safety checks to fast-fail if we detect we're not running on an EC2 instance. The fix here will be to add an additional check to this flow for if the service is reachable. The side effect is that the service will not know your instance's host key, as a result the first time you connect with the EC2 Console's Connect dialog there will be a small additional delay.

[dorki]

thanks for the quick reply.
By "EC2 Console's Connect" you are referring to the java ssh client ?
and any way, I cant think of a reason why not to fire the sshd right away (before eic_harvest_hostkeys), and if the eic_harvest_hostkeys completed successfully restart the sshd. because as far as i know, restarting sshd will not kill any active connections since each one become child ssh server and can live without the master server.

[LordAlfredo]

Close - with the release of EC2 Instance Connect the EC2 Console now has the option to use EIC to open a JavaScript native in-browser terminal instead of the Java client.

The reason we had set this to run before sshd was to ensure hostkey harvest was complete before anyone might attempt to use this feature in the EC2 Console.

After further discussion we've decided a better option is to add/tighten timeouts around the call to the service itself - there are other ramifications to adding an ICMP-ping-based fast-fail.

[fernando-villalba]

I am having issues with this as well when using a private intranet. Is there any endpoint that can be configured to access this? Also, are there any endpoints? I can't find them.

[CptTZ]

I am having issues with this as well when using a private intranet. Is there any endpoint that can be configured to access this? Also, are there any endpoints? I can't find them.

Endpoint varies based on the region of the instance, e.g. in US West (Oregon), the endpoint would be https://ec2-instance-connect.us-west-2.amazonaws.com/. eic_harvest_hostkeys provided a way to get this endpoint programmatically, you can take that as a reference. Anyway, I believe you have to configure your private intranet in order to reach this endpoint.

We are currently working on a fix for such scenario to address the delay issue as mentioned above.

[fernando-villalba]

I meant a private vpc endpoint that could be setup for this. The endpoint "ec2-instance-connect" or anything like it does not seem to exist as an internal endpoint (at least not in eu-west-1).

[CptTZ]

@fernando-villalba Thanks for correcting me! I was think about domain whitelisting for VPC egress gateway.

We does not support PrivateLink endpoints at this time. I will put a feature request for supporting this in our backlog.

[ohitspaul]

This issue has been addressed a while ago, with the timeout added here: https://github.com/aws/aws-ec2-instance-connect-config/blob/master/src/bin/eic_harvest_hostkeys#L164-L166

I believe this was kept open to "track" PrivateLink endpoints. Re-opened #18 for further discussion around PrivateLink, as that issue is directly relevant.