`start ecs` doesn't seem to start ecs-agent

[aws]

`start ecs` doesn't seem to start ecs-agent

[nmeyerhans]

Aside from a fixed sleep, another option would be to write a polling loop, retrying your curl until you get a meaningful response. I'd recommend a short sleep (probably 1 second is fine) between loop iterations and a fatal timeout of maybe 1 or two minutes so you avoid looping forever in case the agent never starts for some reason.

In theory we could implement such a loop in ecs-init itself or in the upstart config we use to start it, so I'll leave this issue open as a feature request. It's possible that there are issues with this idea that I haven't considered yet, so it may not be the right thing to do.

[nmeyerhans]

Hi @davidvuong. ecs-init and ecs-agent both log to files in /var/log/ecs/. It is possible that these will provide some insight into what's happening. My guess is that there's a dependency that hasn't yet finished initializing by the time you call start ecs or curl, but that's just a guess.

[davidvuong]

Hey @narehayrapetyan,

Here's the log for both files in /var/log/ecs/:

ecs-init.log:

2016-12-30T03:16:26Z [INFO] pre-start
2016-12-30T03:16:27Z [INFO] start
2016-12-30T03:16:27Z [INFO] No existing agent container to remove.
2016-12-30T03:16:27Z [INFO] Starting Amazon EC2 Container Service Agent

ecs-agent.log:

2016-12-30T03:16:29Z [INFO] Starting Agent: Amazon ECS Agent - v1.13.1 (efe53c6)
2016-12-30T03:16:29Z [INFO] Loading configuration
2016-12-30T03:16:29Z [INFO] Event stream ContainerChange start listening...
2016-12-30T03:16:29Z [INFO] Checkpointing is enabled. Attempting to load state
2016-12-30T03:16:29Z [INFO] Loading state! module="statemanager"
2016-12-30T03:16:29Z [INFO] Detected Docker versions [1.17 1.18 1.19 1.20 1.21 1.22 1.23]
2016-12-30T03:16:29Z [INFO] Registering Instance with ECS
2016-12-30T03:16:29Z [INFO] Registered! module="api client"
2016-12-30T03:16:29Z [INFO] Registration completed successfully. I am running as 'xxx' in cluster 'yyy'
2016-12-30T03:16:29Z [INFO] Saving state! module="statemanager"
2016-12-30T03:16:29Z [INFO] Beginning Polling for updates
2016-12-30T03:16:29Z [INFO] Initializing stats engine
2016-12-30T03:16:29Z [INFO] Event stream DeregisterContainerInstance start listening...
2016-12-30T03:16:29Z [INFO] Creating poll dialer, host: ecs-a-1.us-east-1.amazonaws.com
2016-12-30T03:16:29Z [INFO] Creating poll dialer, host: ecs-t-1.us-east-1.amazonaws.com
2016-12-30T03:16:39Z [INFO] Saving state! module="statemanager"

[davidvuong]

This might also be useful. Cloud init logs before and after the start ecs call:

echo "before"
docker ps -a

cat /var/log/ecs/ecs-init.log.2016-12-30-03
cat /var/log/ecs/ecs-agent.log.2016-12-30-03

echo ECS_CLUSTER=$ECS_CLUSTER >> /etc/ecs/ecs.config
start ecs

echo "after"
docker ps -a

cat /var/log/ecs/ecs-init.log.2016-12-30-03
cat /var/log/ecs/ecs-agent.log.2016-12-30-03

and this is what I get:

before
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
cat: /var/log/ecs/ecs-init.log.2016-12-30-03: No such file or directory
cat: /var/log/ecs/ecs-agent.log.2016-12-30-03: No such file or directory
ecs start/running, process 3029
after
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
2016-12-30T03:29:11Z [INFO] pre-start
2016-12-30T03:29:12Z [INFO] start
2016-12-30T03:29:12Z [INFO] No existing agent container to remove.
2016-12-30T03:29:12Z [INFO] Starting Amazon EC2 Container Service Agent
cat: /var/log/ecs/ecs-agent.log.2016-12-30-03: No such file or directory

[davidvuong]

@nmeyerhans Yep you're right. It seems like there's a timing issue. I forced a sleep 10 before the curl and everything works as it should.

Is there another way I can fix this aside from forcing an arbitrary sleep time?

[yumex93]

Based on my understanding, the problem here is "start ecs" just works as an interface to tell upstart init system to start agent, but starting agent is async. Adding loop in ecs-init or upstart config will not make "start ecs" command itself to wait for agent successfully started and then return. So the retry mentioned above would be the way to handle your case now.

Close the issue considering there is no action items on our side. Feel free to reopen it if you have any questions.