Unable to start kafka service

[jonathanmv]

Unable to start kafka service

[JuniusLuo]

What is the total memory of EC2 instance? What is the heap size configured for ZooKeeper? Could you please also get the detail logs of the management service from cloudwatch?

[jonathanmv]

I was using t2.small instances for the nodes, they have 2Gb of memory.
The heap size was 512Mb for both zookeeper and kafka.
I can't share the logs because I deleted the stack and all the logs as well

[jazzl0ver]

I had similar issues with small instances. Try t2.large.

[JuniusLuo]

Interesting. I also used t2.small in simple test sometimes, and did not see this problem. You could try t2.large. If you tried t2.small and hit this bug again, please help to collect the logs from cloudwatch.

[jonathanmv]

Ok, I recreated the cluster with t2.small instances and I am still getting the same problem. The kafka service is unable to start regardless of the heap size memory.

[JuniusLuo]

can you attach the logs of filecamp management service?

[jonathanmv]

firecamp-manageserver-logs.txt
Here are the logs of a new cluster created. This one has instances t2.medium and I was unable to create the zookeeper service.

In the logs you may find that I deleted the service and tried to create it again but to no avail.

How are you configuring the stacks? I haven't been able to use this cloudformation template unfortunately

[JuniusLuo]

looks zk container could not be started. this sounds weird. never met this before.

could you please get the firecamp volume log from one of the node? /var/log/firecamp/firecamp-dockervolume.*

also if you could attach one zk log, it would help for the debugging. The zk log is in cloudwatch.

[JuniusLuo]

how did you provision the system? are you using aws quickstart, or provision by yourself? Did you specify the correct availability zones and the corresponding subnets?

[jonathanmv]

Sorry guys, I'm unable to check the logs or spend more time looking into this issue.