Summary
WyDOT reported that during a heavy file upload period, the ADM container shut down. We believe that either Zookeeper or Kafka got stuck for approximately 4 minutes, during which it did not respond to producer or consumer requests. This lack of response then caused the ADM to shut down. Since 4 minutes later Kafka/Zookeeper resumed working, it seems that it would be better for the ADM to pause rather than shut down.
We are still working to determine the cause and possible solutions for why Kafka went down. Since there were no logs for 4 minutes it seems that either the broker crashed and automatically restarted or there was some sort of network outage. The ODE reported network connectivity errors at the same time, reinforcing the network outage theory.
Debugging
The log.error file contains many entries with this message:
[180110 21:12:52.425121] [error] run(): Failure of XER encoding: Local: Queue full
This seems to come the switch statement in this file. There is a timeout condition:
case RdKafka::ERR__TIMED_OUT:
ilogger->info("{}: Waiting for more BSMs from the ODE producer.", fnname );
But Kafka can throw two types of timeout errors:
RD_KAFKA_RESP_ERR__MSG_TIMED_OUT Produced message timed out
RD_KAFKA_RESP_ERR__TIMED_OUT Operation timed out
So I believe that Kafka shut down and the producer threw the ERR__MSG_TIMED_OUT
after failing to get a response, which went into the "default" case of the above switch statement, which sets "data_available" to false and ends the processing run.