azure-iot-arduino fails IoTHub connection after a couple hours

[azure]

azure-iot-arduino fails IoTHub connection after a couple hours

[GregTerrell]

Closing... this issue was rooted within the WiFi code. The AzureIotHub was able to cause the WiFi code to go into a tight retry loop after exhausting the send buffers within the WINC1500 chip. Under certain conditions, still unknown, the WINC1500 driver would stop attempting to service the chip and deadlock waiting for a send buffer.

A pending pull request https://github.com/arduino-libraries/WiFi101/pull/77 provides a better retry mechanism to the WINC no send buffer condition. My application can successfully run for, so far 24+ hours, with this WiFi101 code.

[GregTerrell]

After further testing using the 0.11.0 version of the WiFi101 client I have traced the lockup (solid activity LED) problem down to an infinite loop in SSL send function. I don’t see this as a direct defect in the azure-iot-arduino library, but the library’s actions are hitting a design constraint and\or defect with the WINC1500 code found inside WiFi101 (and the Adafruit WINC1500 library).

I haven’t been able to trace down any Atmel documentation, but it appears that the WINC1500 supports 16 outstanding transmits. The azure-iot-arduino library performs 18 sslClient_Writes() to complete an event send, this results in the TCP send window closing at the CRLF sent between the headers and the body; until outstanding sends are deemed complete via the send callback.

Typically the WiFi client waits appropriately for the send callbacks in order to finish sending the event POST. However, under some conditions the WINC deadlocks never processing send callbacks and consequently never returning to the task of sending out TCP frames. The WiFi101\WINC code in question is within the WINC hif_send() (in m2m_hif.c) and WiFiClient::write() (in WiFiClient.cpp) functions. This logic is waiting for a dma_addr (aka send buffer) from the WINC chip.

I have implemented a work around in our test environment, which consolidates the header and trailing CRLF writes inside SendHeadsToXIO() (in adapter\httpapi_compact.c) to a single send header + CRLF. By reducing the sends per event from 18 down to 11 the WINC code has now been running for over 75 hours, previous to the workaround never made it 8 hours.

[GregTerrell]

Our work around (below) initially solved the issue, however as I added back out user properties I discovered these are sent over the wire as additional HTTP headers, resulting in more than 16 frames to send an event and a reappearance of the issue.

Workaround...

//LOOUQ: compress header and trailing endline into one send()

if(((ret = snprintf(buf, sizeof(buf), "%s \r\n", header)) < 0) || (ret >= sizeof(buf)))
{
	/*Codes_SRS_HTTPAPI_COMPACT_21_027: [ If the HTTPAPI_ExecuteRequest cannot create a buffer to send the request, it shall not send any request and return HTTPAPI_STRING_PROCESSING_ERROR. ]*/
	result = HTTPAPI_STRING_PROCESSING_ERROR;
}
else 
{
	if (xio_send(http_instance->xio_handle, (const unsigned char*)buf, strlen(buf), NULL, NULL) != 0)
	{
	/*Codes_SRS_HTTPAPI_COMPACT_21_028: [ If the HTTPAPI_ExecuteRequest cannot send the request header, it shall return HTTPAPI_HTTP_HEADERS_FAILED. ]*/
		result = HTTPAPI_SEND_REQUEST_FAILED;
	}
}

// if (xio_send(http_instance->xio_handle, (const unsigned char*)header, strlen(header), NULL, NULL) != 0)
// {
// 	/*Codes_SRS_HTTPAPI_COMPACT_21_028: [ If the HTTPAPI_ExecuteRequest cannot send the request header, it shall return HTTPAPI_HTTP_HEADERS_FAILED. ]*/
// 	result = HTTPAPI_SEND_REQUEST_FAILED;
// }
// if (xio_send(http_instance->xio_handle, (const unsigned char*)"\r\n", (size_t)2, NULL, NULL) != 0)
// {
// 	/*Codes_SRS_HTTPAPI_COMPACT_21_028: [ If the HTTPAPI_ExecuteRequest cannot send the request header, it shall return HTTPAPI_HTTP_HEADERS_FAILED. ]*/
// 	result = HTTPAPI_SEND_REQUEST_FAILED;
// }