Choosing The Right Mode For SIMCOM or Any Modem... (AT or DialUP) (IDFGH-12638)

[parthbhat13]

Choosing The Right Mode For SIMCOM or Any Modem... (AT or DialUP) (IDFGH-12638)

[david-cermak]

Hi @parthbhat13

I'm afraid I cannot give a general suggestion about the "right" mode, and I'd just repeat myself on what I've already said in that devcon talk: "depends on your application".
You've been talking about your customer's complaints on unstable connectivity, and also a bit about your application needs -- occasional HTTP request and MQTT communication. So I'd assume the number one priority of your project is the fast and fail save mechanism and recovery on signal loss while the communication is not so frequent and kind of slow. In this case, I'd agree that using the AT commands only is probably a better choice (just mind the disadvantages, like security and portability)

Unfortunately we don't have any example which deals with connectivity losses and recoveries, as it's usually device and application dependent. I'll think about it -- my general suggestion in this scenario would be the CMUX manual mode or the USB dual channel mode.
Here's a simple example which demonstrates these retries and recoveries in the PPPoS mode:

esp-protocols/examples/esp_netif/multiple_netifs/main/ppp_connect_esp_modem.c

Lines 54 to 93 in bd6e120

	// now let's keep retrying
	while (!ppp_info->stop_task) {
	vTaskDelay(pdMS_TO_TICKS(backoff_time));
	if (ppp_info->parent.connected) {
	backoff_time = 5000;
	continue;
	}
	// try if the modem got stuck in data mode
	ESP_LOGI(TAG, "Trying to Sync with modem");
	ret = esp_modem_sync(dce);
	if (ret != ESP_OK) {
	ESP_LOGI(TAG, "Switching to command mode");
	esp_modem_set_mode(dce, ESP_MODEM_MODE_COMMAND);
	ESP_LOGI(TAG, "Retry sync 3 times");
	for (int i = 0; i < 3; ++i) {
	ret = esp_modem_sync(dce);
	if (ret == ESP_OK) {
	break;
	}
	vTaskDelay(pdMS_TO_TICKS(1000));
	}
	if (ret != ESP_OK) {
	CONTINUE_LATER();
	}
	}
	ESP_LOGI(TAG, "Manual hang-up before reconnecting");
	ret = esp_modem_at(dce, "ATH", NULL, 2000);
	if (ret != ESP_OK) {
	CONTINUE_LATER();
	}
	ret = esp_modem_get_signal_quality(dce, &rssi, &ber);
	if (ret != ESP_OK) {
	CONTINUE_LATER();
	}
	ESP_LOGI(TAG, "Signal quality: rssi=%d, ber=%d", rssi, ber);
	ret = esp_modem_set_mode(dce, ESP_MODEM_MODE_DATA);
	if (ret != ESP_OK) {
	CONTINUE_LATER();
	}
	}

you can also check this file, which does the same thing directly, without esp_modem

esp-protocols/examples/esp_netif/multiple_netifs/main/ppp_connect_simple.c

Lines 71 to 95 in bd6e120

	while (retry < max_retries) {
	ESP_LOGD(TAG, "Sending command: %s", init_sequence[cmd_i].cmd);
	uart_write_bytes(UART_NUM_1, init_sequence[cmd_i].cmd, strlen(init_sequence[cmd_i].cmd));
	xQueueReceive(event_queue, &event, pdMS_TO_TICKS(pdMS_TO_TICKS(1000)));
	size_t len;
	uart_get_buffered_data_len(UART_NUM_1, &len);
	if (!len) {
	continue;
	}
	len = uart_read_bytes(UART_NUM_1, reply, BUF_SIZE, 0);
	ESP_LOGD(TAG, "Received: %.*s", len, reply);
	if (strstr(reply, init_sequence[cmd_i].expect) || init_sequence[cmd_i].allow_fail) {
	if (strstr(reply, CONNECTED)) { // are we connected already?
	break;
	}
	cmd_i++;
	continue;
	}
	++retry;
	vTaskDelay(pdMS_TO_TICKS(retry * 1000));
	}
	if (retry >= max_retries) {
	ESP_LOGE(TAG, "Failed to perform initial modem connection");
	vTaskDelete(NULL);
	}

[parthbhat13]

Hi @david-cermak,
Thanks a lot for your reply, I'd like to add few more to what you've mentioned.

1. Fail safe is important yes!
   While you mentioned having fail safe as important aspect, at the same time, it's not needed to have two communication protocols hence it was decided to choose only one protocol either Mqtt or http.

2. You mentioned to use cmux over usb.
   The biggest concern as of now which you can already see I've mentioned, I cannot change the hardware now, neither can I add esp32 S2 or S3. So USB goes out of question.
   In the Devcon you had even mentioned about using the RTS and CTS lines ? Do you think it's important for me for the Mqtt or HTTP And ota updates ?

3. Yes going vanilla AT command sounds good to me to that now you have mentioned. But what DTE driver code would you recommend ? To use the esp-modems C function which allows me to send at commands ? Or make my own? Which I already have made and shown the code above which doesn't really work that fluent at times. ?

Thanks in advance !

[david-cermak]

1. Add fail safe: Yes, it's easier to cover that in the AT mode only, since you'd just get an ERROR or DISCONNECTED reply to your commands, while in the data mode you'd have to take care of different phases of PPP state machine and some devices just exit this data mode.

2. You can also use CMUX over UART, no need for USB. As for the UART flowcontrol signals, there're useful and prevent from buffer overflows, but you can easily tune up your application without them -- set the appropriate baud rate, move the UART ISR to RAM, increase Rx buffer sizes -- and according to my tests, the OTA should work seamlessly.

3. The problem is that you would really have to implement everything on your own, including OTA, and communication.
   You can check this file https://github.com/espressif/esp-protocols/blob/master/examples/esp_netif/multiple_netifs/main/ppp_connect_simple.c which uses UART driver directly, but with queues and interrupts, so should be faster than your code.
   You can also use esp_modem with AT commands only, maybe just the DTE class and the command library. I'd suggest using C++ in this case, as it's easily extensible.

[parthbhat13]

thanks for the response mr.david.

so let me summarize the things i shall test.

1. No need to change the hardware to use the CTS and RTS lines
2. Increase the uart buffer (i already did that i guess) and move uart isr to ram
3. i can choose to go with esp_modem dte class for my communication or i will have to find ways to port the TinyGsm library way of communication.
4. i wonder what to do for OTA? use the TCP/IP custom code or get the Dialup?

what i can see from above list of things which i would do is that, using C++ is what i was avoiding at time. but i wonder how much would it change? since the esp's uart driver is written in C? only some string manipulation would end up being helpful if im using c++.

let me get back to you with all the results. as i can see i will have to do multiple tests. i even had decided to work with CMUX mode where we can simply use the Data as well as AT commands to check for the network stability.

honestly i cant yet come to any conclusion on what method to use. so if you have any more ideas please do share. thanks in advance.

[david-cermak]

just a few comments:

• It's much easier to use the dialup connection for OTA, as everything is ready for you to use (vs. implementing OTA using AT commands)
• suggested C++ only if you decide to use esp_modem's DTE layers, as they're written in C++ so you can leverage some of the generic methods and templates (in case of directly interfacing the uart, there's no need nor benefit in using either C or C++).
• note about connectivity: I'd also suggest looking into the actual issues with unstable connections and your customers problems which you mentioned earlier to understand why they were happening, i.e. to identify the reasons and root causes of these issues. You might as well re-work and re-implement everything using another mode, but end up receiving the same complaints again, since the GSM/LTE/NB connectivity in the field is simply unstable by nature...

[parthbhat13]

Thanks for the response, I'm gonna take 4-5 different production boards and test all the different scenarios.
I recently was with a professor of mine he was working on php and a commercial project which used a7672S and stm32, I was amazed to see how quick the http post request response was being received. Many pointed out to me that apart from the network drop in my product, there might be the server issues as well. So will even test it out and get back to you with results so it helps others as well.

One small help I'd ask for, can you point me to the specific function in esp-modem which waits for the AT response ?

I'd like to wait for gsm to get stable once I toggle the power pin, and wait to receive its own Startup at messages like

• PB Done
• EPS DONE
• SIM READY

And only then I'd send the At commands to the gsm. I saw there's a function to send at command and there's a timeout mentioned, but I'd wanna simply wait for a valid data to be received.

Thanks in advance.

[david-cermak]

You can either send a null command or use dce->command() API or even supply your own callback in got_line argument.

[parthbhat13]

hi,
thanks for the response, since i am not fluent with C++ there are issues i am facing trying to make things working. i was even stuck with initializing the modem where i did not wanna pass the apn and init the

esp_modem_new_dev

perhaps so far this is what i have done and ported some of the espmodem function to work with mine, below are the code snippets.

esp_err_t initGsmUartAndPower(gsmConfig_t *config)
{
    esp_err_t ret;

    // setup the esp modem stack and data 
    ESP_LOGI(TAG, "Setting Up GSM Modem");
    esp_modem_dte_config_t dteConfig = ESP_MODEM_DTE_DEFAULT_CONFIG();

    // Setup Uart Specific Configuration
    dteConfig.uart_config.tx_io_num = config->gsmUartConfig.txPin;
    dteConfig.uart_config.rx_io_num = config->gsmUartConfig.rxPin;
    dteConfig.uart_config.rts_io_num = UART_PIN_NO_CHANGE;
    dteConfig.uart_config.cts_io_num = UART_PIN_NO_CHANGE;

    dteConfig.uart_config.rx_buffer_size = config->gsmUartConfig.rxBufferSize;
    dteConfig.uart_config.tx_buffer_size = config->gsmUartConfig.txBufferSize;
    dteConfig.uart_config.event_queue_size = config->gsmModemTaskConfig.eventQueueSize;
    dteConfig.task_stack_size = config->gsmModemTaskConfig.taskSize;
    dteConfig.task_priority = config->gsmModemTaskConfig.taskPriority;
    dteConfig.dte_buffer_size = config->gsmUartConfig.dteBufferSize;

    // configure the dce 
    // here we can later call the command to set the APN in the library 
    // esp_err_t esp_modem_set_apn(esp_modem_dce_t *dce, const char *apn);
    dceConfig = ESP_MODEM_DCE_DEFAULT_CONFIG("airtelgprs.com");

    // Configure the PPP netif
    esp_netif_config_t netifPPPConfig = ESP_NETIF_DEFAULT_PPP();

    
    // Configure Netif Object
    if (espModemNetif == NULL)
    {
        espModemNetif = esp_netif_new(&netifPPPConfig);
        assert(espModemNetif);
    }

    if (dceConfig == NULL)
    {
        dce = esp_modem_new_dev(ESP_MODEM_DCE_SIM7600, &dteConfig, &dceConfig, espModemNetif);
        assert(dce);
    }


    // setup Power
    ret = gpio_set_direction(config->gsmUartConfig.powerPin, GPIO_MODE_OUTPUT);
    ret = gpio_set_pull_mode(config->gsmUartConfig.powerPin, GPIO_PULLUP_DISABLE);
    ret = gpio_set_pull_mode(config->gsmUartConfig.powerPin, GPIO_PULLDOWN_DISABLE);
    gpio_set_level(config->gsmUartConfig.powerPin, 0);

    return ret;
}

and even the wrapper function which i have called in all the other files, this is what i have done

gsm_err_t DTESendCommandGetResponse(gsmConfig_t *config, char *atCommand, char *atResponse, char *atResponse2, int timeout)
{
    // TODO: well try to keep on resending the data maybe?
    gsm_err_t gsmErr;
    esp_err_t espErr;
    
    char bufData[config->gsmUartConfig.rxBufferSize];
    memset(bufData, '\0', sizeof(bufData));
    
    espErr = esp_modem_at(dce, atCommand, &bufData[0], timeout);

    if (espErr == ESP_OK)
    {

        // we can log the data here
        if (config->debugDTE)
        {
            ESP_LOGI(TAG, "Response: %s", bufData);
            ESP_LOG_BUFFER_HEX(TAG, bufData, strlen(bufData) + 1);
        }
        

        if (atResponse != NULL && bufData != NULL)
        {
            
            if (strstr(bufData, atResponse) != NULL)
            {
                if (atResponse2 != NULL)
                {
                    if (strstr(bufData, atResponse2) == 0)
                        return GSM_OK;
                    else
                        return GSM_ERROR;
                }
                return GSM_OK;
            }
            //TODO: Check if this needs to be fixed
            // this is basically to check if we got an error from the other function 
            // for example during pdp context if we got a known valid response or ERROR, then we should return erro r
            else if (atResponse2 != NULL && bufData != NULL)
            {
                if (strstr(bufData, atResponse2) != NULL)
                {
                    // again ideally we would reach to the else loop but i am confused with this part.
                    return GSM_ERROR;
                }
            }
            else
            {
                // ideally we should send this back to the URC
                if (strstr(bufData, "ERROR") != NULL)
                {
                    if (config->debugDTE)
                        ESP_LOGE(TAG, "Got Error URC Should Be called");
                    return GSM_ERROR;
                }
            }
        }
        else
        {
            ESP_LOGE(TAG, "Data did not receive correctly");
            return GSM_ERROR;
        }
    }
    else if (espErr == ESP_FAIL)
    {
        return GSM_ERROR;
    }
    else if (espErr == ESP_ERR_TIMEOUT)
    {
        return GSM_TIMEOUT;
    }
    else 
    {
        return GSM_ERROR;
    }

    return GSM_ERROR;
}

apologies if my code and concepts are too noob. trying to get my head around things after spending 10months on this project.

[parthbhat13]

You can either send a null command or use dce->command() API or even supply your own callback in got_line argument.

Hi David,

Can you please point me to more right direction on how can I possibly implement it?
I'm too much confused with how can I use those above mentioned functions.

[parthbhat13]

Hi,
Quick update, One of the problem which was mentioned in terms of the HTTP request time used to be 3Seconds, Figured out that if we use AT commands and use the HTTP Stack of the gsm and make HTTPS request it is around 3~4Seconds.
If we use the HTTP url instead of the HTTPS, then we end up much more faster and around 300mS of the time.
Same with the DialUp with HTTPS its 1-2Seconds and with HTTP its 100-200mS. so one thing is clear over here that it is better to use HTTP rather than HTTPS.

what i am planning now is to add a feature and modem of A7672S like how Mr.Franz have added for the gnss, and push this on the esp_modem. i went through all the old issues trying to figure out how we can add custom modem and add more features like using the internal HTTP stack of the gsm using AT commands. i could not find a good documentation for the same, would simply keep going through the issues and see how others have added features. if there is any other better documentation which you can walk me through it will be really helpful.

will keep updating over here so others might as well get better idea on how and what stack to use when, specially in the indian market.

thanks again.