Collect: easier access to SMS-fallback workflow

[getodk]

Collect: easier access to SMS-fallback workflow

[lognaturel]

Thanks for sharing this requirement, @poVoq, it makes a lot of sense.

I see two good options for achieving what you want with existing functionality: 1) only make questions with SMS tags relevant if a certain condition is true 2) show an SMS send prompt for the enumerator.

I think the first option is your best bet. The idea is to have all your normal questions and then have calculates that are only relevant under a specific condition. I haven't tried this form but it would look something like:

type	name	compact_tag	label	calculation	relevant
text	phone_number		Patient phone number
decimal	temperature_f		Patient temperature in Fahrenheit
calculate	sms_phone_number	n		${phone_number}	${temperature_f} > 105
calculate	sms_temperature_f	t		${temperature_f}	${temperature_f} > 105

In that case, if temperature is below 105F, an SMS submission would not be generated. With the feature described in this specification, that means there would be no attempt to auto-send an SMS. On the other hand, if the recorded temperature was greater than 105F, an SMS would be generated and auto-sent if the settings are set to fall-back to SMS. I see that the case where an SMS is not generated is not explicitly called out by the spec and will add that.

Something else you might want to do before this auto-send is implemented is to include a question in the form that sends an SMS (see the documentation and a sample form). I would conditionally show the question if the specific indicator were set (same relevant idea as above).

The downside of this approach is that it does require the enumerator to manually tap "send" on the SMS. See this forum post for more information on why it's currently set up this way. What we might be able to do is get information back from the SMS app to know whether the SMS was successfully sent and only let the user advance in the form if it was. If this sounds interesting to you, please post on that thread so it can be considered. The downside is that if an enumerator truly is in an area with no cell reception, they won't be able to make progress.

As a side note, I see from JhulFramework/PHPServer4ODK#3 that you are interested in self-hosting a server. You may find ODK Central an interesting option. Please continue the conversation on the forum if interested.

[lognaturel]

@shobhitagarwal1612 I added some text below the "flow" diagram specifying that an SMS should only be sent once and I noted that this will require adding a DB column to keep track of the state. Thank you. I also added a section at the very end about alternatives considered. If I misunderstood your suggestion, please let me know!

[lognaturel]

Thank you @alxndrsn for the comment and @adamvert for the extra ping!

Seems weird not to also have a "auto-send as SMS" directly available.

I do agree with you and what you say is similar to @shobhitagarwal1612's complaint. To go in a bit more depth, the reason for no retry attempt is that (as far as I can tell) there isn't any great trigger for retrying an SMS failure. On the XML submission side, it makes sense to retry when there's a change on network availability. On the SMS side, I think the only option would be a time-based retry. We don't currently have a good way of communicating that kind of status so I think it could become quite confusing. But perhaps you have some alternative ideas? EDIT: this is not quite accurate -- there would currently be retries but only as a response to an internet send failure

Or three options:
Auto send:
[ ] wifi
[ ] cellular
[ ] SMS

True that this is not bad. Maybe another way to communicate that SMS retry is different would be to make the third option something like SMS (only tries once)? EDIT: This is not quite accurate -- there would currently be retries but only as a response to an internet send failure. Maybe SMS (not guaranteed to send)? The big difference is that once a network submission is made, we wouldn't send the SMS because for the projects we've spoken to, it wouldn't be needed if the network submission goes through.

It might be clearer to rename SUBMITTED to e.g. SUBMITTED_HTTP (at least in code, if not in the database) to avoid confusion.

👍

If there's already a database field which notes whether a form has been SUBMITTED then assuming this is not a boolean field then I would guess this can also track the SUBMITTED_SMS status.

The new database column would be to ensure an SMS is only auto-sent once. There's currently just a single status column and so let's say the auto-send via internet fails once, the status column would represent failure. Then an SMS submission would be attempted, if it succeeds, the status column represents SMS success. Then internet submission is attempted again, it fails again, the status column represents failure. Now an SMS submission would be attempted again. So I believe we need a separate status to know whether an SMS was ever successfully auto-sent and if so, to prevent an other SMS auto-send attempt.

I think it would be better to keep track of all send attempts in a separate database table so there's history available but that is a larger change that I'm not sure we want to prioritize now. Adding an SMS-specific column should address the problem without adding much complexity (and not closing the door to the history log I just described).

By my reading, the text and diagram disagree on whether multiple SMS can be sent.
I believe that the text supersedes the diagram, and that only one SMS will ever be sent

Yes, my apologies, I didn't update the beautiful diagram! 😉

[lognaturel]

Discussed again at 10/17 TSC meeting and we concluded that:

• There is really no good way to trigger an automatic SMS retry because
  • delivery reports are unreliable
  • there are no "cell network availability/strength" events we can trigger background jobs from
  • "cell network availability/strength" is not always a good indicator of whether a message can be sent
• The "fallback SMS" checkbox is a reasonable compromise for now
• We can specify that a fallback submission is only attempted once

The specification was approved.

[poVoq]

It would be also useful to only send an SMS if a specific answer is selected.

The user story would be:
General vulnerability assessment, but there might be some urgent to response to cases that should not wait for the general data analysis a while later. So if a certain question is answered with a specific response, send out a SMS as an alert with only the most relevant data.

Of course the full survey response should still be added to the database through the normal internet upload.

Edit: In fact this could also be a general mechanism with a separate, shorter form being populated with such kind of alerts.

[alxndrsn]

Generally 👍 👍 from me. Some specific points addressed below:

Leave “SMS submission settings” for setting destination phone number in Server settings but always enable it (currently it’s only enabled if “Send submissions via” is set to SMS or Select on Send)

I wonder if there is some way to make it clear when this option is relevant if a user sets it. Or perhaps flag it as irrelevant (e.g. via a tooltip) if no SMS submission option is currently selected. But likely not important in a first iteration of these tweaks.

Add toggle below “Wifi/Cellular auto send” that says “Fall back to SMS” with description text “Try SMS after Wifi/Cellular send failure”. This toggle is only available if there is an SMS destination phone number set in “Server”. If this is on, every time a send attempt is made (triggered manually, or on auto-send), if that attempt fails, an SMS is attempted. It can be enabled even if auto send is off.

If I'm reading correctly, there will be no "auto-send as SMS" option, but if this were desired, a user would need to enable "auto-send over wifi/cellular" and wait for that to fail every time. Seems weird not to also have a "auto-send as SMS" directly available.

To support this sequencing, introduce an SMS_SUBMITTED status.

It might be clearer to rename SUBMITTED to e.g. SUBMITTED_HTTP (at least in code, if not in the database) to avoid confusion.

a database column will need to be added to keep track of whether a submission has been successfully sent over SMS. This is necessary to send the SMS only once.

If there's already a database field which notes whether a form has been SUBMITTED then assuming this is not a boolean field then I would guess this can also track the SUBMITTED_SMS status.

What this specification proposes is not a true auto-send because there is no retry so it seems potentially misleading to include there. It also seems this would result in 6 options: Wifi only, Cellular only, Wifi or Cellular and those same ones with SMS retry options.

Or three options:

Auto send:

[ ] wifi
[ ] cellular
[ ] SMS

(options will be tried in that order)

---

Sending an SMS should only be attempted if the form has SMS tags and an SMS hasn't been sent before. That is, only one SMS should ever be sent by this workflow.

By my reading, the text and diagram disagree on whether multiple SMS can be sent.

[admbtlr]

@alxndrsn Based on @lognaturel's comment above, I believe that the text supersedes the diagram, and that only one SMS will ever be sent

[lognaturel]

One more thing that I now remember about why I don't feel great about SMS auto-send being a standalone option -- it's if ONLY SMS is selected that the lack of retry seems really problematic. I think it makes sense when it's in combination with a network send so we know the form will be sent somewhere but it doesn't feel right as its own auto-send option when it does in fact only try once. But perhaps the SMS (only tries once) wording described above would be a way to communicate that.

[yanokwa]

@lognaturel Given that Google no longer allows SMS, I think we should close this issue.

[lognaturel]

Good thing we didn't build it! 😅😭

[MartijnR]

Does that mean that we don't actually have tools ourselves that use the compact submission representation (nor have plans to use this)?

[lognaturel]

Sigh, I suppose that is now the case, @MartijnR. Modulo some folks who really need SMS integration and are sideloading older Collect APKs or reviving the SMS code from a fork since it's still optimistically in there.

There have been requests to enable sending compact records to servers. This would require more spec work to define how exactly that blob would get sent.

[MartijnR]

Ok, thx. I believe Medic Mobile is also using it (is that right @garethbowen?). We could leave it in the spec for now imo.

[garethbowen]

Yes we're using SMS, but we use SMS only so we don't use a fallback process. And yes, also having to sideload it now to get around Google 😭