Remembering previously entered values in Collect

[getodk]

Remembering previously entered values in Collect

[cooperka]

Thanks for the feedback @tiritea and @MartijnR. I'm curious what you think of some past discussion about templates here. It was proposed and partially implemented, but ultimately abandoned.

I largely agree with @smoyth's concern in that same discussion:

I don't think many people would, on their own, think to create a template as a way of reusing a value for a single field (team name).

I think templates are a great idea in general, and they have their place, but in my experience they also tend to be messy, confusing, and frustrating when the user just wants a single field or two to recur based on the most recent value.

In the use cases here, the user would have to remember to do all of the following in order:

1. Open a blank form
2. Fill out a template with only the fields they want to be remembered
3. Make a copy of the template to be filled out (making sure never to accidentally fill out the template itself, because that would ruin it and they'd need to start over)
4. [Encounter something that causes the fields to change, such as a new street name]
5. Edit the template before starting the next real form
6. Go to step 3 and repeat

For comparison, with this proposal:

1. Form designer designates fields that are remembered
2. Forms are filled out as usual

Again, I think templates have their place, but I think they're too complex to fully meet the users' needs for these use cases.

[yanokwa]

The heart of this feature is having a question populated with the last saved answer. You are essentially pre-filling the form and we do have a reasonable XForms mechanisms for doing that: secondary instances.

Now the mechanism doesn't have to be the UI! That is, we shouldn't have people make templates because I agree with @smoyth and @cooperka that it's way too complicated. I had a quick chat with @lognaturel about earlier conversations she's had with @MartijnR and here's the rough approach we came up with.

For questions that want the previously saved answer, use a setvalue that fires on xforms-ready. For the value, you can use XPath expressions and some agreed upon syntax to refer to the last filled instance of that form id (e.g., instance(last-finalized)/data/teamname).

That syntax will be a hint to the client to load that last instance as a secondary instance and grab the value. For bonus points, we can parameterize the syntax so we can get the last n values. This would be helpful as a magically historical auto-complete for text widgets (kinda like Option C). I think we have most of the pieces here to get this to work and the only new thing is the syntax for getting the last filled instance.

P.S. I don't think we should use preloads because those should probably be deprecated since we have actions

P.P.S. I think we shouldn't implement templates first because the UI changes to Collect are going to be disruptive and hard to roll back.

[ggalmazor]

Hi all!. Apologies for stalling the conversation for the last week!

It looks like the latest agreement is that we would be preloading the last saved value using setvalue and xforms-ready, which sounds fine to me.

Let me roughly describe what could be the simplest way to go about this with JR/Collect, which could probably translate to other platforms as well:

• Collect gives JR the path to the last saved submission's XML file when preparing for a new submission.
• JR parses it under a virtual (as in "it isn't declared in the blank form") last-saved secondary instance.
• If Collect doesn't provide the path to the last saved submission (maybe it's the first submission being filled), then JR uses a blank submission instead.

This mechanism would involve making changes to the code that handles the instantiation of new submissions only, which feels like a good balance between cost and feature value, with no important foreseeable tradeoffs.

I do think that we should provide some syntactic sugar for our users that would let them set this up more easily through any method that feels suitable, but we can think about this later.

On the other hand, remembering all the values saved on each field is a whole other kind of beast. Honestly, I can't suggest a good way to express this at the specs level or a smooth way to implement this within JR that doesn't feel like overkill. At this point, I would be more inclined to solve this externally with some client-side cache as @cooperka suggested.

In any case, I think that these are two different and complementary use cases: preloading the last saved value for a field vs. virtually making all the controls auto-completable with any previously used value for that field.

I think the latter would require much more discussion, starting from how this would actually affect the UI, and I'd suggest we punt on that while we can deliver the former and the users have a chance to play with it.

[tiritea]

• As a census taker, I write the current street name on a napkin so I can remember it for each house I visit.
• As a natural disaster responder, I enter the exact same earthquake date every time I interview someone from the same area.
• As a forest surveyor, I type in my team name and choose a region code every time I log a new tree. Sometimes I make typos.

These specific usecases suggest that the individual enumerator is deciding they want to reuse certain (arbitrary?) fields in a form, whilst out in the field; that is, they simply want to add (or override) default literal values specified in the form's instance XML; ie the form defaults that the form writer explicitly added when creating the form.

Rather than having to introduce new features for the form definition language itself (new XForm binding attributes, new XForm appearances, new XPath functions, ...) it would seem this could be more easily accomplished, and more applicable generally, by simply initially populating the form with a previously user-defined instance XML template, rather than the instance XML from the form definition. This could be entirely accomplished (optionally) on the XForm client (ODK Collect or otherwise), with no modifications necessary to the ODK XForm spec itself nor any existing forms already deployed out there. By adding a "Save as Template" button which would save the current instance XML contents of the form you are filling onto the device, and correspondingly a "Fill Template Form" button as an alternative to "Fill Blank Form", which would initialize the new form submission from the template instance XML instead of the form instnace XML. Whether we overwrite a single template per form, or allow users to save multiple templates under different names, can be a client-specific implementation detail; again, nothing needs to change in the spec.

At any time whilst out in the field, an enumerator could choose to save what they got as template to re-use again subsequently; eg start a form and fill in just the street (or earthquake date, or team name and region), save it, and reuse this to initialize new forms if they want to, or start a clean form from scratch if they dont. Or they are in the middle of filling in a household survey and suddenly decide they going to need to re-enter a bunch of data for everyone present, so save the partiallt filled form for first household member and use it as a starting point for everybody else. Again, the determination is entirely made by the enumerator, whilst out in the field, as immediate circumstances dictate; no prediction needs to be made by the form writer as to what can/not be reused (ie cached), it can be applied to any existing forms out there, and how or even if its supported (eg 1 or many templates, or none) is entirely client dependent.

FYI I do something very similar wrt re-inspections in our app; basically, basically the inspector determines whether this is a new inspection - and initialize with form instance XML if so - or a re-inspection, in which case the form is initialized with last submitted instance XML [our problem is being able to specify what questions to automatically clear on a re-inspections, eg signatures. But this isnt an issue for user-defined templates].

I would suggest that upon the enumerator deleting the form from the device, that any associates tempates are automatically deleted also (ie clear the cache).

This still leaves open the question of other usecases and potential mechanisms for prepopulating form date from outside sources (eg preload CSV's, extract default properties from accompanying XML files, etc) but at least for the usecase described - "filling out the same base form multiple times on the same device" - it may possible to accomplish this is in a fairly straight-forward way by initializing forms from a user-saved template instance XML data, rather than form instance XML data This could accommodate a wide variety of usecases without needing to change form definitions or add new features to the ODK spec; its merely a client-side implementation detail. At least for my re-inspections, this has turned out to be a very workable field solution that is both easy for our inspectors to understand and imposes no additional burden of our form creators.

[MartijnR]

it would seem this could be more easily accomplished, and more applicable generally, by simply initially populating the form with a previously user-defined instance XML template, rather than the instance XML from the form definition

This approach (purely client-side, no XForms syntax) seems the best to me. I think it is a very cool idea. Very logical, testable, no magic reference to a previous-record-on-same-device-for-same-form.

[tiritea]

Thnx. Looks like there has been some good discussion around such 'templates' already... I've added a comment to that thread (rather than append here).

Overall, honestly, I'm still not convinced templates are too (more) complex; indeed most of these other approaches appear to be more complex to implement, and consequently could result in all sorts of unanticipated gotchas. Also, templates can be implemented today with no changes introduced to the ODK spec - so nothing we may regret wrt legacy support later - and it can be used for any forms in existance today. Why not implement templates, get some feedback from actual field use, and use that to identify what common usecases could be handled better, before actually committing to change the spec? But again, that's just my $0.02... I have no specific objections to the proposed Option A.

[tiritea]

Thank you for the detailed response. I guess I'm just loath to introduce anything new to the ODK XForms spec definition until we've exhausted and fully explored all other options, because it becomes (a) something that we can end up being forced to live with (ie legacy support of arguably 'mistakes', and AFAIK nothing has been officially deprecated yet), and (b) spec changes mean [quite selfishly] more work for other tooling attempting to be compatible with the ODK stack. :-)

[of course, I say that somewhat hypocritically, what with an aborted attempt to introduce an unnecessary numberOrZero() XPath function recently... ;-) ]

[MartijnR]

Apologies for not reading the template discussion in the forum thread.

Though a template solution is still technically far more attractive, I can see they may be too cumbersome for users of mobile clients (It’s different for a web client such as Enketo which can - and does - do this centrally by dynamically populate defaults via the URL).

For questions that want the previously saved answer, use a setvalue that fires on xforms-ready. For the value, you can use XPath expressions and some agreed upon syntax to refer to the last filled instance of that form id (e.g., instance(last-finalized)/data/teamname).

This setvalue approach with a secondary instance may be best the way to do this without a template (also worth looking into virtual endpoints like CommCare instead of instance(…) but that’s essentially the same approach). Clients that don't support the feature would just return an empty '' response because the (virtual) node doesn't exist.

[cooperka]

@ggalmazor thanks for the thorough example above. I've recently begun looking through the code to implement this, and I'd appreciate it if you could sanity-check my thinking so far (all I need is a nice big thumbs up 😸).

As best as I can tell, here's the series of steps taken starting from the user opening a blank form through to loading the formInstances used by that form (soon to include the new last-saved instance).

• FormChooserList#onItemClick (user opens a blank form)
• FormEntryActivity#loadFromIntent (called on new form)
• FormLoaderTask#doInBackground (start loading the form)
• FormLoaderTask#createFormDefFromCacheOrXml (returns FormDef with formInstances)

Case A: formdef is cached

• FormDefCache#readCache (foo.formdef => returns FormDef)

Case B: formdef not cached

• XFormUtils#getFormFromInputStream (foo.xml => returns FormDef)
• XFormParser#parseDoc (parse instance nodes)
• ExternalDataInstance#build (given an external src)
• FormDef#addNonMainInstance (every time it finds an internal/external <instance>)

What I want (i.e. your suggested implementation above) is very simple: add the last saved form instance to the formInstances list via FormDef#addNonMainInstance. This particular instance is referred to by the last-saved key (e.g. instance(last-saved)/data/foo in XML). Once that happens, everything else should work automatically.

Am I on the right path so far?

If so, it looks like I'll need to have the last saved instance (or null) in Collect's FormLoaderTask#createFormDefFromCacheOrXml and somehow pass that along to JavaRosa to be built along with any other external instances. This sounds like what you're suggesting. I'm going to keep forging ahead but I wanted to make sure I'm not totally off-course here.

[ggalmazor]

@cooperka

If so, it looks like I'll need to have the last saved instance (or null) in Collect's FormLoaderTask#createFormDefFromCacheOrXml and somehow pass that along to JavaRosa to be built along with any other external instances. This sounds like what you're suggesting. I'm going to keep forging ahead but I wanted to make sure I'm not totally off-course here.

Everything you've said is looking good. Just let me note that if no last saved form is available to be provided to JR, JR should create a blank one and use it instead.

An optimization would be to search for any use of instance('last-saved') in the form to prevent having to parse it (or generate a blank one) in the first place.

I won't be able to be of much help in the Collect side, although I can review and try stuff as you progress (maybe you can create a WIP PR)

[cooperka]

Exactly 👍 I have this working and will submit PRs soon.

[tiritea]

Couple quick initial comments (while I chew it over... great stuff BTW!):

"...filling out the same base form multiple times on the same device".

• I'd probably highlight the fact this is all about a particular device. This distinguishes/isolates this particular usecase to probably a single enumerator, and probably during a single 'assignment'. Which contrast with, say, prefilling instance data via form-specified defaults in the XForms XML instance data, or fetching previously submitted form results and using that to prefill a form (something we do with re-inspections). Some of the options you mention may or may not be applicable to the other types of "prefilling data" usecases, so it might help to eliminate them from current consideration (or not?).

"The cache would be cleared if you clear app data via system settings, but not if you merely delete the form instance from your device."

Given this is effectively a cache for a particular form on a particular device, it would seem to make sense that the cache should be flushed when said form is removed from the device. Not deleting a form's cache when a form is wiped off an device doesn't seem advantageous, and might well just make life complicated later... why not delete everything associated with the form, cache and all?

[cooperka]

So it sounds like the way forward here is with Yaw's setvalue action with an XPath expression.

I think e.g. instance(last-saved)/data/foo makes sense for the expression (note the name change from last-finalized to last-saved here). It can optionally be parameterized as suggested, e.g. instance(last-saved)[2]/data/foo, but that may be a feature for later depending on how much complexity it adds.

@yanokwa are you okay with me moving forward with a PR? Any other concerns?

[yanokwa]

@cooperka I'm OK with you moving forward with this general approach, but it'd be good to get @ggalmazor and @dcbriccetti to chime in since they've been recently in that code.

@MartijnR I know nothing about virtual endpoints, but I suppose that doesn't prevent me from having an opinon! It looks like Dimagi also calls them artificial instances and they are designed to "load data from the application layer. This data must be served up as an abstract XML data structure which is handled and processed like a real XML document." In our case, we have a regular old XML document, so I think JR can use regular old binary endpoints. Agreed?

[aurdipas]

What if I've enabled the "autosend" and at the same time the "delete after send" on my collect?

[ggalmazor]

What if I've enabled the "autosend" and at the same time the "delete after send" on my collect?

Thanks for the question, @aurdipas!

Collect could still save a copy of the "last saved submission" somewhere safe.

[MartijnR]

In our case, we have a regular old XML document, so I think JR can use regular old binary endpoints. Agreed?

Seems like we can leave that open (some clients may use a relational db) as the URI is the same so something like this?

<instance id="myform">
    <data>
         <a/>
    </data>
</instance>
<instance id="__last-saved" src="jr://instance/last-saved"/>
...
<setvalue event="xforms-ready" ref="/data/a" value="instance('__last-saved')/data/a" />

I think the id attribute value is open and doesn't have to be defined in the XForms Spec (it is only referred to within the same form). May make sense for XLSForm to use something that reduces the chance of conflicts with user-added instances (e.g. by prepending double underscores). (And when we replace preloads, we could do the same for a jr://instance/session instance).

Something we may want to make sure is that xforms-ready doesn't fire when a draft record is loaded for editing (according to W3C). If so, we'd have to introduce another event that doesn't re-fire.

[ggalmazor]

@cooperka, have you thought about what @MartijnR's has commented? I'm taking some time to dig into that tomorrow and we can discuss it later ;)

[cooperka]

so something like this?

<instance id="whatever-you-want" src="jr://instance/last-saved"/>

That sounds great to me, the slightly modified syntax makes it more clear to the user where the data is coming from. It's less "magic" which is good.

Defaulting id to __last-saved when building forms seems reasonable, though that's separate from the changes I'll be making.

It's not too much of a difference in terms of implementation, so I'll move forward with the assumption that's how we'll format it, but the old way is just as easy if there are any objections.

[ggalmazor]

@cooperka, if I understand correctly, with @MartijnR's suggestions, Collect still has to provide the path to a last saved form, and then JR will use it only if the user declares an instance with a src="jr://instance/last-saved". Then, the user may refer to that instance using whatever id they choose.

If this is correct, I think it's an improvement because it's more explicit :)

[lognaturel]

Released in JavaRosa v2.14.0 and Collect v1.21.0.

Pending pyxform implementation: XLSForm/pyxform#337

[lognaturel]

Released in pyxform 1.0.0, XLSForm Online 2.0.0. Documentation: https://docs.opendatakit.org/form-logic/#values-from-the-last-saved-record