branch-standalone v0.1 (discussion)

[nivlab]

branch-standalone v0.1 (discussion)

[danielbrianbennett]

Thanks for the thorough review, Sam. To distill a couple of high-level design philosophy points from your comments, it sounds like we

• would be happy to diverge from the structure of other branches where it makes the code functionality of the stand-alone branch more straightforward (e.g., in the redirect functions)

• should worry less about server load given the far lower expected demand of this branch (giving us the option to save partial datasets, for instance)

In terms of specific changes, I will:

• Consolidate redirect_success and redirect_reject into a single redirect route (maybe? see comment below)
• Add partial data saving in beforeunload (neat trick)
• Change the order of the checks in init.py
• Loosen the strict lockout provisions and instead re-allocate subject IDs on the landing page (this + timestamps on experiment start should give participants flexibility while preserving transparency and ensuring overwrites aren't possible)

The idea of removing the workerId/subId distinction is an interesting one, but I am leaning away from it. My rationale is that, although it is true that for CCNP/Geha we will already have assigned anonymised IDs, we can't guarantee that this will be the case in all use cases. In a use case like sona, for instance, it would be nice to be able to use subjects' student IDs as a workerId without having to worry about stripping that out of the data later for the sake of anonymity in data sharing.

On the question of participants re-starting the experiment, I am trying to think of ways that we would check for this under this system. My thinking is that if a participant were to repeat with the same worker ID, we should be able to detect this by looking either at (a) the number of distinct subject IDs that they have been assigned in their metadata file (if they restart from the landing page each time using the same workerId), or (b) the number of different timestamped files that correspond to the same subject ID (if they restart the experiment.html page a few times). The question in my mind is whether there is any way for us to check the number of times that a participant has restarted with workerId modifications, short of checking the IP address (which might have problems associated in cases where lots of participants are using a similar IP address, as at a university).

[danielbrianbennett]

The last thing I think we might want to work out is what the status of the completion codes and redirect routes should be. I'm inclined to say that we could just drop them completely (since the logic of true versus decoy completions no longer makes much sense here), and replace them with a single all-purpose redirect route. The sole counterexample that worries me is a savvy participant who navigates directly to the complete.html page without doing the experiment. Any thoughts on how to work around this?

As is stands, the redirect_success function takes care of this, which might be a reason to keep it.

[danielbrianbennett]

Per our recent conversation, I have replaced the redirect_success and redirect_routes with a single save_data route.

[szorowi1]

I've gone through the code and made some suggested changes in #45. Most changes were minor except (1) some reorganization of the routes in index to prevent IO errors on participant revisits, and (2) collapsing of the save_data and save_partial_data routes. The latter isn't necessary, but simplifies the code some.

I've also set the version back to 0.1. In keeping with previous habits, I'd suggest keeping the version beneath 1.0 until we've had the opportunity to test this out live with actual participants.

Are there any outstanding issues at this point? The only things I can think of is that:

1. We're leaving experimenters to independently verify that participants have completed the experiment. Given the low volume of SONA participants, this doesn't seem too onerous to me but we can revisit this if need be.
2. Savvy incognito participants could still repeat the experiment many times, but I can't imagine why they would do that.

Short of that, I'm happy with v0.1.

[danielbrianbennett]

Thanks, Sam. Yeah, I'm happy with points 1 and 2 there.

There was one issue with the way that you collapsed the redirect routes. The way it was set up, no matter what status was passed, the participant was redirected to the complete page. The consequence of this was that if a participant acidentally pressed back/refresh during the experiment, then even if they selected not to leave the experiment page in the pop-up dialogue box, they would be redirected to the complete page and would not be able to get back to the experiment. This was my initial reason for keeping the two routes separate. In any case, there is a simple enough fix to resolve this in a single redirect route, which I have implemented in a new commit to the standalone-v0.1-sz branch. See what you think; if you're happy with that, I'm happy to merge and close the issue.

[szorowi1]

Ah, I see. That's entirely my fault. I only ever tested by closing the page -- never tried continuing the experient after an onbeforelunload event. Sorry for the inconvenience!

Otherwise, I think the code looks good. Feel free to merge/close. Thanks for your work on this!