nivlab/nivturk

Tools for serving and storing data from online experiments.

nivlab.github.io

The main hub for Niv lab's technical resource sites.

Development

Modifying the portfolio

For step-by-step instructions for adding to or modifying the project portfolio, see Portfolio Theme README.

Modifying the library

Edit the library.md file as you see fit :)

Previewing the demos page locally

If you'd like to preview the site locally (for example, in the process of proposing a change):

1. Clone down the project's repository (git clone https://github.com/nivlab/nivlab.github.io)
2. cd into the project's directory
3. Run bundle install to install the necessary dependencies
4. Run bundle exec jekyll serve to start the preview server
5. Visit localhost:4000 in your browser to preview the project

This starts a Jekyll server using your theme. Add pages, documents, data, etc. like normal to test your theme's contents. As you make modifications to your theme and to your content, your site will regenerate and you should see the changes in the browser after a refresh, just like normal.

When the theme is released, only the files in _layouts, _includes, and _sass tracked with Git will be released.

Acknowledgments

This resource was made possible thanks to Github Pages and the Portfolio Theme for Jekyll.

nivturk's Issues

screen tutorial in wiki

Document headless server approach.

cloudresearch iframe

Cloudresearch updated its completion code iframe. Need to update complete.html to fix.

add recaptchas to consent form

Should be as easy as following the steps here: https://developers.google.com/recaptcha/docs/v3

Documentation - empty cookbook

update consent form

Update consent form language to reflect latest IRB changes.

[mturk] cloudresearch no longer uses dynamic codes

CloudResearch no longer uses dynamic codes. See their blog for details. We'll need to make the MTurk branch like the Prolific branch.

best practices literature

add new page to docs compiling papers on best practices for online experiments.

Nginx?

Ask IT if we should be using Nginx for securing handling of client requests, or if this is taken care of via IT already. For details, see the Flask Mega-tutorial (section: Setting Up Nginx).

update flask version

Update required flask version based on dependency conflicts (pallets/flask#4494).

Consent form links broken

"Basic Usage" section in the documentation disappeared

Hi. I just noticed that the "Basic Usage" section in the Nivturk documentation disappeared. It was fine yesterday but when I opened the page today it's all gone.

crontab script

It would be nice to write a simple bash script + crontab command that periodically provides summary statistics of the number of participants that have accessed the experiment, completed the experiment, run into errors, and have saved datasets.

detect disconnect

One trend we notice with MTurk workers is that they will revisit the experiment page, reach the 1004 error, and send us an email asking for compensation (despite the warning that this would occur). One possibility is that this is just adversarial (scamming) behavior. A second possibility is that participants are experiencing momentary disconnection from the experiment and refreshing (manually or automatically) on reconnect.

To differentiate between the two, one technical solution might have been to monitor user connect/disconnect events using websockets. Unfortunately, this method would most likely fail to solve our use case as manual and automatic refreshes are indistinguishable from the server side.

At present, there does not seem to be an automatic way of detecting legitimate vs. illegitimate refresh behavior. Thus, the present solution unfortunately is to deal with this issue through email. If a new solution arises, we will reopen this issue and address.

[docs] add `kill all` processes

Describe the "kill all background processes" command in the docs: kill -9 $(lsof -t -i:<process #>)

loading issue (needs to move up to access static)

for example :

current -

<script src="/static/lib/jquery-3.3.1/jquery.min.js"></script>

what I think is the correct version

<script src="../static/lib/jquery-3.3.1/jquery.min.js"></script>

bot check

Add bot check (hidden radio button / text form) in Alert page.

turkprime integration

need to link custom completion code to Turk Prime.

fix tracking on incognito

demo branch

We often need to demo tasks (to collaborators, new RAs, etc.). Should make a branch to make this easier.

[docs] update mturk page

With the recent changes to the MTurk branch (#96, #98), the documentation page for running an experiment on MTurk needs to be updated.

Consent form

Let's make sure the template consent form is the most up-to-date version.

adding timestamps

Per Yeon Soon's suggestion, add timestamps to event logging

Debug mode tweak

Currently line 91 of __init__.py operates whether debug mode is true or false. Perhaps this flag should only operate if debug == false.

docs-v2

• updated code architecture page (flowchart diagram)
• jspsych integration page
• local testing page
• updated serving page
• move all lab admin stuff off the site

Log URL tampering

We've noticed some Turkers attempt to manipulate the URL. Currently this kicks participants from the experiment, but without logging the identity of the tamperer. This would be helpful information as we could add them to a TurkPrime universal exclude list.

A fix should be relatively simple. Anyone being redirected to one of the error pages should have a note written to their metadata.

jspsych v6.2.0

At some point -- when everything is peaceful, quiet, and at minimal risk for breaking active projects -- update NivTurk to latest version of jsPsych (https://github.com/jspsych/jsPsych/releases/tag/v6.2.0).

complete page redirect

Presently, participants are redirected to /complete twice: once via the jspsych-return-data function at the end of an experiment and once via the /datadump POST call. Though this isn't strictly a problem (everything still works) it would be cleaner to redirect only once.

tutorial on browser testing

We should have a protocol in place for testing experiments across multiple browsers (Chrome, Safari, Firefox, Opera, etc.). A tutorial should be written on the subject.

update consent forms

[bug] interference across different nivturk experiments

Bug report: interference across different nivturk experiments

Users affected: @mbhare, @ines-as

Summary

Experimenters @mbhare and @ines-as ran into technical difficulties wherein a (partially) overlapping set of participants were automatically redirected to the completion page after starting an experiment. @ines-as was running a two-day longitudinal experiment (two separate NivTurk apps) and experienced the error on the second day of testing. @mbhare was running a cross-sectional experiment and experienced the error for participants that completed @ines-as' first day of testing. Importantly, participants were redirected with the completion code from the first session of @ines-as' experiment. The experiments were served on using different URLs, ruling out the possibility of a previous cache by URL.

bonusing scripts

Before our February presentation, we might consider writing a couple of scripts that make it more convenient to pay/bonus participants. This would involve matching workerIds to subjectIds, and letting the user input some additional CSV mapping subjectIds to bonus amounts.

delete load-test branch

The load-test branch is hopelessly outdated at this point -- it hasn't been updated since before we switched to a versioning system. We've got a pretty good feel for what our servers can handle, and the load-test branch was never integrated with jsPsych.

@danielbrianbennett, any reason to keep it?

v1.0.0 release

• Code review: review, discuss, and develop plan to address any outstanding code issues or deficiencies.
• Documentation: review wiki, find out-of-date sections, and update to reflect major changes made since January. Maybe even migrate to readthedocs?

improve html

The current HTML pages are a little ugly. Would be nice to clean up.

jspsych 6.3 upgrade

As title suggests, upgrade to latest jspsych release (6.3+)

alert message on refresh/back

As suggested by Hartley lab, look into having an alert message appear when a user presses the back or refresh buttons.

add funding agency info

This development of this software was supported in part by the National Center for Advancing Translational Sciences (NCATS), a component of the National Institute of Health (NIH), under award number UL1TR003017.

jspsych 7.2

it's that time. jsPsych is now up to v7.1. Should we update? Thoughts? @danielbrianbennett @karnigili

intermediate logging

Allow users to pass messages to metadata files during a jsPsych experiment.

branch-standalone v0.1 (discussion)

standalone branch review

Motivation

The purpose of the standalone branch is for running behavioral experiments through the browser with participants recruited from more traditional settings (e.g. university undergraduates, clinic patients, community pool). The expectation is that participants will be directed to the experiment link by the researchers themselves or by the university subject pool software.

Design thinking for this branch should differ from the other branches given the following:

• Lower throughput: Experiments run with the standalone branch should expect to have few active participants at any point in time. This is especially the case for clinical experiments, where experiments are run with patient supervision. As such, we should assume less computational demand for this branch.
• Lower malfeasance: Experiments run with the standalone branch should expect to have lower risk of active tampering. This is because (a) subject anonymity is not guaranteed for undergraduate participants, and malfeasant behavior can be reported, and (b) participants can expect to be supervised in the case of clinical experiments. As such, we should assume lower risk for this branch.
• Higher value: Data collected from experiments using the standalone branch may have higher value. This is especially the case for clinical experiments. As such, we should assign higher value to partial datasets for this branch.

Use Case

The typical participant experience for an experiment using this branch may look as follows:

A participant opens the URL that was given to them by the experimenter. If they are not given a user ID, they are directed to a page where they input their user ID. Alternately they may have been provided a link that contains their user ID in the URL. In either event, the participant is directed next to the consent form and then experiment. Once the experiment is started, participants are not allowed to restart the experiment under the same user ID. Once the experiment is complete, participants are directed to a complete page informing them it is safe to close the browser.

Review: Major Points

The major review point I am trying to think through is the primary use case and how that should inform the code. For projects like CCNP and Geha, participants will always be supervised and locking participants out on accidental/intentional reset seems more of a technical hindrance than help. An alternate solution for these situations would be to make sure that all datasets are saved out with timestamps so there's no chance of overwriting data. Separately, if the inputted subject IDs are already anonymised, we do not need separate mappings between workerId and subjectId.

For the case of SONA and undergraduate participants, I begin to see the need for lockouts on accidental/intentional resets. Though the current architecture prevents users from restarting the experiment if they enter the same workerId (by URL or input page), they are free to restart the experiment an infinite number of times if they simply modify their workerId (e.g. workerId_2). I wouldn't put this sort of behavior past a student who was worried about course credit.

I wonder if this suggests some further modifications to the architecture for this branch, where participants are always free to restart the experiment. In this setup, participants are only tagged by their "workerIds" and additionally assigned timestamps on experiment start to prevent overwriting data.

Relatedly, I wonder if all data (partial or complete) should be saved for this branch. Given that we should expect lower demand on the server for these experiments, I do not think partial data saving (e.g. on browser unload) should pose an issue. One possible implementation for this is discussed below.

In short, I'm wondering if it makes sense to elaborate on the use case some more and proceed from there. I've tagged @jbransonbyers given his experiences with Geha. We can also involve Laura given her likely future use for CCNP.

Review: Minor Points

• In init.py, I think the mobile user check should precede the workerId check. As it stands, mobile/tablet users without a workerId will be allowed to visit the input page and then proceed to the experiment.

• As discussed on slack, we can simplify the codebase and remove unused routes. If redirect_success and redirect_reject do not need to be separate, then this can be collapsed.

• As pointed out by @lizzytong2589, there's a simple trick for saving partial data if that's of any interest. You can add a save data function into the beforeunload event such that partial datasets are saved when a participant goes to refresh/close their browser. This is a simple, low-effort fix that can mean saving partial data in the event a user runs into technical issues. (I wouldn't do this for MTurk or Prolific where partial datasets have low value, but may be useful for SONA or CCNP.)

partial data saving

• add new partial dataset folder
• add code to save data on page close
• make mode toggleable
• update alert text ("you may lose your progress")

Documentation - add intro explaining nivturk

version control

at least add a version variable into the application script to allow users to keep track of what version they're using.

debug mode

Allow for users to test with a debug mode. In configuration file, specify whether in production or debug. Then, allow users to bypass repeat visits in experiment.py. Should also add warning about being in debug mode.

[docs] write testing / troubleshooting guide

Need to write a section walking users thru how to test their experiments before launching, and what bugs to look out for (e.g. need to reset cache)

relax exclusion requirements

Instead of kicking participants out on revisit of home screen, exclude participants on reload only if they make it to experiment page.

flask server password

The current behavior for leaving the Flask server with the default password is a little unsafe. Should add a warning (or error?) to the user to change the password.

user OS

Record user operating system / device.

redirect success/reject logic as default in experiment.html

Basically, the idea is to make the following conditional a default in experiment.html in the MTurk/Prolific branches:

if (low_quality) {
    // Save rejected dataset to disk.
    redirect_reject("1005");
} else {
    // Save complete dataset to disk.
    redirect_success("{{workerId}}", "{{assignmentId}}", "{{hitId}}",  "{{a}}", "{{tp_a}}", "{{b}}", "{{tp_b}}", "{{c}}", "{{tp_c}}");
}

The goal would be to make more explicit the functionality of NivTurk and the choices experimenters have in how they perform rejections.

Session variable not persisting

In standalone branch, session variable sometimes does not persist when workerId is entered as a URL variable. Will attempt a fix based on this advice:

https://stackoverflow.com/questions/39261260/flask-session-variable-not-persisting-between-requests/39261335