Tasks created with Qualtrics and QRTE

– Word association –

A series of spoken words are presented, and for each word, the participant’s task is to type in an associated word. For instance, after hearing the word ‘kitten’, one might type ‘cat’, ‘pet’, ‘whiskers’, etc.

Note: this task does not work in Internet Explorer or Edge browsers due to compatibility issues with HTML5 audio. The task does work on mobile devices.

– Two-word semantic relatedness –

Two words are presented in succession, and the participant’s task is to indicate (as quickly and accurately as possible) whether or not the meanings of the two words are related.

Note: this task does not currently work in Internet Explorer or Edge browsers due to compatibility issues with HTML5 audio.

– Visual digit span –

Participants see a sequence of digits, presented one at a time, and then have to recall the number sequence in the same order. The task contains list lengths 3-10 for forward span and 3-8 for backwards span. There are 3 trials for each list length. The list lengths start at 3 digits and increase to the next list length if at least one of the responses was correct. If all 3 responses for a given list length are incorrect, then the task stops.

– Flanker –

On each trial, a sequence of 5 arrows (e.g. <<><<) is presented and the participant must indicate the direction of the middle arrow by pressing one of two keys on the keyboard.

---

Tasks created with jsPsych, running on JATOS

– Picture selection –

Participants hear a sentence and then see two pictures. The task is to indicate which picture is most closely related to the meaning of the sentence. Some of the sentences in this example use words with multiple meanings, so the participant must select the picture that reflects the meaning of the word that is consistent with the sentence context. Responses can be made with mouse clicks or touch responses.

Note: this task does not currently work in Internet Explorer or Edge browsers

---

Resources for online experiments

Tools

jsPsych: an open-source JavaScript library for creating web-based experiments. [paper, documentation, GitHub]

JATOS: an open-source, GUI-based web application for communicating with a web server to host studies and storing/managing the data. [paper, documentation, GitHub]

The Experiment Factory: a collection of open-source web-based experiments. [paper, website]

PsiTurk: an open-source framework for conducting online experiments, interfaces with Amazon Mechanical Turk. [paper, website]

Xperiment [website]

Gorilla: a GUI-based online experiment building tool by Cauldron. [Gorilla website, Cauldron website]

QRTE (Qualtrics Reaction Time Engine): an open-source JavaScript engine for accurate stimuli presentation and reaction time data collection, which is embedded into the Qualtrics online survey software. [paper, website and tutorials] (Note: QRTE is no longer supported by the developers)

Useful Articles

de Leeuw et al. Online Experiments using jsPsych, psiTurk, and Amazon Mechanical Turk [pdf]

de Leeuw, J. R., & Motz, B. A. (2016). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. Behavior Research Methods. [link]

Germine et al. (2012). Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic Bulletin & Review. [link]

Hilbig, B. E. (2016). Reaction time effects in lab- versus Web-based research: Experimental evidence. Behavior Research Methods. [link]

Meade, A. W., & Craig, S. B. (2012). Identifying Careless Responses in Survey Data. Psychological Methods. [pdf]

Pinet et al. (2016). Measuring sequences of keystrokes with jsPsych: Reliability of response times and interkeystroke intervals. Behavior Research Methods. [link]

Reimers, S., & Stewart, N. (2015). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods. [link]

Semmelmann, K., & Weigelt, S. (2016). Online psychophysics: reaction time effects in cognitive experiments. Behavior Research Methods. [link]

Woods et al. (2015). Conducting perception research over the internet: A tutorial review. The PeerJ. [link]