jsPsych/docs/markdown_docs/features/data.md

Data in jsPsych: permanent and non-permanent data.

There are two very different kinds of data storage: data stored in memory and data stored permanently. Data stored permanently exists even after the browser running jsPsych closes, typically in a database or in a file on a server. Data stored in memory exists only as long the browser window running jsPsych is open.

jsPsych has many features for interacting with data stored in memory, but few for permanent data storage. This is a deliberate choice, as there are dozens of ways that data could be stored permanently. jsPsych does not lock you into one particular solution. However, saving data permanently is obviously a crucial component of any experiment, and the second half of this page contains a few suggestions on how to accomplish permanent data storage.

Storing data in jsPsych's data structure

jsPsych has a centralized collection of data that is built as the experiment runs. Each trial adds to the collection, and you can access the data with various functions, including jsPsych.data.get(), which returns the entire set of data.

In most cases, data collection will be automatic and hidden. Plugins save data on their own so it is not uncommon to have the only interaction with the data be at the end of the experiment when it is time to save it in a permanent manner (see sections below about how to do this). However, there are some situations in which you may want to interact with the data; in particular, you may want to store additional data that the plugins are not recording, like a subject identifier or condition assignment. You may also want to add data on a trial by trial basis. For example, in a Stroop paradigm you would want to label which trials are congruent and which are incongruent. These scenarios are explored below.

Adding data to all trials

Often it is useful to add a piece of data to all of the trials in the experiment. For example, appending the subject ID to each trial. This can be done with the jsPsych.data.addProperties() function. Here is an example:

// generate a random subject ID with 15 characters
var subject_id = jsPsych.randomization.randomID(15);

// pick a random condition for the subject at the start of the experiment
var condition_assignment = jsPsych.randomization.sampleWithoutReplacement(['conditionA', 'conditionB', 'conditionC'], 1)[0];

// record the condition assignment in the jsPsych data
// this adds a property called 'subject' and a property called 'condition' to every trial
jsPsych.data.addProperties({
  subject: subject_id,
  condition: condition_assignment
});

Adding data to a particular trial or set of trials

Data can be added to a particular trial by setting the data parameter for the trial. The data parameter is an object of key-value pairs, and each pair is added to the data for that trial.

var trial = {
  type: 'single-stim',
  stimulus: 'imgA.jpg',
  data: { image_type: 'A' }
}

Data declared in this way is also saved in the trials on any nested timelines:

var block = {
  type: 'single-stim',
  data: { image_type: 'A' },
  timeline: [
    {stimulus: 'imgA1.jpg'},
    {stimulus: 'imgA2.jpg'}
  ]
}

Aggregating and manipulating jsPsych data

When accessing the data with jsPsych.data.get() the returned object is a special data collection object that exposes a number of methods for aggregating and manipulating the data. The full list of methods is detailed in the data module documentation.

Here are some examples of data collection manipulation.

All data generated by the single-stim plugin:

var data = jsPsych.data.get().filter({trial_type: 'single-stim'});

All data generated by the categorize plugin with a correct response:

var data = jsPsych.data.get().filter({trial_type: 'categorize', correct: true});

All data with a response time between 100 and 500ms:

var data = jsPsych.data.get().filterCustom(function(x){ return x.rt >= 100 && x.rt <=500 });

Applying filters consecutively to get all trials from a particular plugin with a response time above 100ms:

var data = jsPsych.data.get().filter({trial_type: 'single-stim'}).filterCustom(function(x){ return x.rt > 100; });

Getting the data from the last n trials:

var n = 3;
var data = jsPsych.data.get().last(n);

Getting the data from the last n trials with a correct response:

var n = 3;
var data = jsPsych.data.get().filter({correct: true}).last(n);

Getting the data from the first n trials:

var n = 3;
var data = jsPsych.data.get().first(n);

Counting the number of trials in a data collection:

var count = jsPsych.data.get().filter({correct: true}).count();

Selecting all of the response times from a data collection:

var response_times = jsPsych.data.get().select('rt');

Calculating various descriptive statistics on the response times in a data collection:

jsPsych.data.get().select('rt').mean();
jsPsych.data.get().select('rt').sum();
jsPsych.data.get().select('rt').min();
jsPsych.data.get().select('rt').max();
jsPsych.data.get().select('rt').variance();
jsPsych.data.get().select('rt').sd();
jsPsych.data.get().select('rt').median();
jsPsych.data.get().select('rt').count();

Storing data permanently as a file

This is one of the simplest methods for saving jsPsych data on the server that is running the experiment. It involves a short PHP script and a few lines of JavaScript code. This method will save each participant's data as a CSV file on the server. This method will only work if you are running on a web server with PHP installed, or a local server with PHP (e.g., XAMPP).

This method uses a simple PHP script to write files to the server:

<?php
// the $_POST[] array will contain the passed in filename and filedata
// the directory "data" must be writable by the server
$filename = "data/".$_POST['filename'];
$data = $_POST['filedata'];
// write the file to disk
file_put_contents($filename, $data);
?>

The file_put_contents($filename, $data) method requires permission to write new files. An easy way to solve this is to create a directory on the server that will store the data and use the chmod command to give all users write permission to that directory. In the above example, I prepend the directory data/ to the filename, and that directory is writable.

To use the PHP script, the JavaScript that runs jsPsych needs to send the filename and filedata information. This is done through an AJAX call.

function saveData(name, data){
  var xhr = new XMLHttpRequest();
  xhr.open('POST', 'write_data.php'); // 'write_data.php' is the path to the php file described above.
  xhr.setRequestHeader('Content-Type', 'application/json');
  xhr.send(JSON.stringify({filename: filename, filedata: filedata}));
}

// call the saveData function after the experiment is over
jsPsych.init({
   // code to define the experiment structure would go here...
   on_finish: function(){ saveData("experiment_data.csv", jsPsych.data.get().csv()); }
});

To use this in an actual experiment, it would be important to tie the filename to some unique identifier like a subject number. Otherwise the file may be overwritten by collecting new data.

Storing data permanently in a MySQL database

The ideal solution for storing data generated by jsPsych is to write it to a database.

There are dozens of database options. MySQL is one of the most popular relational databases, is free to use, and relatively easy to install. This code will assume that you have a MySQL database installed on your server that is hosting the jsPsych experiment, and that your server is able to execute PHP code. If you are trying to run on a local machine, you'll need to install a local server environment like XAMPP.

You'll need two PHP scripts. The first is a configuration file for your database. Save it as database_config.php on your server. Within this file are configuration options for the database. You'll need to change these according to how you have configured your MySQL installation.

<?php
  $servername = "localhost";
  $port = 3306;
  $username = "username";
  $password = "password";
  $dbname = "database";
  $table = "tablename";
?>

The second PHP file will write data to the database. This script reads the database to discover what columns are in the table, and then only allows data to be entered in that matches those columns. This is a security feature. Save this file as write_data.php on your server.

<?php

// this path should point to your configuration file.
include('database_config.php');

$data_array = json_decode(file_get_contents('php://input'), true);

try {
  $conn = new PDO("mysql:host=$servername;port=$port;dbname=$dbname", $username, $password);
  $conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
  // First stage is to get all column names from the table and store
  // them in $col_names array.
  $stmt = $conn->prepare("SHOW COLUMNS FROM `$table`");
  $stmt->execute();
  $col_names = array();
  while($row = $stmt->fetchColumn()) {
    $col_names[] = $row;
  }
  // Second stage is to create prepared SQL statement using the column
  // names as a guide to what values might be in the JSON.
  // If a value is missing from a particular trial, then NULL is inserted
  $sql = "INSERT INTO $table VALUES(";
  for($i = 0; $i < count($col_names); $i++){
    $name = $col_names[$i];
    $sql .= ":$name";
    if($i != count($col_names)-1){
      $sql .= ", ";
    }
  }
  $sql .= ");";
  $insertstmt = $conn->prepare($sql);
  for($i=0; $i < count($data_array); $i++){
    for($j = 0; $j < count($col_names); $j++){
      $colname = $col_names[$j];
      if(!isset($data_array[$i][$colname])){
        $insertstmt->bindValue(":$colname", null, PDO::PARAM_NULL);
      } else {
        $insertstmt->bindValue(":$colname", $data_array[$i][$colname]);
      }
    }
    $insertstmt->execute();
  }
  echo '{"success": true}';
} catch(PDOException $e) {
  echo '{"success": false, "message": ' . $e->getMessage();
}
$conn = null;
?>

To send the data, we use an AJAX request in JavaScript.

function saveData() {
  var xhr = new XMLHttpRequest();
  xhr.open('POST', 'write_data.php'); // change 'write_data.php' to point to php script.
  xhr.setRequestHeader('Content-Type', 'application/json');
  xhr.onload = function() {
    if(xhr.status == 200){
      var response = JSON.parse(xhr.responseText);
      console.log(response.success);
    }
  };
  xhr.send(jsPsych.data.getData().json());
}

You can call the saveData() function using the on_finish handler for the experiment, or by using the call-function plugin.

// with on_finish handler
jsPsych.init({
  on_finish: saveData
});

// with call-function plugin
timeline.push({
  type: 'call-function',
  func: saveData
});