## Data in jsPsych: permanent and non-permanent data.

There are two very different kinds of data storage: data stored in **memory** and data stored **permanently**. Data stored permanently exists even after the browser running jsPsych closes, typically in a database or in a file on a server. Data stored in memory exists only as long the browser window running jsPsych is open.

jsPsych has many features for interacting with data stored in memory, but few for permanent data storage. This is a deliberate choice, as there are dozens of ways that data could be stored permanently. jsPsych does not lock you into one particular solution. However, saving data permanently is obviously a crucial component of any experiment, and the second half of this page contains a few suggestions on how to accomplish permanent data storage.

## Storing data in jsPsych's data structure

jsPsych has a centralized collection of data that is built as the experiment runs. Each trial adds to the collection, and you can access the data with various functions, including `jsPsych.data.get()`, which returns the entire set of data.

In most cases, data collection will be automatic and hidden. Plugins save data on their own so it is not uncommon to have the only interaction with the data be at the end of the experiment when it is time to save it in a permanent manner (see sections below about how to do this). However, there are some situations in which you may want to interact with the data; in particular, you may want to store additional data that the plugins are not recording, like a subject identifier or condition assignment. You may also want to add data on a trial by trial basis. For example, in a Stroop paradigm you would want to label which trials are congruent and which are incongruent. These scenarios are explored below.

### Adding data to all trials

Often it is useful to add a piece of data to *all* of the trials in the experiment. For example, appending the subject ID to each trial. This can be done  with the `jsPsych.data.addProperties()` function. Here is an example:

```javascript
// generate a random subject ID with 15 characters
var subject_id = jsPsych.randomization.randomID(15);

// pick a random condition for the subject at the start of the experiment
var condition_assignment = jsPsych.randomization.sampleWithoutReplacement(['conditionA', 'conditionB', 'conditionC'], 1)[0];

// record the condition assignment in the jsPsych data
// this adds a property called 'subject' and a property called 'condition' to every trial
jsPsych.data.addProperties({
  subject: subject_id,
  condition: condition_assignment
});
```

### Adding data to a particular trial or set of trials

Data can be added to a particular trial by setting the `data` parameter for the trial. The `data` parameter is an object of key-value pairs, and each pair is added to the data for that trial.

```js
var trial = {
  type: 'image-keyboard-response',
  stimulus: 'imgA.jpg',
  data: { image_type: 'A' }
}
```

Data declared in this way is also saved in the trials on any nested timelines:

```js
var block = {
  type: 'image-keyboard-response',
  data: { image_type: 'A' },
  timeline: [
    {stimulus: 'imgA1.jpg'},
    {stimulus: 'imgA2.jpg'}
  ]
}
```

## Aggregating and manipulating jsPsych data

When accessing the data with `jsPsych.data.get()` the returned object is a special data collection object that exposes a number of methods for aggregating and manipulating the data. The full list of methods is detailed in the [data module documentation](../core_library/jspsych-data.md).

Here are some examples of data collection manipulation.

All data generated by the image-keyboard-response plugin:
```js
var data = jsPsych.data.get().filter({trial_type: 'image-keyboard-response'});
```

All data generated by the categorize-image plugin with a correct response:
```js
var data = jsPsych.data.get().filter({trial_type: 'categorize-image', correct: true});
```

All data with a response time between 100 and 500ms:
```js
var data = jsPsych.data.get().filterCustom(function(x){ return x.rt >= 100 && x.rt <=500 });
```

Applying filters consecutively to get all trials from a particular plugin with a response time above 100ms:
```js
var data = jsPsych.data.get().filter({trial_type: 'image-keyboard-response'}).filterCustom(function(x){ return x.rt > 100; });
```

Getting the data from the last n trials:
```js
var n = 3;
var data = jsPsych.data.get().last(n);
```

Getting the data from the last n trials with a correct response:
```js
var n = 3;
var data = jsPsych.data.get().filter({correct: true}).last(n);
```

Getting the data from the first n trials:
```js
var n = 3;
var data = jsPsych.data.get().first(n);
```

Counting the number of trials in a data collection:
```js
var count = jsPsych.data.get().filter({correct: true}).count();
```

Selecting all of the response times from a data collection:
```js
var response_times = jsPsych.data.get().select('rt');
```

Calculating various descriptive statistics on the response times in a data collection:

```js
jsPsych.data.get().select('rt').mean();
jsPsych.data.get().select('rt').sum();
jsPsych.data.get().select('rt').min();
jsPsych.data.get().select('rt').max();
jsPsych.data.get().select('rt').variance();
jsPsych.data.get().select('rt').sd();
jsPsych.data.get().select('rt').median();
jsPsych.data.get().select('rt').count();
```

## Storing data permanently as a file

This is one of the simplest methods for saving jsPsych data on the server that is running the experiment. It involves a short PHP script and a few lines of JavaScript code. This method will save each participant's data as a CSV file on the server. **This method will only work if you are running on a web server with PHP installed, or a local server with PHP (e.g., [XAMPP](https://www.apachefriends.org/index.html)).**

This method uses a simple PHP script to write files to the server:

```php
<?php
// the $_POST[] array will contain the passed in filename and filedata
// the directory "data" must be writable by the server
$filename = "data/".$_POST['filename'];
$data = $_POST['filedata'];
// write the file to disk
file_put_contents($filename, $data);
?>
```

The `file_put_contents($filename, $data)` method requires permission to write new files. An easy way to solve this is to create a directory on the server that will store the data and use the chmod command to give all users write permission to that directory. In the above example, I prepend the directory `data/` to the filename, and that directory is writable.

To use the PHP script, the JavaScript that runs jsPsych needs to send the `filename` and `filedata` information. This is done through an [AJAX](http://www.w3schools.com/xml/ajax_intro.asp) call.

```javascript
function saveData(name, data){
  var xhr = new XMLHttpRequest();
  xhr.open('POST', 'write_data.php'); // 'write_data.php' is the path to the php file described above.
  xhr.setRequestHeader('Content-Type', 'application/json');
  xhr.send(JSON.stringify({filename: name, filedata: data}));
}

// call the saveData function after the experiment is over
jsPsych.init({
   // code to define the experiment structure would go here...
   on_finish: function(){ saveData("experiment_data.csv", jsPsych.data.get().csv()); }
});
```

To use this in an actual experiment, it would be important to tie the filename to some unique identifier like a subject number. Otherwise the file may be overwritten by collecting new data.

## Storing data permanently in a MySQL database

The ideal solution for storing data generated by jsPsych is to write it to a database.

There are dozens of database options. MySQL is one of the most popular [relational databases](http://en.wikipedia.org/wiki/Relational_database), is free to use, and relatively easy [to install](https://www.google.com/search?q=how+to+install+mysql). This code will assume that you have a MySQL database installed on your server that is hosting the jsPsych experiment, and that your server is able to execute PHP code. If you are trying to run on a local machine, you'll need to install a local server environment like [XAMPP](https://www.apachefriends.org/index.html).

You'll need two PHP scripts. The first is a configuration file for your database. Save it as `database_config.php` on your server. Within this file are configuration options for the database. You'll need to change these according to how you have configured your MySQL installation.

```php
<?php
  $servername = "localhost";
  $port = 3306;
  $username = "username";
  $password = "password";
  $dbname = "database";
  $table = "tablename";
?>
```

The second PHP file will write data to the database. This script reads the database to discover what columns are in the table, and then only allows data to be entered in that matches those columns. This is a security feature. Save this file as `write_data.php` on your server.

```php
<?php

// this path should point to your configuration file.
include('database_config.php');

$data_array = json_decode(file_get_contents('php://input'), true);

try {
  $conn = new PDO("mysql:host=$servername;port=$port;dbname=$dbname", $username, $password);
  $conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
  // First stage is to get all column names from the table and store
  // them in $col_names array.
  $stmt = $conn->prepare("SHOW COLUMNS FROM `$table`");
  $stmt->execute();
  $col_names = array();
  while($row = $stmt->fetchColumn()) {
    $col_names[] = $row;
  }
  // Second stage is to create prepared SQL statement using the column
  // names as a guide to what values might be in the JSON.
  // If a value is missing from a particular trial, then NULL is inserted
  $sql = "INSERT INTO $table VALUES(";
  for($i = 0; $i < count($col_names); $i++){
    $name = $col_names[$i];
    $sql .= ":$name";
    if($i != count($col_names)-1){
      $sql .= ", ";
    }
  }
  $sql .= ");";
  $insertstmt = $conn->prepare($sql);
  for($i=0; $i < count($data_array); $i++){
    for($j = 0; $j < count($col_names); $j++){
      $colname = $col_names[$j];
      if(!isset($data_array[$i][$colname])){
        $insertstmt->bindValue(":$colname", null, PDO::PARAM_NULL);
      } else {
        $insertstmt->bindValue(":$colname", $data_array[$i][$colname]);
      }
    }
    $insertstmt->execute();
  }
  echo '{"success": true}';
} catch(PDOException $e) {
  echo '{"success": false, "message": ' . $e->getMessage();
}
$conn = null;
?>
```

To send the data, we use an AJAX request in JavaScript.
```JavaScript
function saveData() {
  var xhr = new XMLHttpRequest();
  xhr.open('POST', 'write_data.php'); // change 'write_data.php' to point to php script.
  xhr.setRequestHeader('Content-Type', 'application/json');
  xhr.onload = function() {
    if(xhr.status == 200){
      var response = JSON.parse(xhr.responseText);
      console.log(response.success);
    }
  };
  xhr.send(jsPsych.data.getData().json());
}
```

You can call the `saveData()` function using the `on_finish` handler for the experiment, or by using the `call-function` plugin.

```javascript
// with on_finish handler
jsPsych.init({
  on_finish: saveData
});

// with call-function plugin
timeline.push({
  type: 'call-function',
  func: saveData
});
```