An influential 2002 study co-authored by Israeli behavioral economist Prof. Dan Ariely has been retracted by the journal Psychological Science after a failed replication and a fresh analysis raised serious questions about the reliability of the underlying data.
The paper, written by Ariely and Prof. Klaus Wertenbroch, examined procrastination and whether different kinds of deadlines improve performance. It concluded that people performed better when given evenly spaced external deadlines than when they set deadlines themselves or faced a single final deadline. The study became widely influential, appearing on reading lists in psychology and economics courses and accumulating more than 2,100 citations, according to Data Colada.
Psychological Science formally retracted the article on September 2 after a replication study by Prof. Kyle Hyndman and Prof. Alberto Bisin failed to reproduce the original findings. The journal said questions had emerged about the data attributed to the original study and that its authenticity and completeness could not be confirmed. Both Ariely and Wertenbroch agreed to the retraction.
The controversy intensified after Uri Simonsohn, Joe Simmons and Leif Nelson, the researchers behind the Data Colada website, examined data files associated with the original experiments. In a post published August 31, they identified what they described as several major red flags and concluded that they could not find an innocent explanation for the combined anomalies. They said they believed the data from one of the studies had been “severely tampered with or fabricated” to produce the reported results.
(Photo: datacolada.org)
Their analysis focused in part on an experiment involving 60 participants who completed three proofreading tasks. The original paper reported that participants assigned weekly deadlines made an average of 136.1 corrections, compared with 71.1 among those who had a single final deadline. Data Colada said the size of that difference was unusually large for a psychological experiment of this kind.
Another concern involved duplicated observations. According to Data Colada, 18 of the 20 participants in the single-deadline group had another participant with exactly the same values across all three proofreading tasks, a pattern that did not appear in the other two groups.
The researchers also pointed to statistical relationships they considered implausible. Performance across three similar proofreading assignments showed almost no correlation in the original dataset, while the replication produced much stronger correlations. Similar discrepancies appeared in participants’ subjective assessments of the tasks.
Data Colada also examined the way participants reported how long they spent on the assignments. Only 11.7% of the reported times in the original data were round numbers, compared with 85% in the replication, despite the expectation that people estimating time would frequently report figures such as 30, 45 or 60 minutes.
(Photo: datacolada.org)
The data came to renewed attention during Hyndman and Bisin’s replication work. Hyndman had received three Excel files in 2006 from Ariely’s email address that appeared to contain data needed to reproduce the original results. Data Colada said the files’ metadata indicated they had last been saved by a user named “Dan Ariely.” Ariely has previously said it was possible the files were not the original datasets.
The replication itself found little evidence for the original study’s main result. Hyndman and Bisin reported that changing the deadline structure had a negligible effect on the performance measures they examined and that evenly spaced externally imposed deadlines did not stand out as particularly effective in reducing procrastination.
In its formal retraction notice, Psychological Science said the replication had raised questions about the original data, while the Data Colada investigation had further called the findings into question. The journal’s editor concluded that the reliability of the paper could no longer be supported.
(Photo: datacolada.org)
Ariely addressed the concerns before the retraction. In a statement cited in the material reviewed for the article, he said he had learned that the data underlying the study contained “serious anomalies.” He said the documentation still available to him was insufficient to answer the questions raised, and that after more than two decades he could not rely on his memory of the experiments to reconstruct what happened.
Ariely said his responsibility now was to help ensure the scientific record is accurate and that he and Wertenbroch were cooperating with the journal’s review and retraction process.
The retraction does not itself establish who, if anyone, altered the data. The journal’s notice says the authenticity and completeness of the dataset cannot be confirmed and that confidence in the paper’s findings has been lost, while the stronger claim that the data were manipulated or fabricated comes from Data Colada’s analysis.