Key takeaways:
- The replicability crisis in empirical research is a significant issue, affecting even highly respected publications and researchers.
- The Replicability Index (R-Index) is a useful tool for estimating the replicability of research findings.
- The empirical basis for many psychological claims in Kahneman's 'Thinking, Fast and Slow' is shaky, with only 12% of all tests conducted yielding significant results.
The replicability crisis in empirical research has been highlighted by various high-profile cases, such as the faking of results by social psychologist Diederik Stapel and the publication of seemingly credible scientific evidence for paranormal phenomena in a respected journal. In 2011, Nobel Laureate Daniel Kahneman's book 'Thinking, Fast and Slow' reviewed his own work and many findings from social psychology, but it is likely that some of his chapters would be different if written just a few years later.
The Replicability Index (R-Index) is a valuable tool for estimating the replicability of research findings. It is based on two statistics: the percentage of significant results and the conversion of exact p-values into estimates of statistical power. The R-Index is a simple correction for the inflation that is introduced by selecting significant results, subtracting the difference between the discovery rate and the average power estimate from the mean power estimate.
In the case of 'Thinking, Fast and Slow', the empirical basis for many psychological claims is shaky. The book focuses on discoveries, with most test-statistics being significant with p < .05. However, the distribution of z-scores shows that these significant results were selected from a larger set of tests that produced non-significant results. The z-curve estimate is that the significant results are only 12% of all tests that were conducted, indicating a problem with the replicability of these findings.
In conclusion, the replicability crisis in empirical research is a significant issue that affects even highly respected publications and researchers. Tools like the R-Index can help estimate the replicability of research findings, but it is clear that the empirical basis for many psychological claims in 'Thinking, Fast and Slow' is shaky. This highlights the need for a more rigorous approach to empirical research and the importance of replication studies.