Current Directions in Psychological Science, Ahead of Print.
Although individual-difference studies have been invaluable in several domains of psychology, there has been less success in cognitive domains using experimental tasks. The problem is often called one of reliability: Individual differences in cognitive tasks, especially cognitive-control tasks, seem too unreliable. In this article, we use the language of hierarchical models to define a novel reliability measure—a signal-to-noise ratio—that reflects the nature of tasks alone without recourse to sample sizes. Signal-to-noise reliability may be used to plan appropriately powered studies as well as understand the cause of low correlations across tasks should they occur. Although signal-to-noise reliability is motivated by hierarchical models, it may be estimated from a simple calculation using straightforward summary statistics.