Journal of Personality and Social Psychology, Vol 127(1), Jul 2024, 31-57; doi:10.1037/pspa0000391
We introduce the test difficulty concept from classical test theory to tackle the issue of low predictive power of implicit association tests (IATs). Following classical test theory, we argue that IATs of moderate difficulty (defined as mean IAT scores of zero) have more predictive power than IATs of extreme difficulties (defined as mean IAT scores deviating strongly from zero). Furthermore, we assume this relationship to be mediated by the true-score variance in IAT scores, with moderate difficulty resulting in more true-score variance. To test our hypotheses, we used nonexperimental (Studies 1 and 2) and experimental designs (Study 3). In Studies 1 and 2, we compared IATs of different test difficulties with regard to their ability to predict direct attitude measures, drawing on the Attitudes, Identities, and Individual Differences study. In Study 1, a subset of 95 attitude IATs (n = 127,259) was analyzed using multilevel structural equation models. As expected, IAT test difficulty strongly moderated the predictive power of IATs, and this effect was mediated by IAT true-score variance. In Study 2, we replicated the results with the same analyses but a different subset of 95 identity IATs (n = 43,745). In Study 3, we experimentally manipulated the IAT test difficulty. In total, three IATs (n = 480) were analyzed using multigroup structural equation models. Again, the IAT closer to moderate difficulty had more true-score variance and predictive power than the IATs of extreme difficulty. Accordingly, for correlational research, we recommend developing moderately difficult IATs to maximize IAT true-score variance and provide suggestions on how to achieve that. (PsycInfo Database Record (c) 2024 APA, all rights reserved)