Psychological Methods, Vol 30(1), Feb 2025, 40-59; doi:10.1037/met0000525
The current guidelines for estimating reliability recommend using two omega combinations in multidimensional data. One omega is for factor analysis (FA) reliability estimators, and the other omega is for omega hierarchical estimators (i.e., ωh). This study challenges these guidelines. Specifically, the following three questions are asked: (a) Do FA reliability estimators outperform non-FA reliability estimators? (b) Is it always desirable to estimate ωh? (c) What are the best reliability and ωh estimators? This study addresses these issues through a Monte Carlo simulation of reliability and ωh estimators. The conclusions are given as follows. First, the performance differences among most reliability estimators are small, and the performance of FA estimators is comparable to that of non-FA estimators. However, the current, most-recommended estimators, that is, estimators based on the bifactor model and exploratory factor analysis, tend to overestimate reliability. Second, the accuracy of ωh estimators is much lower than that of reliability estimators, so we should perform ωh estimation selectively only on data that meet several requirements. Third, exploratory bifactor analysis is more accurate than confirmatory bifactor analysis only in the presence of cross-loading; otherwise, exploratory bifactor analysis is less accurate than confirmatory bifactor analysis. Fourth, techniques known to improve the Schmid-Leiman (SL) transformation are not superior to SL transformation but have different advantages. This study provides an R Shiny app that allows users to obtain multidimensional reliability and ωh estimates with a few mouse clicks. (PsycInfo Database Record (c) 2025 APA, all rights reserved)