Reliable and objective assessment of clinical decision-making skills has been a long-standing goal in occupational testing in the allied health professions. With this goal in mind, the key features problem (KFP) format was developed which elicits targeted decisions about key features of clinical scenarios. To build on a small body of empirical evidence evaluating their efficacy, this study evaluates whether KFPs successfully assess higher order cognitive processes. Analysis of objective data (item length and difficulty, item performance, and response times) and subjective data (expert ratings of cognitive complexity) supported the proposition that KFPs tend to be more cognitively complex than conventional multiple-choice questions. Not only were they rated as more complex, but this complexity accounted for some of the increase in time spent responding to these items. Results support the use of KFPs in standardized assessments for measuring higher order cognitive processes such as clinical decision making.