Evaluation Review, Ahead of Print.
This paper describes how a multi-armed randomized experiment was used to test multiple variants of a behaviorally informed marketing strategy. In particular, we tested whether specific behavioral messages could be used to increase demand for a safety consultation service offered by the U.S. Occupational Safety and Health Administration. Our experiment used a partial factorial design with 19 study arms and a very large research sample—97,182 establishments—to test the impact of various message, formats, and delivery modes compared with an existing (not behaviorally informed) informational brochure and a no-marketing counterfactual. A secondary research goal was to predict the impact of the most successful marketing strategy (i.e., combination of message, format, and mode) so that OSHA would know what to anticipate if that strategy were implemented at scale. We used two related (but distinct) methods to address these two goals. Both begin with a common mixed (i.e., fixed and random effects) ANOVA model. We addressed the first research goal primarily from the fixed effects; we addressed the second research goal by calculating best linear unbiased predictions (BLUPs) from the full mixed model, where the BLUP involves “shrinkage” as in empirical Bayes (EB) approaches. Marketing via brochures was effective overall, nearly doubling the rate of requests for services. However, the behaviorally informed materials performed no better than OSHA’s existing informational brochure. This study also highlights the conditions under which a factorial design can be used to efficiently address questions about which of several program variants are most effective.