We tested the utility of two standardized measures of receptive skills in American Sign Language (ASL) in hearing adults who are novice signers: the ASL Comprehension Test (ASL-CT; Hauser, P. C., Paludneviciene, R., Riddle, W., Kurz, K. B., Emmorey, K., & Contreras, J. (2016). American Sign Language Comprehension Test: A tool for sign language researchers. Journal of Deaf Studies and Deaf Education, 21(1), 64–69. https://doi.org/10.1093/deafed/env051) and the ASL Receptive Skills Test (ASL-RST; Allen, T. E., & Enns, C. (2013). A psychometric study of the ASL Receptive Skills Test when administered to deaf 3-, 4-, and 5-year-old children. Sign Language Studies, 14(1), 58–79. https://muse.jhu.edu/article/581868; Enns, C. J., & Herman, R. C. (2011). Adapting the Assessing British Sign Language Development: Receptive Skills Test into American Sign Language. Journal of Deaf Studies and Deaf Education, 16(3), 362–374. https://doi.org/10.1093/deafed/enr004). We also administered these tests to hearing nonsigners, with the goal of identifying items where the correct answer can be readily deduced through iconicity and/or test-taking skills, even by people with no prior ASL knowledge. We predicted that removing such items would strengthen the correlation between test score and ASL experience (as measured by semesters of instruction). We found that this relationship was stronger for the ASL-CT than the ASL-RST in this population, but still weaker than previously reported, and not substantially improved by any item selection strategy that we employed.