The aim of this study was to examine the Autism Diagnostic Observation Schedule (ADOS) as it is commonly used in clinical practice. ADOS classifications were compared to final diagnoses given by a multidisciplinary team to 584 children referred for evaluation for possible autism spectrum disorder (ASD) at the Cincinnati Children’s Hospital Medical Center. A total of 177 children were evaluated with a Module 1 (87 No Words), 198 with a Module 2 (90 < 5 years) and 209 with a Module 3. Of these, 142 (26%) were diagnosed with autism, 185 (32%) with non-autism ASD, and 257 (44%) with non-spectrum disorders. Sensitivities were moderate to high on both original and revised algorithms, while specificities were substantially lower than those previously reported. This difference is likely attributable to the composition of the sample that included many children with a broad array of developmental and behavioral disorders. The clinical impression of the team member who administered the ADOS was critical to the accuracy of the overall diagnosis. Using numeric scores alone resulted in misclassification from false positive results. The study highlights the importance of the qualitative interactions of the ADOS activities as well as the score in diagnostic decision making.