Asthma is a phenotypically complex disease requiring nuanced data to generate clinically and scientifically robust real-world evidence. A quantitative measure of data quality is important for variables key to the research questions at hand. Using electronic health record (EHR) data, this study compared accuracy for asthma features between traditional real-world evidence approaches using structured data and advanced approaches applying artificial intelligence technologies to unstructured clinical data.
Methods:
We extracted 18 protocol-defined features from 6037 healthcare encounters among 3481 patients. Features included asthma severity subtypes, comorbidities, symptoms, findings, and procedures. We created a manual reference standard through chart abstraction, with two annotators reviewing each record. We assessed interrater reliability using Cohen’s kappa score and accuracy against the reference standard as an F1-score.
Results:
In the traditional study arm, average recall was 40.8%, precision 72.5%, and F1-score across features was 52.2%. In the advanced study arm, average recall was 95.7%, precision 93.8%, and F1-score was 94.7%. There was an absolute increase of 42.5% and a relative increase of 81.4% in the F1-score between traditional and advanced approaches. Cohen’s kappa score indicated 0.80 inter-rater reliability, reflecting a credible reference standard.
Conclusions:
Use of advanced approaches can enable high-quality real-world data sets in asthma, including granular clinical features such as disease subtypes and symptomatic outcomes. Data quality can be measured and, when high, can support generation of high-validity real-world evidence using routinely collected healthcare data.