Background:
Emphasis is increasingly being placed on the monitoring of clinical outcomes for health careproviders. Funnel plots have become an increasingly popular graphical methodology used toidentify potential outliers. It is assumed that a provider only displaying expected randomvariation (i.e. ‘in-control’) will fall outside a control limit with a known probability. Inreality, the discrete count nature of these data, and the differing methods, can lead to trueprobabilities quite different from the nominal value. This paper investigates the trueprobability of an ‘in control’ provider falling outside control limits for the StandardisedMortality Ratio (SMR).
Methods:
The true probabilities of an ‘in control’ provider falling outside control limits for the SMRwere calculated and compared for three commonly used limits: Wald confidence interval;’exact’ confidence interval; probability-based prediction interval.
Results:
The probability of falling above the upper limit, or below the lower limit, often varied greatlyfrom the nominal value. This was particularly apparent when there were a small number ofexpected events: for expected events [less than or equal to]50 the median probability of an ‘in-control’ providerfalling above the upper 95% limit was 0.0301 (Wald), 0.0121 (‘exact’), 0.0201 (prediction).
Conclusions:
It is important to understand the properties and probability of being identified as an outlier byeach of these different methods to aid the correct identification of poorly performing healthcare providers. The limits obtained using probability-based prediction limits have the mostintuitive interpretation and their properties can be defined a priori. Funnel plot control limitsfor the SMR should not be based on confidence intervals.