A model for assessing the sensitivity and specificity of tests subject to selection bias: Application to exercise radionuclide ventriculography for diagnosis of coronary artery disease

doi:10.1016/0021-9681(86)90119-0

Journal of Chronic Diseases

Volume 39, Issue 5, 1986, Pages 343-355

Journal of Chronic D...

https://doi.org/10.1016/0021-9681(86)90119-0 Get rights and content

Abstract

A probabilistic model was developed which allows one to estimate sensitivity and specificity of diagnostic tests for coronary artery disease without reference to angiography. The feasibility of the model was evaluated first in a series of computer simulations, and the model was then applied to the assessment of ejection fraction in 933 patients without prior myocardial infarction who underwent exercise radionuclide ventriculography. In 196 patients who were referred to angiography, the conventional abnormal ejection fraction criterion—an absolute rise of less than 0.05 with exercise—had a sensitivity of 79% and a specificity of 68% when referenced to coronary angiography. In 737 patients who were not referred for angiography and who were analyzed instead by our probabilistic model, sensitivity was 63% (p = 0.004 compared to that in the 119 angiographically diseases patients) and specificity was 79% (p = 0.036 compared to that in the 77 angiographic normals). Both the higher sensitivity and lower specificity in the catheterized patients are consistent with a preferential referral of positive test responders to angiography, and of negative test responders away from angiography. The distortion of test sensitivity and specificity which results from this selection bias can be circumvented by substituting a probabilistic estimate of disease for conventional angiographic ascertainment.

References (40)

GA Diamond
A clinically relevant classification of chest discomfort
J Am Coll Cardiol
(1983)
GA Diamond et al.
Computer-assisted diagnosis in the noninvasive evaluation of patients with suspected coronary artery disease
J Am Coll Cardiol
(1983)
PS Greenberg et al.
Comparison of the multivariate analysis and CADENZA systems for determination of the probability of coronary artery disease
Am J Cardiol
(1984)
JA Melin et al.
Alternative diagnostic strategies for coronary artery disease in women: demonstration of the usefulness and efficiency of probability analysis
Circulation
(1985)
J Neter et al.
Applied Linear Statistical Models
JS Borer et al.
Real-time radionuclide cineangiography in the noninvasive evaluation of global and regional left ventricular function at rest and during exercise in patients with coronary artery disease
N Engl J Med
(1977)
JS Borer et al.
Sensitivity, specificity, and predictive accuracy of radionuclide cineangiography during exercise in patients with coronary artery disease: comparison with exercise electrocardiography
Circulation
(1979)
TD Christopher et al.
Bayesian analysis of data from radionuclide angiocardiograms for diagnosis of coronary artery disease
Circulation
(1984)
E Russek et al.
The effect of assuming independence in applying Bayes' theorem to risk estimation and classification in diagnosis
Comput Biomed Res
(1983)
CB Begg et al.
Assessment of diagnostic tests when disease verification is subject to selection bias
Biometrics
(1983)

DF Ransohoff et al.

Problems of spectrum and bias in evaluating the efficacy of diagnostic tests

N Engl J Med

(1978)

A Rozanski et al.

The declining specificity of exercise radionuclide ventriculography

N Engl J Med

(1983)

T Pearson et al.

Selection bias in persons undergoing coronary arteriography

Am J Epidemiol

(1982)

GA Diamond

An improbable criterion of normality

Circulation

(1982)

A Rozanski et al.

Alternative referent standards for cardiac normality: implications for diagnostic testing

Ann Intern Med

(1984)

BR Chaitman et al.

Angiographic prevalence of high-risk coronary artery disease in patient subsets (CASS)

Circulation

(1981)

A Rozanski et al.

Improvement of resting myocardial asynergy with cessation of bicycle exercise

Circulation

(1983)

AH Hakki et al.

Prediction of maximal heart rates in men and women

Cardiovasc Rev Rep

(1983)

GA Diamond et al.

Analysis of probability as an aid in the clinical diagnosis of coronary artery disease

N Engl J Med

(1979)

GA Diamond et al.

Application of conditional probability analysis to the clinical diagnosis of coronary artery disease

J Clin Invest

(1980)

Cited by (91)

Accuracy of photogrammetry for detecting adolescent idiopathic scoliosis progression
2019, Spine Journal
Citation Excerpt :
On the other hand, they believe that the automation of the photogrammetric curve measurement, along with the development of specific software for this task is feasible and, hypothetically, may reduce the time necessary to obtain the curvature angle. Another limitation in this study is the possible violation of the assumption of independent observation of Bayesian statistics through which the concepts of sensitivity and specificity were developed [27,28], based upon the fact that the study was done at a tertiary referral center. The data collected at a tertiary center may introduce selection bias and, therefore, the conclusions of this study may not be generalizable.
The gold standard method of monitoring the evolution of scoliosis has been serial standing, posteroanterior, full-length spine radiographs with curvature measurements using the Cobb method. However, over the course of follow-up, patients can receive high radiation doses. Various studies have shown that repeated exposure to radiation in children and adolescents can be harmful to their health.
To determine the accuracy of photogrammetry in evaluating the progression of adolescent idiopathic scoliosis in comparison with radiography.
Diagnostic study.
Ninety adolescents subjected to radiographic follow-up of idiopathic scoliosis.
The angle of scoliotic curvature was measured using the Cobb radiographic method and photogrammetry. An increase of 5° or more between two radiographic exams was considered a progression of the curvature and was defined as the standard for calculations of sensitivity, specificity, predictive value, and accuracy of the photogrammetric method for measuring scoliosis progression.
Patients were subjected to radiographic and photogrammetric exams concomitantly and were reevaluated after an average of 8.6 months. The exams were analyzed separately and independently by two examiners for progression of scoliosis.
The measurements of the curves at the beginning of the study were 39.5±16.7° and 39.5±14.3° for radiographic and photogrammetric exams, respectively (p=1.0). At the end of the study, the measurements of the curves were 40.2±16.2° and 41.3±15.1° for the radiographic and photogrammetric exams, respectively (p=.310). The photogrammetric method had an accuracy of 89% (Confidence interval [CI] 95%=82.5–95.5) for the detection of scoliosis progression, with a sensitivity of 94.4% (CI 95%=89.6–99.2), a specificity of 86.7% (CI 95%=79.7–93.7), a positive predictive value of 75.5% (CI 95%=66.6–84.4), a negative predictive value of 97.2% (CI 95%=93.8–100), and a Kappa index of 0.75 (CI 95%=66.1–83.9). The interclass correlation coefficient between the two methods was 0.74 (CI 95%=0.65–0.81; p=0).
The photogrammetric method showed good performance for detecting the progression of adolescent idiopathic scoliosis in comparison with the radiographic exam method.
How to Use an Article About a Diagnostic Test
2008, Journal of Urology
Citation Excerpt :
Nevertheless, these patients appear similar to those whom you evaluate and they appear to represent a reasonable spectrum in terms of bladder cancer risk. An assessment of diagnostic accuracy is susceptible to bias if the index test results influence the decision to perform the reference test (verification bias).19–25 Verification bias is more common when the reference standard is an invasive procedure.24
Urologists frequently confront diagnostic dilemmas, prompting them to select, perform and interpret additional diagnostic tests. Before applying a given diagnostic test the user should ascertain that the chosen test would indeed help decide whether the patient has a particular target condition. In this article in the Users' Guide to the Urological Literature series we illustrate the guiding principles of how to critically appraise a diagnostic test, interpret its results and apply its findings to the care of an individual patient.
The guiding principles of how to evaluate a diagnostic test are introduced in the setting of a clinical scenario. We propose a stepwise approach that addresses the question of whether the study results are likely to be valid, what the results are and whether these results would help urologists with the treatment of their individual patients.
Some of the issues urologists should consider when assessing the validity of a diagnostic test study are how the authors assembled the study population, whether they used blinding to minimize bias and whether they used an appropriate reference standard in all patients to determine the presence or absence of the target disorder. Urologists should next evaluate the properties of the diagnostic test that indicate the direction and magnitude of change in the probability of disease for a particular test result. Finally, urologists should ask a series of questions to understand how the diagnostic test may impact the care of their patients.
Application of the guides presented in this article will allow urologists to critically appraise studies of diagnostic tests. Determining the study validity, understanding the study results and assessing the applicability to patient care are 3 fundamental steps toward an evidence-based approach to choosing and interpreting diagnostic tests.
Cardiac CT Angiography (CTA) and Nuclear Myocardial Perfusion Imaging (MPI)-A Comparison in Detecting Significant Coronary Artery Disease
2007, Academic Radiology
Citation Excerpt :
Patients with a negative noninvasive test are not routinely referred for coronary angiography, therefore creating a referral bias. The effect of referral bias on sensitivity and specificity has been shown in studies with exercise radionuclide ventriculography (29) and exercise thallium imaging (30). Caution must be used in applying these results in a higher risk population to overall patients likely to have coronary CTA or MPI, who are a much lower risk group.
Endowed with sufficient diagnostic accuracy, electron beam computed tomography angiography (CTA) is being increasingly used to evaluate coronary arteries. However, data on direct comparisons with nuclear myocardial perfusion studies are limited. In this study, we sought to compare the accuracies of CTA and myocardial perfusion imaging (MPI) for identifying symptomatic patients with hemodynamically significant obstructive coronary artery disease (CAD).
In a single-center study, symptomatic outpatients who were scheduled for cardiac catheterization were prospectively enrolled. Only patients with exertional angina or dyspnea were included. After fulfilling the inclusion criteria, 30 patients were enrolled in the study (mean age 54 ± 9 years and 70% males). Patients underwent MPI, CTA including coronary artery calcification (CAC) measure, and invasive coronary angiography for evaluation of obstructive coronary artery disease. Significant CAD was defined as >50% left main artery stenosis or >70% stenosis of any other epicardial vessel by invasive angiography. The sensitivities, specificities and predictive values of MPI, CAC, and CTA were analyzed per patient
CTA demonstrated significant higher sensitivity than MPI (95% vs. 81%, P < .05). CTA demonstrated significantly higher specificity than both MPI (89% versus 78%, P = .04) and CAC (56%, P = .002). CTA also performed better in a per-vessel analysis (sensitivity 94%, specificity 96%) than both nuclear and CAC. There were no significant differences between the sensitivities and specificities of MPI and CAC.
CTA accurately detects obstructive CAD in symptomatic patients and may be more accurate than MPI or CAC assessment. Larger studies in a more diverse population are needed.
Diagnostic accuracy of myocardial perfusion imaging in a study population without post-test referral bias
2005, Journal of Nuclear Cardiology
Most previous studies on the accuracy of myocardial perfusion imaging (MPI) are hampered by post-test referral bias, in that referral for coronary angiography was influenced by the MPI result. In this way, patients with a normal MPI result less frequently underwent catheterization, a tendency supposed to cause an underestimation of test specificity and an overestimation of test sensitivity.
MPI by use of a gated dual-isotope protocol was undertaken before angiography in 357 patients referred for angiography for suspected stable angina pectoris. The MPI reports were kept secret to prevent post-test referral bias. The MPI study was normal in 215 patients (60%) and showed reversible perfusion abnormalities in 118 (33%) and fixed defects in 24 (7%). Angiography was normal in 231 patients (65%) and revealed 1 or more significant stenoses in 126 (35%). With angiography as the reference, the sensitivity and specificity of MPI for detecting significant coronary artery stenosis were 75% and 79%, respectively.
In this prospective study without post-test referral bias, we found a lower sensitivity and slightly higher specificity than in studies with post-test referral bias. The imperfect accuracy may reflect differences between anatomic and physiologic imaging.
Caveat emptor: The treachery of work-up bias
2004, Journal of Thoracic and Cardiovascular Surgery
Citation Excerpt :
We would suggest that as a minimum, such articles acknowledge that accuracy of testing has not been corrected for bias. Perhaps in the face of the rampant misinterpretation of test accuracy, whenever it is possible to estimate magnitude of the bias, correction of referent values for bias should be required.12,13 If the reader's appropriate profound disillusion with diagnostic testing has now reached the level of despair, we suggest that just because a test performs poorly diagnostically (once work-up bias is accounted for) does not necessarily mean it is useless clinically.
Diagnostic research on routine care data: Prospects and problems
2003, Journal of Clinical Epidemiology
A diagnosis in practice is a sequential process starting with a patient with a particular set of signs and symptoms. To serve practice, diagnostic research should aim to quantify the added value of a test to clinical information that is commonly available before the test will be applied. Routine care databases commonly include all documented patient information, and therefore seem to be suitable to quantify a tests' added value to prior information. It is well known, however, that retrospective use of routine care data in diagnostic research may cause various methodologic problems. But, given the increased attention of electronic patient records including data from routine patient care, we believe it is time to reconsider these problems. We discuss four problems related to routine care databases. First, most databases do not label patients by their symptoms or signs but by their final diagnosis. Second, in routine care the diagnostic workup of a patient is by definition determined by previous diagnostic (test) results. Therefore, routinely documented data are subject to so-called workup bias. Third, in practice, the reference test is always interpreted with knowledge of the preceding test information, such that in scientific studies using routine data the diagnostic value of a test under evaluation is commonly overestimated. Fourth, routinely documented databases are likely to suffer from missing data. Per problem we discuss methods that are presently available and may (partly) overcome each problem. All this could contribute to more frequent and appropriate use of routine care data in diagnostic research. The discussed methods to overcome the above problems may well be similarly useful to prospective diagnostic studies.

View all citing articles on Scopus

^☆: This work was supported in part by a Specialized Center of Research (SCOR) grant from the National Institutes of Health (HL-17651) and a grant-in-aid from the American Heart Association, Greater Los Angeles Affiliate.

View full text

Variance and dissentA model for assessing the sensitivity and specificity of tests subject to selection bias: Application to exercise radionuclide ventriculography for diagnosis of coronary artery disease☆

Abstract

J Am Coll Cardiol

J Am Coll Cardiol

Am J Cardiol

Circulation

N Engl J Med

Circulation

Circulation

Comput Biomed Res

Biometrics

Problems of spectrum and bias in evaluating the efficacy of diagnostic tests

N Engl J Med

The declining specificity of exercise radionuclide ventriculography

N Engl J Med

Selection bias in persons undergoing coronary arteriography

Am J Epidemiol

An improbable criterion of normality

Circulation

Alternative referent standards for cardiac normality: implications for diagnostic testing

Ann Intern Med

Angiographic prevalence of high-risk coronary artery disease in patient subsets (CASS)

Circulation

Improvement of resting myocardial asynergy with cessation of bicycle exercise

Circulation

Prediction of maximal heart rates in men and women

Cardiovasc Rev Rep

Analysis of probability as an aid in the clinical diagnosis of coronary artery disease

N Engl J Med

Application of conditional probability analysis to the clinical diagnosis of coronary artery disease

J Clin Invest

Variance and dissent
A model for assessing the sensitivity and specificity of tests subject to selection bias: Application to exercise radionuclide ventriculography for diagnosis of coronary artery disease☆