Rules for the road: an evidence-based approach to understanding diagnostic test performance of point-of-care ultrasound for pediatric abdominal emergencies

I like to think there are many similarities between ultrasound machines and automobiles. Just as cars come in all shapes, sizes, functions (sports-car, minivans, pickup trucks, etc.), and price ranges, so do ultrasound machines—from the newer handheld ones now entering the market, laptop-sized ultrasound machines (which has allowed point-of-care ultrasound to rapidly diffuse into many areas of medicine), and to console-type ultrasound machines used by radiology departments [1]. Learning (and teaching someone) how to use an ultrasound machine is not so different from learning (and teaching someone) how to drive a car. Cars with automatic transmission are easier to learn to drive than those with manual transmission; driving in a large parking lot, or a rural area with no other cars on the road is easier to learn than driving in places like New York City or London. Similarly, some ultrasound applications are easier to learn than others: from fairly simple—looking at the bladder to assess urine volume prior to collection; a little more complex—a Focused Assessment with Sonography in Trauma (FAST) examination; to the very difficult—systematically finding and demonstrating a normal appendix.

We follow rules for the road in driving and when they are not followed we know on a daily basis in our emergency departments that catastrophes can occur; ultrasound in a clinician’s hands is no different—especially in those just learning to scan. Point-of-care ultrasound is a diagnostic test like any other and understanding the diagnostic test performance characteristics of ultrasound will help us stay safe on the “road to diagnosis” and here is where evidence-based medicine, or more specifically “evidence-based critical ultrasound [2]” can help us. Most discussion of errors in medicine have been related to preventing treatment-related harm, with little attention given to preventing diagnostic errors and misdiagnosis-related harm which are more common than harm to patients from medication-related errors [3]. When we look at the sensitivities and specificities for ultrasound to diagnose various abdominal conditions afflicting children such as free fluid in the abdomen in pediatric trauma [4, 5], pyloric stenosis [6–8], intussusception [9, 10], and appendicitis [11, 12], in general, the specificities are higher (usually >90%) than the sensitivities (Table 1). How does it help us to know that a diagnostic test tends to have higher specificity than sensitivity?

Table 1 Test characteristics of ultrasound for pediatric abdominal applications

In David Sackett’s evidence-based medicine book [13], there is a handy mnemonic: A high Specificity test, a Positive result effectively “rules IN” disease (SpPIN). For tests with a high Sensitivity, a Negative result effectively “rules OUT” disease (SnNOUT). In general, ultrasound with its higher specificities relative to sensitivities is more accurate for “ruling in” disease, than “ruling out” disease (with very few exceptions). Thus, ultrasound is more reliable in identifying pathology (e.g., free fluid on a FAST exam, a shadowing appendicolith, the classic target of intussuception, etc.) than in completely ruling out the presence of disease or abnormality (e.g., absence of free fluid on FAST exam, normal muscle wall thickness of a vomiting infant’s pylorus, etc.).

However, sensitivity and specificity are traditional measures of diagnostic test utility that can be difficult to interpret, particularly when the values are not very high or not very low. Furthermore, they do not directly address the probability of disease in an individual patient after a diagnostic test has been applied. Likelihood ratios (LRs) are a measure of the predictive power of a test and may be more intuitive to understand than sensitivity and specificity. They are defined as the likelihood that a given test result would be found in a patient with a target disorder, relative to the likelihood of the same test result occurring in a patient without the target disorder. LRs can be derived for any diagnostic test, whether it is a clinical sign or symptom, laboratory test, or diagnostic imaging modality like ultrasound (presented in Table 1). LRs can be applied to a pre-test probability of disease in order to generate a post-test probability of disease. The direction and magnitude of change in probability from pre-test to post-test are determined by a test’s performance characteristics, which are incorporated by the likelihood ratio. Diagnostic tests with LRs >10 or <0.1 significantly alter pre-test probabilities, LRs >5 or <0.2 have moderate effects on pre-test probabilities, while LRs closer to 1 have little effect. Looking at Table 1, likelihood ratios for positive ultrasound results tend to increase pre-test probabilities more than it decreases pre-test probabilities with negative ultrasound results. If a resulting post-test probability of disease is high enough to make the diagnosis certain, it facilitates the clinician’s decision to treat. However, if the post-test probability is not high enough to confirm the diagnosis, a clinician may pursue further diagnostic testing. Detailed discussions of likelihood ratios are available elsewhere [13, 14].

Thus, the prime rule for the road when using point-of-care ultrasound: do not rely on ultrasound alone to “rule out” disease. With respect to likelihood ratios for negative ultrasound results, the post-test probability will probably not be low enough to confidently exclude disease. This will keep you from sending home the intussusception, pyloric stenosis or appendicitis solely based on a “negative” ultrasound. The worst thing that will happen is a confirmatory test or someone will tell you that the child you admitted did not have “x”, “y” or “z”. If you have an inconclusive or “negative” ultrasound (e.g., you did not see free fluid on a FAST exam, target sign of intussusception, or a big fat >6 mm appendicitis that hurts the patient on sono-palpation, etc.), take a step back, reassess the clinical picture, and get that confirmatory study or continue to observe your pediatric patient. An unaddressed area of research is the effect of serial ultrasound exams on overall diagnostic accuracy (either by same operator or by a different one) as an advantage of point-of-care ultrasound is that it is easily repeatable with abnormal findings highly reproducible. Most likely, with an abnormal or positive ultrasound result—you have probably clinched the diagnosis with a sufficiently high post-test probability that allows you to move forward with further evaluation if necessary, treatment and then onto your next emergency pediatric patient.

It is recognized that we need to minimize the amount of ionizing radiation that children receive from imaging studies [15], and with ongoing research, we can be hopeful that point-of-care ultrasound may have a role in this. In the coming years, we will likely see research studies that will investigate not just the accuracy of clinician-performed point-of-care ultrasound to diagnose various pediatric abdominal emergencies and other pathologies, but also to identify where preventable diagnostic errors may arise and to develop strategies to avoid them—whether it involves standardized scanning techniques, improved ultrasound machine functionality, or operator training factors. Just as road-traffic safety is an important discipline that has led to use of air-bags, car seats for infants, and seat-belt laws, efforts to study diagnostic errors related to point-of-care ultrasound use can help us avoid misdiagnosis-related harm in our patients. Bottom line: in general, think of point-of-care ultrasound as a “rule-in” diagnostic test and not a tool to “rule-out” disease.