Can AI Automatically Assess Scan Quality of Hip Ultrasound?

Featured Application

Our AI-based approach flags low-quality ultrasound hip images as inadequate for diagnosis. This would help sonographers to collect high-quality hip scans suitable for early diagnosis of DDH.

Abstract

Ultrasound images can reliably detect Developmental Dysplasia of the Hip (DDH) during early infancy. Accuracy of diagnosis depends on the scan quality, which is subjectively assessed by the sonographer during ultrasound examination. Such assessment is prone to errors and often results in poor-quality scans not being reported, risking misdiagnosis. In this paper, we propose an Artificial Intelligence (AI) technique for automatically determining scan quality. We trained a Convolutional Neural Network (CNN) to categorize 3D Ultrasound (3DUS) hip scans as ‘adequate’ or ‘inadequate’ for diagnosis. We evaluated the performance of this AI technique on two datasets—Dataset 1 (DS1) consisting of 2187 3DUS images in which each image was assessed by one reader for scan quality on a scale of 1 (lowest quality) to 5 (optimal quality) and Dataset 2 (DS2) consisting of 107 3DUS images evaluated semi-quantitatively by four readers using a 10-point scoring system. As a binary classifier (adequate/inadequate), the AI technique gave highly accurate predictions on both datasets (DS1 accuracy = 96% and DS2 accuracy = 91%) and showed high agreement with expert readings in terms of Intraclass Correlation Coefficient (ICC) and Cohen’s kappa coefficient (K). Using our AI-based approach as a screening tool during ultrasound scanning or postprocessing would ensure high scan quality and lead to more reliable ultrasound hip examination in infants.

1. Introduction

Developmental Dysplasia of the Hip (DDH) is common in infants, present in one to three per 1000 live births [1] with large variations among ethnic groups [2]. Undiagnosed DDH is a major risk factor for early hip Osteoarthritis (OA) [2,3] which is associated with a huge economic burden [4,5]. It is also seen as the underlying cause for more than ⅓ of hip replacements in adults who are less than 60 years of age [6]. If diagnosed in early infancy (<3 months), DDH can be treated by simple techniques such as bracing (e.g., Pavlik harness) [7]. Despite obvious advantages of early diagnosis [8,9], hip screening programs for infants are not common in most countries, due in large part to the high variability in scan assessment [10].

Clinically, DDH is diagnosed by physical examination including Barlow and Ortolani maneuvers which have poor sensitivity beyond the neonatal period, resulting in missed cases, particularly of mild DDH or fixed dislocations [11,12,13]. Compared to physical examination, ultrasound imaging is more reliable and more sensitive [11,12,13] and is ideal for screening programs as it is safe, easily portable and inexpensive.

During hip examination, 2D ultrasound images are usually acquired and interpreted based on the Graf technique [14] which measures the alpha angle between the ilium and acetabular roof. Alpha angles greater than 60 degrees are considered normal and those less than 43 degrees are considered severely dysplastic [14]. This technique relies heavily on obtaining a single 2D image in the Graf plane containing all landmarks such as illum, labrum, os ischium and femoral head (Figure 1A). Acquiring such images requires many hours of training, since slight variations in probe orientation can result in suboptimal images. Previous studies have shown that deviation from the Graf plane by novice users can result in incorrect diagnosis in two-thirds of neonates and half of infants evaluated [15]. These variations can be reduced to some extent by using 3D Ultrasound (3DUS). Since 3DUS covers a larger area, it is more likely to contain the Graf plane, thereby making it more reliable than 2DUS, especially when scanning is performed by novice sonographers [16].

However, in both 2DUS and 3DUS, reliability of the ultrasound examination depends to a large extent on scan quality. In current clinical practice, sonographers manually assess scan quality based on visibility of key landmarks. This approach is prone to high inter-observer variance. As a result, poor-quality images can be presented to radiologists, risking misdiagnosis. Examples of images of varying scan quality are shown in Figure 1. In earlier work [17], a semi-quantitative scoring technique that evaluates individual imaging landmarks (such as ilium, labrum, femoral head and os ischium) and artifacts (such as movement and other artifacts) is proposed. While such approaches make scan quality assessment less subjective, there can still be variability in the manual assessment of individual features.

Automatic assessment of scan quality could address these issues and provide the sonographer with a reliable and non-subjective assessment of image adequacy at the time of scanning. Automatic assessment of ultrasound is challenging due to various factors such as the presence of spurious image effects that resemble anatomical structures, blurred image boundaries, mirror artifacts and shadowing artifacts. The 3DUS images (as well as 2D sweeps) also contain artifacts resulting from patient movement or hand movement at the time of scanning. As a result, commonly used image processing techniques such as template-matching [18,19] and feature extraction [20,21] cannot be directly applied to ultrasound.

Related Work on Deep Learning in Hip Ultrasound

With the recent success of Convolutional Neural Networks (CNNs) in medical image processing, data-driven approaches have been proposed for automatic image segmentation and classifications in hip ultrasound. Hareendranathan et al. [22] developed a CNN for segmenting the acetabulum that used multi-scale super pixels as inputs to an AlexNet model.

The 3D convolutional networks (3DconvNet) such as C3D and I3D have been used for video classification. The C3D network uses a homogeneous architecture with 3 × 3 × 3 convolution kernels in all layers and gave 52.8% accuracy on the UCF101 dataset [23]. The Inflated 3D ConvNet (I3D) used a new two-stream approach to combine 2D and 3D convolutions. With pretraining on Kinetics, the I3D model gave accuracies of 80.9% on the HMDB-51 dataset and 98.0% on the UCF-101 dataset [24]. Similarly, 2D segmentation models such as the 3D U-Net [25], 3D V-Net [26] and the 3D Deeply Supervised network [27] have also been used for image segmentation in videos. Extending these models to 3DUS volumes of the hip is non-trivial, mainly due to the lack of large datasets.

Generally, deep learning models are trained on hip ultrasound images acquired by well-trained sonographers in research settings. Such images contain all necessary imaging landmarks such as straight and horizontal acetabulum and round femoral head. The scan quality of the images used in these datasets was assessed subjectively by the sonographer. In this paper, we develop a deep learning technique to automatically assess the image quality of hip ultrasound images and categorize it as adequate vs. inadequate for DDH diagnosis. Similar deep learning techniques have been developed using CNNs [28,29] and Recurrent Neural Networks (RNNs) [30] for 2DUS scan quality assessment. Along these lines, we propose a new 3D CNN model to predict the adequacy of hip scans for DDH diagnosis in 3D ultrasound images and validate it on two large clinical datasets.

This is the first study to validate an automatic hip quality assessment technique on a large dataset, of 2187 hip scans performed by 3DUS (Dataset 1). Any systematic error in prediction is also examined by evaluating the algorithm on subgroups of images based on age {0–3 months, >3 months} and sex {Male, Female}. On a separate dataset of 107 images (Dataset 2), we conducted a multi-reader study with four readers who semi-quantitatively evaluated scan quality using a 10-point scoring system [17] and compared it to predictions from our AI-based approach.

2. Materials and Methods

We recruited infants who had been referred for ultrasound examination based on clinical suspicion of DDH (due to risk factors such as hip laxity, asymmetrical skin creases, breech position, female sex, positive family history of DDH, first-born infants and ethnicity). In this institutional health research ethics board-approved study, we added 3DUS scanning to the routine 2DUS scanning protocol and obtained written informed parental consent from each participant. Since DDH can be unilateral or bilateral, we included all dysplastic hips separately but, in cases where both hips were normal, we only included one hip per subject in the study. Each hip was categorized as normal or abnormal based on the assessment of the consulting radiologist.

2.1. Ultrasound Scanning

Routine ultrasound examination for both hips was performed per American College of Radiology recommendations [31]. In addition to 2D scans, we also acquired coronal 3DUS images of both hips using a Philips iU22 scanner with a Philips 13VL5 (Philips Healthcare, Andover, MA, USA) linear array transducer (having a center frequency of 13 MHz). Each 3DUS image consisted of 3.2 s sweeps (sweep angle +/− 15 degrees, 256 slices), with the head of the transducer positioned near the greater trochanter of the infant. Sweeps were acquired such that the central slice of the 3DUS volume roughly approximated the Graf plane. Each slice was 411 × 193 pixels (with each pixel measuring 0.11 mm × 0.20 mm) and a slice thickness of 0.13 mm.

2.2. Model Development

We used a 3D CNN consisting of convolutional and fully connected layers to predict scan quality. The optimal architecture for the network (as shown in Figure 2) was dynamically determined based on accuracy of classification on the validation set. Like the C3D model, we used a homogenous kernel size of 3 × 3 × 3 in all layers. To minimize the memory requirement in training, we resized the original 800 × 600 × 256 image to a 32 × 32 × 32 tensor. Since the input tensor is already much smaller compared to the original image, we applied max pooling only in the slice dimension, i.e., a pooling ratio of 1 × 1 × 2. Hence, the height and width of each slice was maintained at 32 × 32 pixels. We used a stride size of 1 in all convolutional layers and used 16, 32 and 64 kernels in the convolutional layers conv1, conv2 and conv3, respectively. The model was trained for 100 epochs to minimize a categorical cross entropy loss optimized using rmsprop at learning rate lr = 0.001. To prevent the model from overfitting, we used 20% dropout in the fully connected layers.

Labels for training the network were obtained from the manual scoring on 1548 other 3DUS scans excluded from the test set by one reader (MM with 4 years of experience in hip ultrasound image analysis). Factors such as the straightness of the ilium, visibility of labrum, os ischium and femoral head and presence of movement artifacts were considered while scoring the images. These labeled images were divided into the training (70%, 1083 images, 651 high-quality images, 432 low-quality images) and validation set (30%, 465 images, 300 high-quality images, 165 low-quality images).

In our dataset, we had a relatively low number of poor-quality images. This is typical of ultrasound scanning taking place in research settings where scans are acquired by well-trained sonographers. We tried to address this issue by assigning a higher weight (0.65 vs. 0.45) for misclassified low-quality scans.

2.3. Model Evaluation

We tested the AI technique on two datasets—(1) Dataset 1 (DS1): A large dataset of 2187 3D hip ultrasound images obtained from 508 infants (each 3D image from a patient scan was analyzed separately for image quality) in which one reader (MM), a radiology resident with 4 years of expertise in hip dysplasia ultrasound, scored images from 1–5, and (2) Dataset 2 (DS2): A separate set of 107 3D images from 101 subjects in which 4 readers evaluated the quality more systematically using a previously published 10-point scoring system [17] as summarized in

Table 1. Method of scoring used to evaluate Datasets 1 and Dataset 2. #+, #− indicate the number of adequate-quality and inadequate-quality images in each dataset.

Dataset	# Images, (#+, #−)	# Readers	Method	Scan Adequacy Criteria
Dataset 1	2187 (1568, 619)	1	Holistic score: Low quality: 1 High quality: 5	>3
Dataset 2	107 (75, 32)	4	10-point scoring 1: High quality: 10 Straight and horizontal ilium: 2 Clearly visible Os ischium: 2 Visible labrum: 1 Round femoral head: 1 No motion artifacts: 2 No other image artifacts: 2	>6

¹ Details on the 10-point scoring system can be found in [17].

.

2.4. Statistics

The readers included one expert (reader 4) who is our lead radiologist, with fellowship training in pediatric and musculoskeletal radiology and 13 years of experience (JJ) and three non-expert readers (reader 1, BC, a radiology resident with 2 years’ experience in hip ultrasound, reader 2, EO, a graduate student in radiology with 1 year experience in hip ultrasound, reader 3, AH, a research associate with 5 years of experience in ultrasound hip image analysis). For binary classification, scans with a score of above 3/5 in DS1 and above 6/10 in DS2 were considered ‘adequate’ quality.

Accuracy, sensitivity, specificity, Negative Predictive Value (NPV), Positive Predictive Value (PPV) and Area Under the ROC Curve (AUC) were calculated for each image with the manual categorization treated as ground truth. Descriptive statistics of agreement of AI prediction with manual readings are also reported in terms of ICC (3,k) and kappa. We evaluated the Accuracy, Sensitivity (SN), Specificity (SP), and Positive and Negative Predictive Value (PPV, NPV) of the AI network as a diagnostic test for image quality in which a true-positive result was an image manually defined as inadequate (i.e., a test that flags poor-quality scans). Calculations were performed using our inhouse software developed in Python 3.6 using sklearn and pingouin libraries.

3. Results

As seen in

Table 2. Accuracy, sensitivity, specificity, Negative Predictive Value (NPV), Positive Predictive Value (PPV) and AUC of our AI-based approach. AUC represents the area under the Receiver Operating Characteristic (ROC) curve.

Dataset	Reader	Accuracy	Sensitivity	Specificity	PPV	NPV	AUC
Dataset 1	AI	0.96	1.0	0.87	0.94	1.0	0.93
Dataset 2	AI	0.91	0.9	0.93	0.97	0.76	0.91
	Reader 1	0.89	0.9	0.85	0.95	0.74	0.88
	Reader 2	0.7	0.59	1.0	1.0	0.46	0.8
	Reader 3	0.75	0.72	0.81	0.92	0.5	0.77

, within DS1 (2187 3DUS images from 508 infants) the AI technique correctly predicted 596/619 images as inadequate quality and 1501/1568 as adequate quality. In DS2 (107 3DUS images), the AI technique gave the highest accuracy, sensitivity, specificity, NPV and PPV when compared to three non-expert readers. We also tried to address the data imbalance (651 high-quality images vs. 432 low-quality images) by assigning a higher weight to the underrepresented class (i.e., low-quality images). However, this gave a lower AUC of 0.89.

In order to check for any systematic error in predictions, we divided the images into subgroups based on age and sex and evaluated all parameters within the subgroups as summarized in

Table 3. Comparison of accuracy, sensitivity, specificity, PPV, NPV and AUC in subgroups of data categorized based on age and sex. Note that there was no systematic bias in any subgroup.

Variable	Subgroup	Accuracy	Sensitivity	Specificity	PPV	NPV	AUC
Age	0–3 months	0.97	1.0	0.84	0.96	1.0	0.92
	>3 months	0.93	1.0	0.89	0.83	1.0	0.94
Sex	Male	0.96	1.0	0.87	0.94	1.0	0.94
	Female	0.96	1.0	0.86	0.94	1.0	0.93

. Accuracies were similar in all subgroups, indicating no systematic error in prediction.

Agreement of AI-based predictions with manual scoring was quantified using ICC and Cohen’s kappa as shown in

Table 4. Agreement of AI predictions with scores provided by expert and non-expert readers. The 95% Confidence Intervals (CIs) of each value is provided in square brackets.

Dataset	Readers	Ground Truth	Kappa	ICC
DS1	AI	Expert	0.90 [0.97, 0.82]	0.95 [0.94, 0.95]
DS2	AI	Expert	0.77 [0.85, 0.68]	0.88 [0.84, 0.91]
	Reader 1	Expert	0.72 [0.79, 0.65]
	Reader 2	Expert	0.43 [0.50, 0.35]
	Reader 3	Expert	0.44 [0.50, 0.38]

. We used kappa to measure the inter-observer agreement between AI and the expert reader in Dataset (DS) 1. The DS was evaluated by three non-expert readers and one expert (JJ). Compared to the non-expert readers, AI showed higher agreement (0.77 vs. 0.72) with the expert. The difference in kappa scores between AI and non-expert human readers was statistically significant (p < 0.05) for readers 2 and 3.

Overall agreement of AI with manual reading was quantified using ICC considering AI-based prediction as the 2nd reader in DS1 and as the 5th reader in DS2. In both cases, agreement among readers was high (DS1 −0.95; DS2 −0.88).

Some examples of images of varying quality that were evaluated by AI are shown in Figure 3. All images in row 1 are high-quality images with all necessary landmarks that the CNN correctly identified as adequate. Similarly, images in row 2 are images that are correctly categorized as inadequate by AI. Images in row 3 are moderate-quality images in which there exists variability in manual assessment. The expert radiologist scored two of these images (A, B) as adequate (with score 7) and one image (C) as inadequate (with score 6). AI classified images A and B as inadequate and C as adequate. This row of images highlights the inevitable subjectivity of assessment of intermediate-quality scans.

4. Discussion

We developed a 3D CNN for automatic ultrasound scan quality assessment and validated it on two datasets, one large (>2000 scans) and one as part of a multi-reader exercise (four human readers, 107 scans). This is the first study of automatic ultrasound scan quality assessment on a large dataset with wide representation from infants of different age groups and sex. On the large data set, our CNN was 96% accurate (with 100% sensitivity and 87% specificity) when compared to expert manual assessment of quality. In the multi-reader study, the AI technique actually performed slightly better than each of the three non-expert human readers in agreeing with the 4th expert human reader on diagnostic quality as rated in a 10-point scoring system [17]. Based on the commonly used interpretation of ICC (where values less than 0.5 indicate poor, 0.5–0.75 moderate, 0.75–0.9 good and greater than 0.90 excellent [32]), the CNN showed excellent reliability.

The 3D CNN approach flags low-quality images containing imaging artifacts that are commonly seen in 3DUS (as well as 2D sweeps). These artifacts usually occur due to patient movement, hand movement and ultrasound shadowing. Although we do not specifically segment these artifacts, since our technique considers temporal and spatial information around each pixel, images with high occurrence of image artifacts are categorized as inadequate.

Since 3DUS scans consist of a large number (~250) of slices, manual assessment of scan quality is tedious. As a result, anatomical landmarks crucial to DDH assessment could be missed in 3DUS. Our automatic assessment technique is fast (average execution time ~2 s/image on the Compute Canada Cedar cluster with an NVIDIA V100 GPU) and accurate and it can be used to provide feedback on scan quality to the sonographer in real time during hip examination.

This can also be used as a preprocessing step for automatic interpretation of hip ultrasound. Automatic interpretation usually involves calculation of alpha angle and/or predicting the probability of DDH from ultrasound images. Since these techniques [33,34,35] generally rely on imaging landmarks such as ilium, acetabulum, os ischium and femoral head, their accuracy depends to a large extent on adequate visualization of these features. Our technique could flag inadequate images upfront and thereby improve the accuracy of these automatic hip interpretation systems. Since the focus of this paper is to apply a CNN model on the new use case of hip image quality assessment, we have not compared it to other 3D networks such as C3D, I3D or 3D U-Net.

One limitation of our study is the variability in ground truth as it is based on manual assessment. Images in Dataset 1 were holistically scored from 1–5 by a manual reader. There could be variability in these scores if the same images were to be scored by a second reader. We address this issue to some extent in Dataset 2 which was semi-quantitatively assessed using the scoring system described in [17]. Using the scoring system, the human reader makes a series of (mostly binary) decisions based on landmarks found in the image. Although this reduces the variability, there are inevitably some variations in manual assessment of these individual features, most relevant in intermediate-quality images. For example, as shown Figure 2 row 3, two images were assessed as adequate quality by two human readers but with a low score of 7 points. These images were assessed as inadequate by AI with a probability of 0.42 and 0.4. Such images represent edge cases where there exists ambiguity in ground truth assessment.

Images used in our study were acquired in a research environment by experienced sonographers in dedicated sessions which represent the idealized scenario. Hence, most images were of high quality. For example, in Dataset 2 more than 90% of the images have clearly visible labrum and femoral head. As future work, we plan a multi-center study on 2D sweep images acquired from tertiary centers and small clinics using handheld pocket-sized devices. In this study, we expect to acquire more low-quality scans that can be used to further validate the utility of our AI-based approach. Demonstrating validity of the AI technique for assessing scan quality in these settings is critical to the feasibility of using 2DUS for population screening of DDH.

5. Conclusions

We developed a new AI technique for automatic interpretation of ultrasound scan quality and validated it on two datasets of 3D ultrasound images. Our approach was fast, highly reliable and showed agreement with an expert reader that was equivalent to or better than that of three other human readers. This new AI approach can be used to provide feedback on scan quality to sonographers during or immediately after ultrasound examination. It can also be used as a preprocessing step in interpretation systems to filter low-quality images and improve the reliability of hip ultrasound examinations.