Patients
The present study was conducted using secondary data from the previous prospective study. The study was approved by the local institutional review board of the Graduate School of Medicine, Chiba University. The additional requirement for informed consent was waived by the local institutional review board of the Graduate School of Medicine, Chiba University because of the retrospective analysis. All procedures involving human participants were in accordance with the 1964 Declaration of Helsinki and its later amendments. Patients who visited our foot and ankle clinic from February to June in 2016 and underwent weight-bearing dorsal X-rays of the foot were recruited. Patients with acute inflammatory diseases such as cellulitis and gout, or with a history of ankle surgery, fractures, or dislocation within the past year, were excluded. There were a total of 131 patients enrolled in this study.
Photography of the Feet
Patients took photographs and also received radiographs during their outpatient visits. Patients photographed their feet using a digital camera or smartphone according to an instruction sheet, which was given to each subject to standardize the foot position for the photograph. The camera or smartphone type was not specified. Participants who did not have a camera or smartphone were provided with a digital camera (IXY 150, Canon, Ota Ward, Tokyo). Participants did not receive additional instructions or assistance from the research staff. This was to simulate a situation where a patient uses a smartphone app to take a picture of his or her foot to be diagnosed with HV. The images of the feet were divided into right and left and cropped to a minimum region, which included the ankle to the toes. (Fig.2a, b) The background was removed semi-automatically using PowerPoint (Microsoft Corporation, Redmond, WA). (Fig.2c). In order to correctly identify the big toe and little toe, the photo of the left foot was flipped horizontally so that all foot orientations were recognized as the right foot. A total of 346 images of feet were acquired. One hundred and seventeen patients took images of both feet, and 43 of the 117 took twice. In addition, 14 patients took pictures of one foot, and 3 out of 14 took images twice.
Radiographic dataset
For the weight-bearing, dorsoplantar-view radiographs of the feet, patients were instructed to stand in a relaxed position, distribute the weight evenly on both feet, and keep the feet parallel. The central beam is angled to approximately 15-20 degrees towards the heel at a distance of 100 cm, directly parallel to the long axis of the foot, and centered on the second tarsometatarsal joint 12. The radiographic parameters of HV including the HVA, M1-M2 angle, and M1-M5 angle were measured. The HVA refers to the angle formed by the axis of the proximal phalanx of the hallux and the axis of the first metatarsal (Fig 3a). The M1-M2 angle is the angle formed by the longitudinal axis of the first and second metatarsals (Fig 3b). The M1-M5 angle is the angle formed by the longitudinal axis of the first and fifth metatarsals (Fig 3c).
Hallux valgus, defined as HVA of≥20°, was classified as mild [20°, 30°), moderate [30°, 40°) or severe (>40°) 13.
HV was measured using the angular measurement function of the Picture Archiving and Communication System and rounded to the nearest whole number for analysis. All images were measured by a board-certified orthopaedic surgeon (SM 14-years of experience).
A total of 248 radiographs were taken, with 117 patients taking radiographs of both feet and 14 people taking radiographs of one foot.
CNN model construction
The CNN architecture was constructed using Python 3.6.7 and Keras 2.2.4 with Tensorflow 2.0.0 at the backend. The models were separately constructed for the HVA, M1-M2 angle, and M1-M5 angle. In this study, we adapted the Xception architectural model, which had been previously trained using images with ImageNet 14,15. The input images were resized to 299 × 299 pixels. We replaced the final layer of the model with a global average pooling layer and a fully connected layer to make the classification model into a regression model 16. Then, we fine-tuned the pre-trained model using the photographs of the feet and the measured radiographic parameters. The first 26 layers were frozen and the weights were not modified during the training process, then the rest of the layers were retrained with our data. The network was trained over 1000 epochs with a learning rate of 0.1, which was reduced if no improvement was seen. Adam was used for the optimizer, and the RMSE was used for the loss function. The data augmentation was done by ImageDataGenerator including a rotation angle range of 90°, a width shift range of 0.1, a height shift range of 0.1, a shear range of 0.1, and a horizontal flip of 50%. The CNN was trained and validated using a computer with a GeForce RTX 2060 graphics processing unit (NVIDIA, Santa Clara, CA).
Performance evaluation of the model
To train and evaluate the CNN model, we performed a five-fold cross-validation. Photographs of the foot were randomly divided into five equal-sized independent subgroups. Images taken from the same patients were assigned to the same subgroup. In each iteration, data from the four subgroups were selected as a training set and the remaining independent subgroups served as validation data. In the validation phase, the performance to predict the radiographic parameters was assessed using the remaining independent subgroup. This cross-validation process was repeated five times. The R2 and the RMSE were calculated to evaluate the performance of the CNN using the sklearn.metrics.r2_score and the square root of sklearn. metrics.mean_squared_error function from the Scikit-learn library (version 0.23.2), respectively. The severity of hallux valgus was graded as normal, mild, moderate, and severe 13 from the degree of the predicted HVA angle, and the agreement with the severity grading based on the radiographic HVA measurement was evaluated using a Cohen's kappa coefficient. The Cohen's kappa coefficient was calculated using sklearn. metrics.cohen_kappa_score function from the Scikit-learn library.