Adaptive Thresholding of CNN Features for Maize Leaf Disease Classification and Severity Estimation

Abstract

Convolutional neural networks (CNNs) are the gold standard in the machine learning (ML) community. As a result, most of the recent studies have relied on CNNs, which have achieved higher accuracies compared with traditional machine learning approaches. From prior research, we learned that multi-class image classification models can solve leaf disease identification problems, and multi-label image classification models can solve leaf disease quantification problems (severity analysis). Historically, maize leaf disease severity analysis or quantification has always relied on domain knowledge—that is, experts evaluate the images and train the CNN models based on their knowledge. Here, we propose a unique system that achieves the same objective while excluding input from specialists. This avoids bias and does not rely on a “human in the loop model” for disease quantification. The advantages of the proposed system are many. Notably, the conventional system of maize leaf disease quantification is labor intensive, time-consuming and prone to errors since it lacks standardized diagnosis guidelines. In this work, we present an approach to quantify maize leaf disease based on adaptive thresholding. The experimental work of our study is in three parts. First, we train a wide variety of well-known deep learning models for maize leaf disease classification, then we compare the performance of the deep learning models and finally extract the class activation heatmaps from the prediction layers of the CNN models. Second, we develop an adaptive thresholding technique that automatically extracts the regions of interest from the class activation maps without any prior knowledge. Lastly, we use these regions of interest to estimate image leaf disease severity. Experimental results show that transfer learning approaches can classify maize leaf diseases with up to 99% accuracy. With a high quantification accuracy, our proposed adaptive thresholding method for CNN class activation maps can be a valuable contribution to quantifying maize leaf diseases without relying on domain knowledge.

1. Introduction

Crop diseases remain a global challenge to food security. According to [1], 20 to 40% of global crop production is lost to pests and diseases annually. Accordingly, the application of machine learning and deep learning models for identifying and quantifying crop diseases has emerged as one of the important themes of the 21st century, especially in precision agriculture [2]. This follows multiple efforts to automate disease monitoring in agriculture, for example [3,4]. Most deep learning models in the literature have achieved tremendous success in general image classification tasks. Therefore, it is natural to leverage the advances made in this field to meet the demands of precision agriculture [5]. Central to this is the selection of leaf image features that contain sufficient data for training the models, or the use of specialized deep learning models to extract features on the fly using a combination of various convolution and pooling layers. In most cases, deep learning models are selected based on meeting the requirements of the project [6,7], such as training time and computational resources.

Maize leaf disease quantification, or maize leaf severity analysis, is a very interesting research problem that seeks to determine the extent to which a leaf has been affected by disease [8,9,10]. This process relies on trained experts to visually estimate the affected area and assign a severity label. In the literature, multiple works have successfully applied multilabel automated visual classification frameworks to identify diseases and evaluate damage based on a human-defined scale [11]. The intuition is that an image can be associated with multiple labels. For example, a diseased leaf image can be classified as blight and then categorized as “low severity”, “medium severity” or “high severity”. Such methods rely on nominal scales for quantification, and have been used by Owomugisha et al. [11], among others, to indicate various levels of disease severity. This method of quantifying leaf diseases has been widely accepted to solve the problem of leaf disease quantification; however, with thousands of leaves to annotate, it becomes tedious and prone to errors. Although this solution is very popular in the literature, it cannot be applied to solve all disease quantification problems since leaf diseases have patterns that are statistical in nature, and hence more challenging for a human eye to assign a severity label.

Machine learning methods that have been applied to solve maize leaf disease classification problems can be grouped into two sets: hand-crafted feature-based models [12] and automatic feature extraction models [13]. The parameters of hand-crafted feature-based models are estimated mainly based on the domain knowledge of the problem and the statistical characteristics of the leaf image. However, one of the biggest challenges, in general, of relying on statistical characteristics such as mean, standard deviation and variance, among others, is that: (i) variations may be indistinguishable for leaf diseases with low intraclass and high interclass variability [14]; (ii) different diseases may have similar effects on leaves; (iii) a single leaf may have multiple diseases, and (iv) disease symptoms may vary. On the other hand, automatic feature extraction models, particularly CNNs, require large amounts of data to learn the discriminative features of the image classes [14], which is a major limiting factor considering the difficulties of gathering the data.

This work is motivated by the ability to quantify maize leaf diseases based on the information contained in the class activation maps (CAM) of the CNN. Unlike qualitative leaf disease severity estimation methods that rely mainly on domain knowledge, this approach uses the same deep learning model for leaf disease classification and leaf disease quantification, as described in Section 3. The critical component of our system is based on identifying which regions of the diseased leaf image have contributed more to the final decision of the CNN, and we express those regions as a percentage of the total leaf area to estimate disease severity. We compare the performance of the proposed method with state-of-the art image segmentation techniques for leaf disease estimation. The proposed system can assign a severity estimate as a ratio without prior information from experts, and identify the following maize leaf conditions with high accuracy: Northern Corn Leaf Blight, Gray Leaf Spot, common corn rust and healthy leaves.

The main goals of the suggested method are: (1) Use transfer learning to reuse CNN models to assign the correct maize leaf disease class to the given image. (2) Use CNN class activation heatmaps to derive a percentage severity estimate based on the suggested adaptive thresholding technique. Our hypothesis is that the extracted ROI from the class activation heatmap correlates with the area covered by the disease in the original leaf image. Therefore, the area of the ROI as a percentage of the total leaf area gives us an estimate of the disease severity. We summarize the main contributions of this work as follows:

• We propose a novel method for simultaneously classifying and quantifying maize leaf diseases without any prior knowledge of the lesions.
• To improve the classification accuracy of maize leaf disease identification, this work investigates several models: custom CNN, Inception V3 [15], DenseNet 121 [16], DenseNet 201 [16], MobileNetV2 [17], VGG16 [18], EffcientNetB0 [19], ResNet50 [20] and LeNet [21]. The performance of the custom CNN is further examined by comparing the effect of regularization techniques such as batch normalization and dropout operations. Finally, we compare the experimental results of the custom CNN with those of the transfer learning techniques to determine the effect of CNN depth and number parameters on the performance of the model.
• We propose a unique and computationally efficient adaptive thresholding method for quantifying the diseased regions of leaf images in an unsupervised manner. The advantage of this method is that it uses spatial relationships between the pixels to predict regions that are affected by the disease.
• Experimental results on the dataset demonstrate two things. First, the transfer learning techniques achieve higher classification accuracies of 99%. Second, our proposed adaptive thresholding approach performs well, giving good results, and therefore is a good approach for maize leaf disease quantification.
• Comparison evaluation with the most common maize leaf disease severity analyses such as Bernsern’s, Niblack’s and Sauvola’s thresholding reveal that our approach is robust and gives good results.
• We created an augmented dataset consisting of 46,898 samples from the original PlantVillage [22] dataset, available at: https://github.com/harryD1/maize_leaf_diseases_classification_and_quantification (accessed on 1 June 2022).

We structure this paper as shown in Figure 1. We begin by giving an overview of related studies on leaf disease classification and severity estimation in Section 2. Section 3 presents the materials and methods that are used in the experiments. In Section 4, we present the results and performance of the methods based on their ability to predict the correct class among four maize disease classes. In Section 5, we present the discussion, give an outline of the work’s limitations, and give future directions of our research. Section 5 concludes the work.

2. Related Work

In this section, we give an overview of the research and methods that are relevant to the present study. In Section 2.1, we summarize transfer learning techniques that have been used for maize leaf disease classification. Then, in Section 2.2, we give a summary of the most recent work on maize leaf disease severity estimation. Finally, we conclude by studying the current state-of-the-art thresholding techniques in Section 3.2.3.

2.1. Transfer Learning on Maize Leaf Disease Classification

In its simplest terms, transfer learning is adapting or reusing the knowledge that the CNN has learned previously and applying that knowledge to a new set of problems [23]. For example, in Figure 2, we can have a CNN such as VGG16 that has learned to recognize objects from the ImageNet database and then use all or part of that knowledge to recognize the maize leaf diseases in our database. In most cases, we transfer the knowledge gained from a lot of data to a problem that has relatively little training data. This is helpful because most image recognition problems from popular datasets such as ImageNet (14 million images and 1000 classes), MS-COCO (1.5 million images and 1000 classes) and CIFAR10 (60,000 images and 10 classes) rely on similar low-level features, and the CNN will have learned the basic structure of image objects, thereby increasing the learning rate on the new dataset.

Transfer learning has been used in several studies to address some of the limitations of deep learning models in generic image recognition problems. From the literature, it is clear that transfer learning leverages less training data, decreases training time and greatly improves the performance of deep learning models, as we shall see in this paper.

Most early studies in this field focus on the use of handcrafted features for leaf disease detection and classification. Notably, the work by Panigrahi K.P et al. proposes an early maize leaf disease detection and classification system using machine learning [24]. In their work, they extract distinctive maize leaf features such as shape, color and texture and apply machine learning techniques such as Naive Bayes, Decision Tree, K-Nearest Neighbor, Support Vector Machine and Random Forest to classify the leaf images [24]. Their results demonstrated that the Random Forest classifier performs well, giving accuracies of over 79%, while the Support Vector Machine achieves a lower accuracy of just over 77%.

In a similar study, Alehegn et al. developed maize leaf disease recognition and classification using only a Support Vector Machine [25]. They extracted similar features (texture, color and morphology), but in this instance, the authors managed to obtain good accuracies of 95% from a dataset of only 800 images. This can be attributed to a small dataset.

These results and other works [26,27,28,29] demonstrate the feasibility of using handcrafted features for crop leaf disease classification; however, the limitations of relying on such techniques naturally include: (1) the need for expert knowledge, (2) lack of robustness and (3) extensive time needed to identify relevant features from leaf images [30].

In order to improve performance, some authors have presented a unique approach to identifying maize leaf diseases. They consider a learning paradigm that automatically learns the features of maize leaf diseases without the need for expert knowledge. For instance, DeChant et al. developed an automated system to identify Norther Leaf Blight in maize plants using deep learning [31]. They trained several CNNs to identify the presence of the disease on maize leaf images. Although their task was a binary classification problem, the authors managed to achieve a classification accuracy of 96% on the test set.

A common technique in the literature is to combine a deep learning feature extraction module with a supervised classifier such as SVM [32], Decision Tree [33] or K-Nearest Neighbor [33,34]. This kind of architecture leverages the convolutional operation of the feature extraction layers and the power of the supervised classifier to improve classification performance. This combination has already offered good results in generic image classification tasks that require modelling spatial dependencies [35]. However, significant results were demonstrated by authors who introduced the concept of transfer learning for plant leaf disease classification [36].

This is a promising approach to address the requirements of maize leaf disease identification: First, performance is greatly improved compared with developing models from scratch [36]. Second, transfer learning from a pre-trained network does not require a lot of training data compared with training a model from the ground up, and the training time is significantly lower. To customize the base convolutional network, we first delete the classifier from the model, and then we add a new classifier that will be trained from the ground up so that we can reuse the historical features learned previously from the initial problem, as shown in Figure 2.

2.2. Maize Leaf Severity Analysis

Bock et al. [37] define severity analysis as determining the total area of the leaf that is affected by the disease, and such calculations can be estimated by using the leaf color or shape features [38]. Typically, the procedure follows a three-step process to estimate the severity: image pre-processing, image segmentation and image analysis. Most disease severity estimation work in the literature can be grouped into one of two sets: (1) color image processing techniques or (2) machine learning and deep learning methods. In color image processing, researchers use digital image processing to segment the diseased parts of a leaf and determine the total affected area, while in (2), researchers extract the relevant features of the leaf diseases [38] and pass the information to either machine learning or deep learning algorithms. Some authors have reported significant progress using third-party image processing software [39].

2.2.1. Color Image Processing

The motivation for using color image processing techniques for leaf disease severity analysis stems from the fact that it is possible to segment the leaf image based on the color of the pixels [39]. Basically, we subdivide the image into its constituent regions or objects until all the regions of interest have been isolated [39]. In what follows, we review works that adopt color changes to segment leaf images.

In 2006, Barbora et al. assessed the impact of the two-spotted spider mite on host plants using third-party image processing software (Mathematica). In their work, they converted the leaf images to greyscale and then measured the total damaged area based on a pre-determined threshold. Then, they expressed the result as a ratio of the total number of pixels of the leaf image to obtain a severity estimate [40]. Their results show that this method is reliable and can identify significant differences in pixel damage due to different concentrations of spider mites.

2.2.2. Segmentation

Several studies have used a threshold [40,41,42,43,44,45,46,47] for leaf disease severity analysis. However, such techniques generally suffer from low classification rates due to poor sensitivity. Namita addresses this problem by using an adaptive threshold for automatic segmentation. In their work, they first convert the RGB color images to Lab color space, and then the adaptive threshold value is obtained from the maximum intensity values of the blue color channel. Finally, disease quantification is performed by taking the ratio of the diseased parts to the ratio of the entire leaf image. Their method is computationally efficient, and, above all, delivers superior results even on poor-quality images. In particular, they were able to achieve a 99% classification accuracy on cherry crop images.

2.2.3. Machine Learning and Deep Learning

Diseases mainly alter the color, shape and intensity of leafs [48]. It is, therefore, reasonable to extract meaningful information from such deformations and use it as features to represent the respective diseases. The most common features that authors use from the color moments are the mean, standard deviation and skewness, as defined in Equations (1)–(3) [48]:

$${\mu }_i=\frac{1}{N}\sum _{j=1}^N{f}_{ij}$$

(1)

$${\sigma }_i={\left[\frac{1}{N}\sum _{j=1}^N{(f_{ij}-{\mu }_i)}^2\right]}^{\frac{1}{2}}$$

(2)

$${\gamma }_i={\left[\frac{1}{N}\sum _{j=1}^N{(f_{ij}-{\mu }_i)}^3\right]}^{\frac{1}{3}}$$

(3)

where ${\mu }_i$, ${\sigma }_i$ and ${\gamma }_i$ $(i=1,2,3\dots )$ represent the mean, standard deviation and skewness of each image, respectively [48].

Several authors have also reported success with features extracted from the histograms of diseased leaf images. Without a doubt, the histogram of an image provides important statistics that are relevant in image processing tasks [49]. They are simple to calculate, making them a tool of choice in near real-time digital image processing applications [49]. For example, we can use Equations (4) and (5) to extract some statistical variables that can help us segment the digital image. Developing from the previous intuition of Section 2.2.3, if we have a grayscale image I with a total of N pixels and K possible intensity values $0\le g<K$, we can compute the mean and variance from the histogram of the image as follows [49]:

$${\mu }_I=\frac{1}{N}·\sum _{u,v}I(u,v)=\frac{1}{N}·\sum _{g=0}^{K-1}g·h\left(g\right),$$

(4)

$${\sigma }_I^2=\frac{1}{N}·\sum _{u,v}{[I(u,v)-{\mu }_I]}^2=\frac{1}{N}·\sum _{g=0}^{K-1}{(g-{\mu }_I)}^2·h\left(g\right).$$

(5)

where ${\mu }_I$ and ${\sigma }_I^2$ represent the mean and the variance of the image, respectively.

In general, extracting leaf features such as color, shape or intensity is the basis of any leaf disease quantification problem. In fact, a majority of digital image processing techniques and third-party software packages for severity analysis rely on using one or more such features.

3. Materials and Methods

This section describes the materials and methods we used for our work. First, we describe our dataset, then the CNN models, and lastly, the proposed algorithm for maize leaf disease severity estimation.

3.1. Disease Dataset and Configuration of the System

In this study, we approach the problem of maize leaf disease quantification by building and testing a wide range of deep learning models on the publicly available PlantVillage maize leaf dataset. The original PlantVillage dataset for maize leaf diseases contains 4188 images spread across 4 different maize leaf diseases. In total, 1162 of the images are healthy, 574 are Gray Leaf Spot, 1306 are Common Rust, and 1146 are Blight. Images in the dataset are raw RGB color images of different sizes. Since deep learning algorithms require large amounts of training data, we build an augmented dataset from the existing PlantVillage dataset by applying six different augmentation methods, and we obtain a total of 46,896 images. We apply the following transforms:

• Resizing—($224\times 224$);
• Shear—level $0.2$;
• Rotation—90 degrees;
• Flipping—up_down flip;
• Saturation;
• Mean filtering;
• Flipping meaned image;
• Cropping—central_fraction = 0.8;
• Cropping—central_fraction = 0.5.

The resulting augmented PlantVillage dataset contains 9379 training images and 2345 test images for each class. Figure 3 shows the histogram of the train–test distribution of the final images, and Figure 4 shows the four different maize leaf disease classes randomly selected from the training set.

We then use this augmented dataset to train several well-known deep learning models, along with the proposed shallow CNN that contains only three covnet layers. The purpose of the CNN model is to classify the input image into the four maize leaf disease classes. The CNN identifies the specific class of the image by using class activation maps. The next process is leaf disease severity estimation using class activation heatmaps. The method accepts the class activation heatmap as a 2D matrix and then identifies the number of pixels that have contributed to the class decision. These pixels are expressed as a percentage of the total leaf area. Finally, the method returns the percentage area of the affected region.

3.2. Proposed Methodology

In this section, we present the architecture and the methodology of our proposed maize leaf disease classification and quantification framework. Unlike other works, the framework presented here accepts raw RGB maize leaf images and outputs the predicted class as well as a severity estimate. Below are the steps taken to achieve the objective. A diagramatic view of this framework is shown in Figure 5.

Step 1:

We train a wide variety of well-known deep learning models for the maize leaf disease classification problem. Then, we compare the performance of the deep learning models and extract the class activation heatmaps from the prediction layers of the CNN models.

Step 2:

We extract the class activation heatmap as a 2D matrix from the covnet layers and determine the area affected by the disease (region of interest).

Step 3:

We use these regions of interest to estimate image leaf disease severity.

3.2.1. Architecture of the Proposed CNN

Here, we define our custom CNN model. Most state-of-the-art CNNs are very deep, and are hence more complex. For example, VGG16 has 16 layers, while ResNet50 has 50 layers, and so on. In order to investigate the influence of CNN depth on classification accuracy, two separate CNN pipelines are implemented. The first consists of a shallow CNN (only three convolutional layers) and is trained from scratch, while the second uses deep CNN architectures (up to 250 convolutional layers) and employs transfer learning to speed up network training.

Below summarizes the network architecture of the shallow CNN shown in Figure 6.

• Input image data: The images in our dataset have different sizes of width × height. However, the pre-processing stage of the CNN models resizes the images to a 3D matrix of $224\times 224\times 3$. We have three channels since we are using RGB images.
• First hidden layer: The first hidden convolutional layer is called conv2d_1. It has 32 feature maps of size $3\times 3$ and a ReLu activation function.
• Second hidden layer: The second hidden layer is called conv2d_2. It has 32 feature maps of size $3\times 3$ and a ReLu activation function.
• Third hidden layer: Referred to as conv2d_3, it has 64 filters of size $3\times 3$.
• Pooling layer: The pooling layer is called MaxPooling2D and is added after every convolutional layer.
• Dropout layer: A dropout layer is added to reduce overfitting; in this model, it has been set to exclude 30% of the neurons.
• Flatten layer: This layer is added to convert the 2D feature maps to a 1D vector.
• First fully connected layer: This has 128 neurons with a ReLu activation function.
• Output layer: This has four neurons and a softmax activation function.

3.2.2. Transfer Learning

Finally, transfer learning is applied to several pre-trained deep CNNs using the Keras and Tensorflow Python packages. The following parameters are used to train the network: base learning rate, 0.0001; momentum, 0.9; number of epochs, 500; and 80% training and 20% test samples. The PlantVillage maize leaf dataset had been augmented using several transforms (Section 3.1) to increase the size of the dataset to 46,896 images.

3.2.3. State-of-the-Art Thresholding Techniques

In what follows, we review some of the state-of-the-art thresholding techniques that have been developed for general image processing tasks.

Bernsern’s Thresholding

The Bernsern’s thresholding technique calculates an adaptive threshold for each heatmap position $(u,v)$ using the maximum and minimum values of a sliding window. If $I_{max}(u,v)$ and $I_{min}(u,v)$ are the minimum and maximum values, respectively, found in an $n\times m$ sliding window, then the Bernsern’s threshold, $T_{u,v}$ is given as [51]:

$$T_{u,v}=\frac{I_{min}(u,v)+I_{max}(u,v)}{2}$$

(6)

Equation (6) depends on the value of a local contrast, $c(u,v)$, as long as $c(u,v)=I_{max}(u,v)-I_{min}(u,v)$ is above a certain limit $c_{min}$. However, if the limit exceeds $c(u,v)$, then the pixels are assumed to belong to a single class [51].

Niblack’s Thresholding

Unlike the previous method, Niblack’s thresholding technique calculates an adaptive threshold for each heatmap position $(u,v)$ using the mean $m(u,v)$ and the standard deviation $\sigma (u,v)$ of the pixels within an $n\times m$ sliding window. If k is an arbitrary constant that can take any value between 0 and 1, then Niblack’s threshold $T_{u,v}$ is given as [52]:

$$T_{u,v}=m(u,v)+k\times \sigma (u,v)$$

(7)

From Equation (7), it is clear that the value of the threshold is affected by k and the size of the sliding window.

Sauvola’s Thresholding

As shown in Equation (8), Sauvola’s technique also uses the mean and the standard deviation to determine the adaptive threshold, just like Niblack’s method. However, Sauvola’s method adapts the threshold based on the contrast of the pixels in the sliding window [53].

$$T_{u,v}=m(u,v)\left[1+k\left(\frac{\sigma (u,v)}{R}-1\right)\right]$$

(8)

where $m(u,v)$ is the mean, $0.2\le k\le 0.5$ and $R=max\left({\sigma }_{(u,v)}\right)$ [53].

3.2.4. Visualizing the Class Activation Map (CAM)

Class activation mapping (CAM) is a technique for identifying the discriminative regions used by a particular class [54]. GradCam is a modification of CAM that seeks to address the limitations of CAM, particularly (i), the need for CAM to have a Global Average Pooling layer in the architecture, and (ii) the drawback of visualizing only the heatmap of the final layer. We need to compute the gradient of $y^c$ with respect to the feature maps A of the last convolutional layer, i.e., $\frac{\mathsf{\delta }{y}^{\left(c\right)}}{\mathsf{\delta }{A}_{ij}^{\left(k\right)}}$. These gradients flowing back are global-average-pooled to obtain weights $w_k^c$, as shown in Equation 9.

$$w_k^c=\frac{1}{Z}\sum i\sum j\frac{\mathsf{\delta }{y}^{\left(c\right)}}{\mathsf{\delta }{A}_{ij}^{\left(k\right)}}$$

(9)

where $w_k^c$ captures the importance of feature map K for target class c.

The GradCam heatmap in Figure 7, is a weighted combination of feature maps with a ReLu.

$$L_{GradCam}^c=ReLu\left(\sum _{k=1}^K{w}_K^c{A}^K\right)$$

(10)

3.2.5. Heatmap Extraction

Equation (10) produces a heatmap that shows how the model makes predictions. This heatmap is a rectangular array of real numbers $y^c=f(A^1,\dots ,A^{512})$ that can be used to visualize the diseased parts of an image and hence derive an estimate of the severity level. In other words, the brighter parts of the heatmap indicate the importance of those parts with respect to disease mapping. However, this heatmap not only contains the required regions of interest (ROI) but also the interference regions, which vary in magnitude.

3.2.6. Thresholding the Heatnap of the CAM

The aim of this algorithm is to isolate the region of interest from the background. Thus, given a heatmap image I, fixed thresholding groups the pixels of the heatmap into sets $H_0$ and $H_1$, where $H_0$ contains all the elements less than or equal to a particular threshold T, and $H_1$ contains all the elements greater than the threshold, as shown in Equation (11):

$$(i,j)\in \left\{\begin{array}{cc}{H}_0,\hfill & \mathrm{if}I(i,j)\le T.\hfill \\ H_1,\hfill & \mathrm{if}I(i,j)>T.\hfill \end{array}\right.$$

(11)

However, this thresholding cannot adequately extract the ROI from the heatmap, for example when confronted with pixels that contain varying levels of interference, or ROIs that have lower values than the fixed threshold.

Therefore, to efficiently separate the ROIs from interference, we first create an $n\times m$ sliding window that moves through the entire 2D heatmap image. The sliding window contains reference cells, guard cells and the cell under test. We then determine an adaptive threshold value $T(m,n)$ for each sliding window position based on the average intensity of the reference cells, and then we compare this value with the magnitude of the cell under test (CUT) $x(i,j)$. If the magnitude of the CUT is greater than the threshold, we declare it a region of interest. We test all the cells of the heatmap until we get a clearer picture of all ROIs. We use the guard cells to create a buffer zone around the CUT. By doing this, we avoid the ROIs from leaking into the reference cells, which could affect the value of the threshold. We estimate the interference levels from the reference cells or neighboring cells of the jth sliding window using Equation (12):

$$Z=\frac{1}{N}\sum _{i=1}^N{x}_i$$

(12)

If Z is the average intensity value of the reference cells, the adaptive threshold value of the jth sliding window is calculated using Equation (13):

$$T_{i,j}=\frac{\alpha }{N}\sum _{i=1}^N{x}_i$$

(13)

where

• $T_{i,j}$ is the adaptive threshold for each sliding window position;
• $\alpha$ is the threshold factor;
• N is the number of samples in the sliding window;
• $x_{i,j}$ is the reference cell.

Algorithm 1 describes the implementation of the proposed adaptive threshold method. It takes a 2D heatmap and the number of reference cells and guard cells as inputs, and it returns a 2D heatmap of the ROIs. Figure 8 shows a graphical description of the algorithm.

Algorithm 1 Disease Extent Detection from CNN Heatmaps

1: procedurecalculate(2D heatmap, number of reference and guard cells (N and M), cell under test (CUT)). 2:       Determine the number of reference and guard cells for each dimension. 3:       Determine the length of the heatmap for each dimension ($L_i$ and $L_j$). 4:       Create a new matrix b for storing the averaged values. 5:       for <each index i from ($N_i$ + $M_i$) to $L_i$ − ($N_i$ + $M_i$)> do 6:             for <each index j from ($N_j$ + $M_j$) to $L_j$ − ($N_j$ + $M_j$)> do 7:                    Determine the average noise level across the reference cells. 8:                    Add the offset to the average threshold value. 9:                    if <CUT[i,j] > b[i,j]> then 10:                        Assign the original value.         ▹ Detection result, (True or False). 11:                  else 12:                        Assign a value of zero. 13:       Suppress the edges to zero. 14:       Return b        ▹ Final heatmap showing the regions of interest.

4. Results

In this section, we conduct experiments to validate the feasibility of simultaneous maize leaf disease classification and quantification using a single deep learning model. Visualizing the CNN class activation heatmaps provides support for the hypothesis that the areas covered by bright pixels represent the areas affected by lesions. Therefore, by extracting the affected areas (ROIs) and expressing them as a ratio of the total leaf area, it is possible to obtain a severity estimate as a percentage.

4.1. Maize Leaf Disease Classification

The proposed study demonstrates the superior performance of transfer learning for retraining convolutional neural networks by comparing the accuracies of CNNs such as Inception V3, DenseNet 121, DenseNet 201, MobileNetV2, VGG16, EffcientNetB0, ResNet50 and LeNet in the context of maize leaf disease classification.

Table 1. State-of-the-art CNN models summary.

CNN	ImageNet Accuracy	Layers	Params (M)
EfficientNetB0	77.1%	132	5.3
InceptionV3	77.9%	189	23.9
Lenet		7	5.6
DenseNet121	75.0%	242	8.1
DenseNet201	77.3%	402	20.2
MobileNet	70.4%	55	4.3
VGG16	71.3%	16	138.4
Custom CNN	–	7	6.5

shows the comparison results.

The results of the experiments indicate that with a large enough dataset, a shallow CNN trained from the ground up can achieve remarkable accuracy compared with mainstream convolutional neural networks. Our shallow CNN with only three convolutional layers manages to achieve high accuracy of 96% compared with the 99% obtained from DensNet121 with 121 layers.

This paper visualizes the class activation maps of the convolutional layers to estimate disease severity. The class activation heatmap has pixels of varying intensities. Bright pixels represent areas that contribute more to the decision of the CNN, while darker pixels represent regions that do not carry any features related to the predicted class.

The adaptive thresholding algorithm adopted in this paper is not an entirely new concept. In fact, it was introduced in the field of radar target detection to identify objects buried in background noise [55]. We adopted the same concept due to the need to extract regions of interest that are often surrounded by background noise in class activation heatmaps. Background noise in this context refers to high-intensity pixels that are not part of the ROI. Fixed thresholding algorithms do not work well for these kind of problems, as the intensity levels of the ROI vary from pixel to pixel.

The adaptive thresholding algorithm’s performance is compared with that of the wellknown adaptive thresholding techniques in image processing. From the results, we can conclude that the proposed adaptive thresholding framework, borrowed from radar signal processing, performs much better than Bernsen’s, Niblack’s or Sauvola’s thresholding. This is because thresholding relies on determining the average intensity of a region while isolating the cell being tested.

The performance of the proposed adaptive thresholding method is highlighted in Figure 9. Here, we demonstrate its ability to extract the ROI from three maize leaf disease classes: Blight, Common Rust and Gray Leaf Spot. Consequently, leaf disease severity is then obtained by expressing the area of the ROI as a percentage of the total area of the leaf.

4.2. Adaptive Thresholding Performance

Figure 10 shows the performance of the various thresholding techniques on extracting the ROI from the noisy CAM heatmap. A good thresholding algorithm must extract the entire ROI while eliminating the background pixels. Any residual background pixels affect the performance of leaf disease quantification. To determine the performance, we manually extract the ROI from Figure 10b and use this as a standard for comparing the thresholding algorithms. Next, we quantify the background noise that remains in the heatmap after applying the adaptive thresholding techniques. Results show that our adaptive thresholding method can remove up to 98.81% of the background noise in the heatmap, while Bernsen’s thresholding removes 76.09%, Niblack’s method removes 92.12%, and Sauvola’s method performs better with 98.64%.

4.3. Leaf Disease Quantification

Once the ROI is successfully extracted, the final step is to determine the area of the ROI and express it as a percentage of the total leaf area. A simple workaround to this problem is to count the non-zero (ROI) and the zero (background) pixels from the heatmap. It then follows that:

$$\mathrm{Diseased}\mathrm{area}=\frac{\mathrm{Area}\mathrm{of}\mathrm{ROI}}{\mathrm{Total}\mathrm{Leaf}\mathrm{area}}\times 100\%$$

(14)

To assess the performance of leaf disease quantification, we use ImageJ software, a Java-based image processing tool to analyze image dataset [56]. We manually segment the diseased area in ImageJ, as shown in Figure 11, and calculate its area. We then compare the area obtained by manual ROI extraction from the heatmap to that obtained by automatic ROI extraction (using the best-performing adaptive thresholding technique). The results are shown in

Table 2. Leaf disease quantification.

Sample Image	% Area of the Diseased Region
ROI from the raw RGB leaf image	4.87%
ROI extracted manually from the heatmap	4.32%
ROI extracted from the heatmap using	3.09%
our adaptive thresholding technique

.

The results in Table 2 show a slight decrease in the area calculated from the heatmap as compared with the area of the original RGB leaf image. This happens because Gradcam does not highlight the entire area covered by the leaf Blight disease. Further, our adaptive thresholding technique estimates a lower area of the ROI because it does not completely remove the background noise from the heatmap. Nonetheless, this presents an important step toward leaf disease quantification. We believe we can improve performance by varying the value of the constant k in our algorithm.

The results also confirm the influence of batch normalization in deep learning. In essence, batch normalization can improve the accuracy and convergence rates by good margins, as shown in

Table 3. Comparison of performance of custom CNN on maize leaf disease dataset against state-of-the-art methods (specificity = 2345 and NPV = 0.2).

Method	Class Label	Precision	Recall	F1-Score	Overall Accuracy
Inception V3	Blight	0.99	0.98	0.98	99%
	Common Rust	0.99	1.0	1.0
	Gray Leaf Spot	0.99	0.98	0.98
	Healthy	1.0	1.0	1.0
DenseNet 121	Blight	0.99	0.99	0.99	99%
	Common Rust	1.00	1.00	1.00
	Gray Leaf Spot	0.99	0.99	0.99
	Healthy	1.00	1.00	1.00
DenseNet 201	Blight	0.99	0.99	0.99	99%
	Common Rust	1.00	1.00	1.00
	Gray Leaf Spot	0.99	0.99	0.99
	Healthy	1.00	1.00	1.00
MobileNet	Blight	0.98	0.98	0.98	99%
	Common Rust	0.99	1.00	0.99
	Gray Leaf Spot	0.98	0.98	0.98
	Healthy	1.00	1.00	1.00
VGG16	Blight	0.99	0.97	0.98	99%
	Common Rust	1.00	1.00	1.00
	Gray Leaf Spot	0.97	0.99	0.98
	Healthy	1.00	1.00	1.00
EfficientNetB0	Blight	0.98	0.98	0.98	99%
	Common Rust	0.98	1.00	0.99
	Gray Leaf Spot	0.98	0.97	0.98
	Healthy	1.00	1.00	1000
LeNet	Blight	0.94	0.87	0.90	94%
	Common Rust	0.90	0.98	0.93
	Gray Leaf Spot	0.93	0.90	0.92
	Healthy	0.98	1.00	0.99
Custom CNN (One dropout layer)	Blight	0.92	0.92	0.92	95%
	Common Rust	0.96	0.96	0.96
	Gray Leaf Spot	0.93	0.90	0.92
	Healthy	0.97	1.00	0.99
Custom CNN (Batch Nomarlization)	Blight	0.93	0.95	0.94	96%
	Common Rust	0.96	0.98	0.97
	Gray Leaf Spot	0.97	0.92	0.94
	Healthy	0.98	1.00	0.99
Custom CNN (No Regularization)	Blight	0.92	0.91	0.92	95%
	Common Rust	0.94	0.97	0.96
	Gray Leaf Spot	0.93	0.92	0.92
	Healthy	0.99	1.00	1.00

and Figure 12. In Table 3, batch normalization increased the accuracy of the shallow CNN by 1%, while Figure 13 shows a steeper learning curve, and the corresponding confusion matrices are shown in Figure 14.

5. Discussion and Conclusions

In the collected literature, severity analysis has been largely based on qualitative assessment, with authors switching between nominal and ordinal scales of measurement, i.e., low-severity, medium-severity and high-severity. There is no quantitative framework for assigning a percentage to the problem of maize leaf disease quantification. In this paper, we present an important step towards simultaneous crop disease classification and quantification. We analyzed maize disease data using a variety of deep convolutional neural network models together with deep CNN feature visualization. The ultimate goal is to limit the number of data features and image processing stages while obtaining a discrete measure of disease severity. To this end, we employed a combinations of techniques from leaf image augmentation, transfer learning and CAM visualization (extracting the ROI) to applying an adaptive thresholding method for estimating the leaf area affected by the lesions. In comparison to other methods on the same classification problem, the proposed method obtained a competitive level of accuracy for classifying maize leaf diseases. The key motivation for our approach is that it can work in low-resource environments, e.g., when data are scarce or on low-computational-power systems, but most importantly, it can quantify crop diseases without the need of expert knowledge.

Additionally, our study used the most-common data augmentation techniques and obtained good accuracies of over 99% for most CNN models. Lenet was the worst performing model, with an accuracy of 94%. Our proposed shallow CNN model that uses only three convolutional layers managed to obtain an accuracy of 95% without any regularization. This demonstrates that with a larger dataset, shallow CNNs can produce impressive accuracies comparable with some deeper CNNs. All the benchmark CNN models used in this work rely on multiple deep convolutional layers to obtain high performance on our dataset. Additionally, when shallow CNN was used together with regularization, we obtained a slight improvement in performance (96% accuracy). Our method of extracting the ROI specifies an adaptive threshold for each CUT based on the average intensity values of the reference cells in the 2D sliding window. Problems caused by busy backgrounds and specular lighting are well-known in the literature, but in this study, we report no impact of such factors on the performance of our algorithm. This work analysed the performance of several CNN architectures: Inception V3, DenseNet 121, DenseNet 201, MobileNet, VGG16, EfficientNetB0, LeNet, ResNet50 and our proposed shallow CNN.

In the future, the current findings of this work can be extended to: (i) Disease classification problems in other crops where yield is largely affected by crop disease, thus affecting the livelihoods of farmers, such as beans, bananas and cassava. (ii) Develop new algorithms for simultaneous multi-label leaf disease classification and quantification. Developing a robust method for quantifying leaf diseases with multiple symptoms is the next logical step. (iii) Validate the proposed CNN model on standard image processing datasets such as ImageNet and CiFar.