Next Article in Journal
Physical Fitness and the Level of Technical and Tactical Training of Kickboxers
Next Article in Special Issue
Epileptic Seizures Detection Using Deep Learning Techniques: A Review
Previous Article in Journal
Effects of a Home Literacy Environment Program on Psychlinguistic Variables in Children from 6 to 8 Years of Age
Previous Article in Special Issue
A Smart Service Platform for Cost Efficient Cardiac Health Monitoring
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Automatic Sleep-Stage Scoring in Healthy and Sleep Disorder Patients Using Optimal Wavelet Filter Bank Technique with EEG Signals

by
Manish Sharma
1,*,
Jainendra Tiwari
1 and
U. Rajendra Acharya
2,3,4
1
Department of Electrical and Computer Science Engineering, Institute of Infrastructure, Technology, Research and Management (IITRAM), Ahmedabad 380026, India
2
School of Engineering, Ngee Ann Polytechnic, Singapore 599489, Singapore
3
Department of Bioinformatics and Medical Engineering, Asia University, Taichung 41354, Taiwan
4
School of Management and Enterprise, University of Southern Queensland, Springfield 4300, Australia
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2021, 18(6), 3087; https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph18063087
Submission received: 30 December 2020 / Revised: 4 March 2021 / Accepted: 5 March 2021 / Published: 17 March 2021
(This article belongs to the Special Issue Application of Deep Learning for Neural Systems)

Abstract

:
Sleep stage classification plays a pivotal role in effective diagnosis and treatment of sleep related disorders. Traditionally, sleep scoring is done manually by trained sleep scorers. The analysis of electroencephalogram (EEG) signals recorded during sleep by clinicians is tedious, time-consuming and prone to human errors. Therefore, it is clinically important to score sleep stages using machine learning techniques to get accurate diagnosis. Several studies have been proposed for automated detection of sleep stages. However, these studies have employed only healthy normal subjects (good sleepers). The proposed study focuses on the automated sleep-stage scoring of subjects suffering from seven different kind of sleep disorders such as insomnia, bruxism, narcolepsy, nocturnal frontal lobe epilepsy (NFLE), periodic leg movement (PLM), rapid eye movement (REM) behavioural disorder and sleep-disordered breathing as well as normal subjects. The open source physionet’s cyclic alternating pattern (CAP) sleep database is used for this study. The EEG epochs are decomposed into sub-bands using a new class of optimized wavelet filters. Two EEG channels, namely F4-C4 and C4-A1, combined are used for this work as they can provide more insights into the changes in EEG signals during sleep. The norm features are computed from six sub-bands coefficients of optimal wavelet filter bank and fed to various supervised machine learning classifiers. We have obtained the highest classification performance using an ensemble of bagged tree (EBT) classifier with 10-fold cross validation. The CAP database comprising of 80 subjects is divided into ten different subsets and then ten different sleep-stage scoring tasks are performed. Since, the CAP database is unbalanced with different duration of sleep stages, the balanced dataset also has been created using over-sampling and under-sampling techniques. The highest average accuracy of 85.3% and Cohen’s Kappa coefficient of 0.786 and accuracy of 92.8% and Cohen’s Kappa coefficient of 0.915 are obtained for unbalanced and balanced databases, respectively. The proposed method can reliably classify the sleep stages using single or dual channel EEG epochs of 30 s duration instead of using multimodal polysomnography (PSG) which are generally used for sleep-stage scoring. Our developed automated system is ready to be tested with more sleep EEG data and can be employed in various sleep laboratories to evaluate the quality of sleep in various sleep disorder patients and normal subjects.

1. Introduction

Sleep is indispensable for maintaining optimal health and well-being. Getting adequate sleep is equally crucial like daily exercise and balanced diet. According to health professionals, few of many benefits of getting good sleep include elevated productivity, improved calorie regulation, better concentration, reduced heart disease risks, superior athletic performance, increased social and emotional intellect and averting depression [1]. Considering an average human taking a sleep of 7–8 h per day, almost one-third of our life is spent sleeping. Our health is largely impacted by the quality of sleep we get. Any sleep related disorder directly affects our mental, physical and social as well as emotional well-being. According to International Classification of Sleep Disorders, Second Revision (ICSD-2) [2], sleep disorders are widely categorized into major categories like insomnia, parasomnia, hypersomnia of central origin, sleep-related disorder, circadian rhythm sleep disorder, sleep-related movement disorders and other disorders which also includes those caused due to some medical or psychological conditions [3]. Among them, insomnia is found to be the most prevailing sleep disorder [4]. Insomnia is described as a condition where one finds it extremely difficult to fall asleep and/or stay asleep. Symptoms of insomnia include fatigue, daytime sleepiness, cognitive impairment, irritability, impulsiveness or aggression. The person lasting in this condition for more than 4 weeks is diagnosed as suffering from insomnia. Based on several studies conducted in countries like the USA (3161 patients), Canada (5622 patients) and the UK (2363 patients), the reported prevalence of insomnia is around 35–37% [5]. In general, around 25–30% of the total population experience symptoms of insomnia. If patients with co-morbid conditions are also considered, this number may reach above 50% based on the type and severity of disease [5]. Bruxism is a movement disorder that is characterized by involuntary grinding, gnashing or clenching of teeth. Bruxism is listed in the ICSD and is the third most common form of sleep disorder after sleep talking and snoring [6]. There are mainly two classes of bruxism namely awake bruxism and sleep bruxism which refer to bruxism during awake stage and sleep stage, respectively. Pervasiveness of awake bruxism and sleep bruxism in the adult population is about 20% and 80%, respectively [7]. Studies have shown that awake bruxism is more dominantly observed in females as compared to males, whereas there is no such bifurcation in case of sleep bruxism [8]. Narcolepsy is a long-term rapid eye movement (REM) sleep disorder listed in ICSD and is characterized by irresistible deep sleep attacks during daytime which often occurs with or without cataplexy and hypnagogic hallucinations [9]. The cataplexy refers to the sudden loss of muscle power. The cyclic alternating pattern (CAP) sleep database also contains forty patients suffering from nocturnal frontal lobe epilepsy (NFLE) which accounts for 37% of the complete database. The NFLE is a sleep disorder of heterogeneous etiology [10] and is mainly characterized by epileptic seizures occurring due to the frontal lobe mainly during night (nocturnal) sleep. There are no such epidemiological data available for prevalance of NFLE as many cases of NFLE are misdiagnosed as parasomnias, especially when children are considered [11]. The database contains 10 patients suffering from periodic leg movement (PLM). The PLM refers to sleep disorder involving repetitive and rhythmic flexing or jerking of legs for about 20–40 s over a certain interval of sleep duration. This disorder has more prevalence than epilepsy [12]. This database contains 22 patients suffering from REM behaviour disorder (RBD). It is a parasomnia which is characterized by lack of normal muscular tension and other abnormal behaviour like enactment of dream during REM sleep. A study conducted by Ohayon et al. [13] showed the estimated prevalence of RBD to be 0.5%. Majority of RBD patients happen to be elderly males with age between 40 and 70 years [14]. Traditional diagnostic procedure for the above mentioned sleep disorders include night-long polysomnographic (PSG) analysis. It is a time-consuming, labour-intensive process and prone to human errors as long hours of continuous evaluation are required. Thus, an automated classification of sleep stages will help to overcome these drawbacks. Sleep scoring is an effective indicator and can help in the detection of various sleep related disorders.
The sleep-stage scoring is mostly done as per the rules presented by Rechtschaffen and Kales in 1968 [15]. According to R & K rules, sleep is mainly categorized into two stages namely rapid eye movement (REM) sleep and non-rapid eye movement (NREM) sleep. NREM sleep stage accounts for 75–80% and REM sleep stage usually lasts for around 20–25% of total sleep duration. NREM stage is further categorized into four sleep stages namely stage-1, 2, 3 and 4. Thus, a total of five sleep stages are known (1, 2, 3, 4 and REM).
Several studies have been proposed for the automated sleep stages classification. Recently, Loh et al. [16] have presented an excellent review on the classification of sleep stages using deep learning (DL) methods. This study indicated that the CAP database has not yet been explored for sleep scoring using DL methods, despite the data containing a diverse variety of subjects, signals, sleep disorders and sizes. Boostani et al. [17] have published a review paper on sleep stage classification using various databases and different modalities including PSG and EEG. Zhu et al. [18] presented an automated sleep scoring system using the sleep-EDF database and visibility graphs with graph domain features. They obtained an overall accuracy of 87.5% using support vector machine (SVM) classifier in classifying six sleep stages. Kim et al. [19] used CAP database for sleep stage classification using heart rate variability (HRV) obtained from ECG signals obtained from 13 healthy subjects [20]. In addition, they did not consider a 6-class classification task, but used a binary classification problem. They applied the empirical mode decomposition (EMD) method for noise reduction in (HRV) detrended fluctuation analysis (DFA) and related the noise-reduced fractal property of HRV to the sleep stages of the subjects. Cui et al. [21] used the Institute of Systems and Robotics, University of Coimbra (ISRUC) sleep database and performed 5-class sleep stage classification using EEG, EOG and EMG channels and convolutional neural networks. Sharma et al. [22] performed six-class sleep stage classification by employing a three-band time-frequency localized (TBTFL) wavelet filter bank (FB) approach. They used an EEG channel of 100 Hz sampling rate from the sleep-EDF database and obtained an overall accuracy of 89.5% using SVM classification. However, they have not used a CAP database. Timplalexis et al. [23] carried out 5-class sleep stage classification using a combination of time- and frequency-based features and obtained an overall classification accuracy of 88.88% using an EBT classifier. Tripathi et al. [24] used dispersion entropy and bubble entropy features and a hybrid classifier. They used only 25 subjects among which six were healthy (H), seven were insomniac (Ins), one brux patient, one sleep-disordered breathing (SDB) patient and 10 REM-behaviour disorder (RBD) patients from the CAP database. In addition, they obtained an overall accuracy of 71.68% for 6-class sleep stage classification, which is significantly lesser than the model proposed by us in this study. Recently, Widasari et. al. [25] employed only 51 subjects from the CAP database (16 were healthy, nine were insomniac, four were suffering from SDB and 22 were suffering from RBD) in their study. In addition, they performed only 4-class (W, S1 + S2, S3 + S4, REM) sleep stage classification using sleep quality features and EBT classifier and achieved overall classification accuracy of 86.27%. It is to be noted that all the previous studies have used only unbalanced data and we are the first group to use the balanced dataset in this study. A summary of the state-of-the-art automated sleep stage classification studies conducted is given in Table 1.
In this work we have proposed an automated sleep stage classification system using multi-level wavelet decomposition and norm-based feature extraction, followed by classification using various supervised classifiers. The EEG signals of healthy and sleep disorder patients are fed as input to the automated system to obtain the sleep stages scoring. We have used only one or two EEG channels, hence practical installation is simple and easier as compared to other state-of-the-art techniques, which used PSG [26,27] or several EEG channels and other physiological signals for automated sleep-stage scoring. Subject’s comfort level is also improved as compared to sleep scoring using multi-modal signals.

2. Material Used

The EEG dataset used in this study was taken from openly accessible physionet’s CAP sleep database [28,29]. The sleep scoring was done by a team of sleep experts of the Sleep Disorders Centre of the Ospedale Maggiore of Parma, Italy.
The CAP sleep database consists of 108 PSG (polysomnographic) recordings registered at the Sleep Disorders Centre of the Ospedale Maggiore of Parma, Italy. It contains atleast three EEG (Electroencephalogram) channels ( C3 or C4, F3 or F4 and O1 or O2, with reference to A1 or A2), electrooculogram (EOG) channels, submentalis muscle Electromyogram (EMG), respiration signals, bilateral tibial EMG and one electrocardiogram (EKG). The EEG channels additionally includes bipolar channels like F3-C3, Fp1-F3, P3-O1, C3-P3, C4-P4, Fp2-F4, P4-O2 and F4-C4. Among all these channels, two EEG channels namely F4-C4 (bipolar) and C4-A1 (unipolar) are considered for this study as they are present in the maximum number of PSG recordings in this database. The CAP sleep database includes healthy subjects and patients having seven different kinds of sleep disorders such as insomnia, bruxism, narcolepsy, NFLE, PLM, RBD and SDB. The age of subjects varies in the range of 14–82 years, and their average age is around 45 years. A total of 61% of the subjects are men (66 people), and 38% are women (42 people). Most of the studies conducted using the CAP sleep database are on cyclic phase detection [30,31,32,33,34,35]. There are many studies on sleep stage detection using other datasets but there is no study available in literature on sleep stage classification using the CAP sleep database.
Out of 108 subjects, 16 are completely healthy, 40 are diagnosed with nocturnal frontal lobe epilepsy (NFLE), 22 are troubled by REM behaviour disorder, 10 by periodic leg movement (PLM), nine are insomniac, five were narcoleptic, four are facing sleep-disordered breathing and two are diagnosed with bruxism [36]. Sampling frequencies of EEG signals varied between 100 Hz and 512 Hz. A total of 80 subjects with EEG recordings sampled at 512 Hz, as mentioned in Table 2, are considered for this study (48 are male and 32 are female subjects). These subjects are taken on the basis of availability of F4-C4 and C4-A1 EEG channels with sampling frequency of 512 Hz.
The sleep scoring was provided by trained experts, in accordance with the Rechtschaffen and Kales rules [15]. Different stages were annotated as W for wake, S1–S4 for NREM sleep stages and R for REM (Rapid Eye Movement) stage. The details of the total number of epochs of individual stages for all types of patients are shown in Table 3.

Balancing the Dataset

Intervals corresponding to various sleep stages vary from one person to another person. An average adult’s one-night sleep consists around 2–5% of total time in S1 stage, 45–50% of total time in S2 stage, 5–10% in S3 and S4 stage, and remaining 20–25% time in the REM stage [22,37]. As a result, sleep stage recordings inherently contain unbalanced data and require data balancing in order to yield better models which provide unbiased and robust classification results. In this work, we have also balanced all the data subsets using over-sampling and under-sampling techniques. It can be clearly observed from Table 3 that the number of epochs corresponding to S1 sleep stage constitute only 4.36% of total number of epochs whereas S2 sleep stage constitute around 35.50% of complete dataset. Thus, in order to carry out an unbiased classification, number of epochs corresponding to S1 are increased to 16.67% of total epochs using over-sampling by replacement method. At the same time, epochs corresponding to S2 stage are decreased to 16.67% of total epochs using under-sampling technique. Similarly, all other sleep stages are also brought to 16.67% of total epochs by either over-sampling or under-sampling as per the requirement. Thus, the number of epochs in all six classes are made equal in proportion to perform better classification. The number of epochs used in various classes for different data subsets is shown in Table 4.

3. Methodology

The flow diagram of the proposed method is given in Figure 1. We have explained the data acquired and method used in the following subsections.

3.1. Data Acquisition: Acquiring PSG Recordings

As already mentioned in the previous section, data of 80 subjects including six healthy and 74 subjects with sleep disorders were downloaded from physionet’s CAP sleep database. We created a total of 10 data subsets namely ‘healthy’, ‘insomnia’, ‘bruxism’, ‘narcolepsy’, ‘NFLE’, ‘PLM’, ‘RBD’, ‘SDB’, ‘all disordered’ and ‘all subjects combined’. We performed sleep stage classification on each of these data subsets. For each of these data subsets, a matrix containing epochs corresponding to all six sleep stage was formed. The description of these data subset is given below:
  • Healthy: This data subset contains sleep stage epochs of only healthy subjects with no significant pathology reported. Six healthy subjects are taken in this data subset with a total of 6063 epochs of 30 s duration each.
  • Insomnia: In this data subset, sleep stage epochs of seven patients suffering from insomnia disorder are taken with a total of 8551 epochs.
  • Bruxism: This data subset represents 427 epochs obtained from one patient suffering from bruxism sleep disorder.
  • Narcolepsy: It contains a total of five patients with 5614 epochs of 30 s duration each.
  • NFLE: This data subset contains 26,883 epochs corresponding to 27 patients suffering from nocturnal frontal lobe epilepsy (NFLE) sleep disorder.
  • PLM: (Periodic leg movement disorder) This data subset contains 7574 epochs of 30 s duration obtained from 9 PLM patients.
  • RBD: (Rapid eye movement (REM) behaviour disorder) In this data subset, 22,676 epochs are obtained from 22 RBD patients.
  • SDB: (Sleep disordered breathing) A total of 2879 epochs of 30 s duration are obtained from three patients with SDB sleep disorder.
  • All Disordered: This data subset is comprised of 74,574 epochs obtained by combining all 74 disordered subjects.
  • All subjects combined: This data subset contains 80,667 epochs obtained by combining all types of 80 subjects considered in this study. It includes healthy subjects as well as all disordered subjects.
Each PSG recording contained several types of signals including EEG, EOG, EMG and ECG. We have used two channels F4-C4 and C4-A1 EEG signals individually and jointly.

3.2. Segmentation of Sleep Stages into 30 s Epochs

With the help of hypnogram, segregation of different sleep stages present in the EEG recordings was done as per the R & K criteria [15]. Thus, each epoch has been labelled as wake, S1, S2, S3, S4 and REM.

3.3. Wavelet Filtering

There are multiple applications of two-channel filter banks (FB) in various fields like analysis of biomedical signals, image processing and communication [22,38,39,40,41,42,43,44]. The FB used in this work is designed in using an Eigenfilter-based approach. The FB is designed by optimizing an objective function and a multiple objective function. The objective function is a convex quadratic function of errors in the pass-band and stop-bands, joint bandwidth duration localization. We used a linear phase optimal biorthogonal wavelet filter bank (OBWFB) in which the analysis filter is a halfband filter. While designing the filter bank, the first step was to design a halfband analysis lowpass filter (HALF) formulating a linearly constrained convex quadratic optimization problem that employed an objective function having a convex combination of passband and stopband errors and bandwidth-duration concentrations of the filter. After designing the half-band analysis filter (HALF), the next step is to design the synthesis lowpass filter (SLPF) in a manner similar to designing HALF but with some variations like (i) avoiding SLPF to be constrained as halfband filters, and (ii) along with the regularity conditions, so that the perfect reconstruction conditions is satisfied.

3.4. Wavelet Decomposition

Five level one-dimensional wavelet decomposition of each epoch is done using an optimal biorthogonal wavelet filter as explained above. The five level of decomposition produced six different sub-bands of EEG epochs.
These sub-bands are later used to compute discriminating features.

3.5. Extraction of l 1 , l 2 and l Norm

The discriminating features used for classifying six different classes (W, S1, S2, S3, S4 and REM) are l 1 n o r m , l 2 n o r m and l n o r m . The l m n o r m [45] of any discrete-time signal u[n] is defined as
u m = n = 1 | u [ n ] | m 1 m , m Z +
In this study, we used m = 1 and m = 2 .

l n o r m or Peak Absolute Value

The l n o r m of a signal gives the maximum absolute value among all the samples of a discrete-time signal. Thus, it is also known as peak absolute value.
u = | u | m a x
Thus, we get a total of 18 features after combining all three norms.

3.6. Classification and Validation

Classification of all six stages is performed using all extracted norm-based features. These norm-based features are fed to all the available supervised machine learning classifiers namely decision trees [46,47], logistic regression [48], naive bayes [49], support vector machines (SVM) [46,50], K-nearest neighbours (KNN) [51], ensemble bagged trees (EBT) [52], classifiers and discriminant analysis [53] to select the optimum performing classifier. All the classifiers are developed using a 10-fold cross-validation strategy.
Among all classifiers mentioned above, EBT classifier has yielded optimum performance. EBT classifier is a combination of bagging algorithm and decision tree classifier [52]. In the bagging algorithm, several subsets of data from the training sample are chosen randomly with replacement. Now, each collection of subset data is used to train their decision trees. As a result, we end up with an ensemble of different models and improve classification performance and reduce over-fitting [54]. Averages of all the predictions from different trees are used which is more robust than a single decision tree. Bagging algorithm is used to reduce variance of decision tree.
In order to optimize hyper parameters for each of ten classification tasks, we observed the mis-classification error rate by varying the number of splits and maximum number of trees. The maximum number of trees is varied in the range of 10 to 250, number of splits is varied in the range of 1 to n 1 , in the steps of 100 from 1 to n 1 , (where n is the total number of epochs) and then we drew a graph between mis-classification rate versus number of trees (Figure 2). We obtained the converging performance for the split equal to ( n 1 ). The parameters corresponding to minimum error are chosen as the optimal parameters. Figure 2 shows sample EEG plot of tuning of hyper parameters of EBT classifier. The figures also shows the optimal values for number of learners = 65, number of splits = 8550 and learning rate = 1.

4. Results

In this work, we have proposed an automated classification of sleep stages using EEG channels. Six stages of sleep, namely: wakefulness (W) stage and five sleep stages namely S1, S2, S3, S4 and REM are classified for normal and abnormal sleep patients. This work was performed on a machine having 8 GB RAM and AMD Ryzen-5 3550 H processor with MATLAB R2020a (version 9.8.0.1323502) installed in it. For classification, we used EEG recordings of six healthy subjects with a total of 6063 epochs (30 s each) and 74 sleep disorder patients which yielded a total of 74,604 epochs (of 30 s each). Thus, a total of 80,667 epochs (of 30 s each) are used for this study. A detailed summary related to number of epochs of different sleep stages and wake (W) of all subjects can be found in Table 3.
We have analysed the features extracted using analysis of variance (ANOVA) technique with Fisher’s least significant difference post-hoc test [55]. The p-values obtained from ANOVA test indicate the clinical significance of features. The ranking (of features), p-values, mean and standard deviation of all six channels obtained using bipolar (F4-C4) and unipolar channels (C4-A1) are shown in Table 5 and Table 6, respectively. It can be noted from these features that p-values corresponding to each feature are almost zero, which indicates that all the features considered in the classification task are statistically significant. As the number of features are only 18 and p-values are almost zero, we have not included any features selection algorithm.
One-way ANOVA [56] shows whether one or more group on which it is performed have any statistical differences based on their means. One-way ANOVA tests the null hypothesis which says that the statistical mean of all groups in consideration is equal. If the one-way ANOVA returns a statistically significant result, then we reject the null hypothesis and accept the alternate hypothesis which says that the statistical mean of all groups in consideration are not equal and hence it provides the evidence of difference. Here, we have used one-way ANOVA with confidence level of 95% and we observe that the null hypothesis is rejected and alternate hypothesis is accepted. However, one-way ANOVA does not signify which particular group is different from the other. Therefore, we used a post-hoc Fisher’s lease significant difference (LSD) [57] test to see which group differs from other and by what margin. We used l 1 n o r m feature of sub-band 1 present in ‘all subjects combined’ dataset and observed the difference in the statistical mean of all six classes as shown in Figure 3.
Sleep recordings inherently contained unbalanced data due to the varying duration of different sleep stages during night sleep. This may result in unequal number of epochs in different sleep stages. As a result, we may get biased and improper sleep stage classification results in favour of the sleep stage having the maximum number of epochs. To avoid this, we have also balanced each data subset using over-sampling (by replacement) and under-sampling techniques. In the over-sampling (by replacement) method, we randomly selected a few epochs from already available epochs and replicated them to increase the epoch count and make it comparable to the epoch count of other classes. Thus, each classification task mentioned above is carried out in two phases as mentioned below:
(i)
Classification without balancing the number of epochs among sleep stages.
(ii)
Classification after balancing the number of epochs among sleep stages.
We collected sleep data corresponding to both unipolar (C4-A1) and bipolar (F4-C4) EEG channels and performed classification individually as well as after combining both channels. Unipolar EEG channel yielded better classification accuracies as compared to the bipolar channel. However, a combination of both channels (F4-C4 + C4-A1) performed better than individual channels. The classification results are obtained using EBT with 10-fold CV. In 10-fold CV, a complete database is segmented into 10 equal folds and the training is done using nine folds and the remaining one fold is used for testing. Thus, the classification is done in 10 iterations, and then average of 10 folds is considered as the average accuracy. To ensure the robustness of the model, the classification using 10-fold CV is repeated five times for each of 10 datasets and the average of accuracies obtained in each run is then taken as the final measure of classification performance. The details of classification measures corresponding to each of five trials of 10-fold CV and their average are presented in Table 7. Table 8 summarizes the results obtained for all individual data subsets and both EEG channels individually and with the combination of the two channels.
Detailed descriptions corresponding to each classification are given below. In classification task 1, we started with sleep stage classification of only healthy subjects with an unbalanced epoch distribution containing 6063 epochs. It yielded an average classification accuracy of 78.3% using the EBT classifier. The confusion matrix obtained for the classification of healthy subjects is shown in Table 9. The individual classification accuracies obtained for various sleep stages namely: W, S1, S2, S3, S4 and REM are 95.84%, 94.94%, 85.87%, 92.4%, 96.59% and 91.95%, respectively. Generally, in classification problems, the accuracy rate (ACC) is used to compare the performance of studies in this domain, but this metric is accurate if the number of observations is equal to classes. It can be clearly seen in Table 3 that the number of observation among different classes are not equal. Therefoe, ACC is not the best metric to evaluate such a system. Hence, in recent years, Cohen’s Kappa coefficient ( κ ) is used to evaluate such systems. As a rule of thumb, the value of κ in the range of 0.75 to 1 is considered an excellent classification, κ in between 0.4 and 0.7 is interpreted as fair to good agreement and κ below 0.4 is said to be poor classification agreement [58]. The value of κ = 0.7212 ± 0.0069 is obtained for healthy subjects. The unbalanced healthy data subset had 6063 epochs for S1 and S2 sleep stages with 280 and 2172 epochs, respectively. In order to make the epoch count in all six classes equal, we increased the number of epochs in S1 stage from 280 to 1000 by randomly shuffling and replicating random epochs from the already available 280 epochs of S1 stage. For the same reason, we randomly removed 1172 epochs from the already available 2172 S2 stage epochs to make the epoch count equal to 1000. Similarly, all other stages are also balanced in the same manner. After balancing, the overall classification accuracy is improved from 78.3% to 87.9%. The confusion matrix obtained for the balanced healthy data using the EBT classifier with 10-fold CV can be seen in Table 10. The confusion matrices also included a column corresponding to F1 score for all six classes.
We also have data subsets belonging to seven types of sleep disorders, namely insomnia, bruxism, narcolepsy, NFLE, PLM, RBD and SDB. In CT-2, we started with sleep stage classification of insomniac subjects containing 8551 epochs (30 s) in total. It yielded an average classification accuracy of 85.4% using the EBT classifier. The confusion matrix obtained after classification of insomniac subjects is shown in Table 11. Stages like W, S1, S2, S3, S4 and REM yielded an accuracy of 93.94%, 97.1%, 89.5%, 95.92%, 98.27% and 95.97%, respectively. The value of Cohen’s Kappa coefficient is found to be 0.7867 ± 0.0056, which is in good agreement. After balancing, 8400 epochs out of 8551 are taken from insomnia data subset leading to 1400 epochs in each class. This led to the improvement in the classification accuracy from 85.4% to 92.8%. The confusion matrix corresponding to the balanced insomnia data is shown in Table 12.
In CT-3, we have analysed bruxism sleep disorder subjects. The CAP has only two subjects with bruxism and only one out of two has a sampling frequency of 512 Hz for EEG channel. Hence, we considered only one bruxism subject with 427 sleep stage epochs (30 s) corresponding to bruxism. We obtained an average classification accuracy of 66.7% using the EBT classifier with κ of 0.5578 ± 0.0297. The confusion matrix obtained after classification of bruxism subject is shown in Table 13. The sleep stages of W, S1, S2, S3, S4 and REM yielded an accuracy of 88.06%, 88.99%, 79.86%, 87.82%, 90.4% and 96.49%, respectively. The lower values of overall classification accuracy and κ are due to less number of sleep stage epochs. In bruxism data subset, 426 epochs are considered for balancing which led to the increase in classification accuracy from 66.7% to 82.4%. The confusion matrix for the same can be seen in Table 14.
In CT-4, we analysed five subjects belonging to narcolepsy sleep disorder which yielded a total of 5614 sleep stage epochs with unbalanced epoch distribution among six classes. We obtained an overall classification accuracy of 79.3% using the EBT classifier and κ = 0.7301 ± 0.0070. The confusion matrix obtained after the classification is shown in Table 15. The six sleep stages, namely W, S1, S2, S3, S4, REM yielded an accuracy of 93.73%, 94.58%, 87.41%, 93.82%, 97.27% and 91.5%, respectively. After balancing the epochs, a total number of 5610 sleep epochs are used for classification leading to 935 epochs in each sleep stage. After balancing, its classification accuracy increased from 79.3% to 88.2%. The confusion matrix corresponding to the balanced narcolepsy data is shown in Table 16.
In CT-5, we processed 27 NFLE subjects yielding a total of 26,883 epochs which forms one-third of the complete dataset. We observed an average classification accuracy of 77.5% using the EBT classifier and κ = 0.6914 ± 0.0035 for the unbalanced NFLE data subset. The confusion matrix for the same can be seen in Table 17. Individual stages, namely W, S1, S2, S3, S4, REM yielded an accuracy of 95.16%, 96.29%, 84.05 89.81%, 95.69% and 93.3%, respectively. After balancing, from a total of 26,880 epochs 4480 epochs are considered for each of the six classes. Then the classification accuracy is increased from 77.5% to 86.6%. The confusion matrix for the balanced NFLE data is shown in Table 18.
The CT-6 which includes PLM subjects has obtained an overall classification accuracy of 78.0% using the EBT classifier and κ coefficient = 0.7296 ± 0.0061. The confusion matrix for the same is shown found in Table 19. Sleep stage namely W, S1, S2, S3, S4, REM yielded individual classification accuracy of 94.35%, 95.84%, 86.55%, 90.22%, 95.97% and 95.48%, respectively. For the PLM data subset, the number of epochs in each class is made equal to 1262 and for this balanced dataset, we observed the classification accuracy reaching 85.8% from 78.0%. The confusion matrix for the balanced PLM data classification is shown in Table 20.
In CT-7, all 22 RBD subjects present in the CAPSLPDB are taken for this study yielding a total of 22,676 sleep stage epochs. Overall classification accuracy of 71.9% is obtained using the EBT classifier and κ is found to be 0.6372 ± 0.0039. The confusion matrix for the same is shown in Table 21. Sleep stage namely W, S1, S2, S3, S4, REM yielded individual classification accuracy of 92.26%, 96.3%, 80.89%, 89.88%, 95.65% and 89.23%, respectively. During balancing of epochs in the RBD data subset, the number of epochs in each sleep stage is made equal to 3779. It increased the classification accuracy form 71.9% to 81.0% and the confusion matrix for the same is shown in Table 22.
CT-8, which analysed SDB subjects, obtained an overall classification accuracy of 74.3% using the EBT classifier with κ of 0.6276 ± 0.0117. The confusion matrix for the same is presented in Table 23. Sleep stages W, S1, S2, S3, S4, REM yielded individual classification accuracy of 91.28%, 89.89%, 82.81%, 92.01%, 95.97% and 96.28%, respectively. The balancing of data resulted in 480 epochs in each sleep stage. As a result, we obtained an overall classification accuracy of 86.9% for the balanced data and the confusion matrix is shown in Table 24.
After processing all seven types of disorders individually in previous CTs, we combined them in a single data subset named all disordered which is a combination of 74 patients with sleep disorders with 74,604 epochs of 30 s duration each with unbalanced epoch distribution among sleep stages. After classification of this data subset, we observed an overall accuracy of 75.6% using the EBT classifier and κ = 0.6780 ± 0.0021. The confusion matrix for the same classification is shown in Table 25. Individual sleep stages namely W, S1, S2, S3, S4, REM yielded an accuracy of 93.22%, 95.92%, 83.13%, 90.87%, 96.19% and 91.94%, respectively. We have also performed similar balancing on the ’all disordered’ data subset which contained epochs of all seven types of disordered subjects and considered the number of epochs in each of six sleep stages equal to 12,500 with total of 75,000 epochs for all six classes. Balancing operation on this data subset improved the overall classification accuracy to 84.8% from 75.5% for the same classifier. The confusion matrix for the balanced data is shown in Table 26.
Lastly, epochs corresponding to six healthy subjects are also added in the disordered data subset and a new data subset namely all combined is formed for CT-10, which contained a total of 80,667 sleep stage epochs corresponding to all types of subjects. For unbalanced data, classification yielded an overall accuracy of 75.5% using the EBT classifier and κ = 0.6697 ± 0.0020. The confusion matrix corresponding to this classification is shown in Table 27. Individual sleep stages, namely W, S1, S2, S3, S4 and REM yielded an accuracy of 93.07%, 95.89%, 82.56%, 90.81%, 96.11% and 91.48%, respectively. While balancing, the number of epochs in each sleep stage is made equal to 14,000 with total of 84,000 epochs for all six stages. The classification of balanced data yielded an accuracy of 85.1% and the confusion matrix for the same is shown in Table 28. Table 29 represents the epoch distribution and obtained accuracies for ‘healthy’, ‘all disordered’ and ‘all combined’ data subsets.
Table 29 and Table 30 quickly summarize the epoch distribution and obtained accuracies for ‘all subjects combined’, ‘healthy’ and ‘all disordered’ datasets. All the accuracies mentioned below are obtained from the EBT classifier.
The receiver operating characteristic (ROC) curves are generally plotted for the binary classification problem. Since, the classification in this work involves six-class classification tasks, we have drawn ROC taking one class at a time as positive class and the remaining five classes as negative class. In Figure 4, we have shown the ROC plot corresponding to the whole database. From the figure it is clear that the area under the curve varies between 0.80 to 0.94 which indicates the effectiveness of the proposed model in discriminating the sleep stages.

5. Discussion

The results obtained by our study indicate that the proposed model achieved high classification performance using EEG signals with unbalanced and balanced sleep datasets. This study is the first to attempt 6-class sleep stage classification using single/dual channel EEG signals with high sampling frequency of 512 Hz. In addition, this is the first study to consider the sleep stages of different sleep disorders such as insomnia, bruxism, narcolepsy, NFLE, PLM, RBD and SDB. In this work, we presented the results of unipolar and bipolar EEG channels individually as well as combined. The EEG epochs are decomposed into sub-bands using a new class of optimized wavelet filters. Different optimal wavelet filters of varying lengths and vanishing moments with different levels of decomposition have been used to obtain optimum performance. Two different EEG channels, namely F4-C4 and C4-A1, along with their combination have been used to obtain deeper insights. Wavelet-based norm features of six sub-bands have been computed and fed to various classifiers including EBT to choose the optimum performing classifier.
The PSG is considered the gold standard to score sleep stages and diagnose sleep disorders. The PSG-based techniques require multiple wired sensors to record the activities of multiple physiological signals (such as EEG, EMG, ECG, EOG, respiratory signals) and time-consuming analysis procedures. Moreover, the sleep recordings need to be conducted overnight in a specialized sleep laboratory or hospital. Further, sometimes PSG recording process may cause inconvenience to older people who often suffer from sleep disorders. Hence, it is desirable to explore some new techniques and methods that can produce accurate results similar to manual sleep staging PSG-based methods, which are simple, less time-consuming, inexpensive and convenient to the patients. Our proposed study is a humble attempt in that direction which needs to be tested independently with more diverse and bigger databases or cohorts.
We perform the wavelet processing and feature extraction using single channel or dual channel EEG epochs of duration 30 s only (instead of 1 min or higher duration). Due to this simplification, the proposed method has low computational cost and therefore it that can be implemented in an embedded hardware device.
We have considered the classification of six classes, whereas other studies [19,25] considered two class or three-class classification tasks. The most challenging task is classification of subclasses of NREM (i.e., S1–S4). The classification of S1–S4 is missing in the existing literature but we got very good accuracy for the classification of subclasses of NREM also. The most challenging task in sleep stage classification is to score S1 stages accurately and we have attained reasonable accuracy in identifying the S1 stage along with S2–S3.
Novelty of the work is that the model developed can be used for sleep-scoring of not only good sleeper but can also be used for the subjects suffering from various sleep disorders in automated speedy fashion without much complexities. There are seldom any study which focus on sleep-scoring of sleep disorder subjects. Since in the CAP data mostly subjects are elderly, the model can be further used and tested for identifying sleep disorders in the elderly. An accurate sleep-stage scoring of sleep disordered patients will help in diagnosis and prognosis of disorders, which is much needed and highly desirable for elderly persons.
We have used a novel optimal filter bank. We have used linear phase optimal biorthogonal wavelet filter banks (OBWFB) in which the analysis filter is a halfband filter [59]. The halfband pair filter bank (HPFB) design technique introduced by Phoong et al. [60] and its other variants [61] are all indirect approaches for filter design. These technique have restrictions like lack of control on frequency responses, joint bandwidth-duration localization and control between frequency selectivity and smoothness of filters [62,63,64]. In order to overcome these limitations, we used a filter which is designed using a direct, time-domain approach that avoids the need for the design of intermediate kernels. Unlike the Phoong et al. and Tay et al. [60,61] HPFB technique, the design technique of our linear phase optimal biorthogonal wavelet filter bank is simple and efficient to control the smoothness, frequency selectivity and joint bandwidth-duration localization of filters [65,66,67]. It is to be noted that the analysis filter of the wavelet filter bank used is a half band filter in which half of the filter coefficients are zero; hence, the computational cost of finding sub-bands using the proposed wavelet filter is exactly half of the standard Daubechies wavelet used in the literature [42,68,68]. Moreover, in designing the (HALF), we need not design intermediate kernels unlike other methods [60,61]. Thus, our design method has lower design complexity also.
The notable aspect of this work is that the publicly available CAP sleep database containing normal and seven different types of sleep disorders are used for the first time to develop an automated sleep scoring system. In this work, we have also balanced all the data subsets using over-sampling and under-sampling techniques. In this work, epochs corresponding to minority classes (example: S1) are increased by oversampling to make them proportion to other classes. At the same time, epochs corresponding to classes having large portion of total distribution are brought down by under-sampling. Thus, epochs in all six classes are made equal in proportion to make efficient classification. To develop a robust model and avoid possible over fitting, we have also balanced the data as original data are unbalanced and we obtained the results using both balanced and unbalanced data. To the best of our knowledge, this is the first study to use the balanced CAP database.
It is observed that the unipolar channel C4-A1 performed better for the classification of healthy as well as all seven disordered classes. Thus, we can conclude that single channel (unipolar channel) performed better than bipolar channel. When both the channels are combined then the performance of classification has improved. Since we used only single/dual channel EEG signals instead of complex multichannel multimodal PSG recording, the system complexity is low. Further, we employed EEG epochs of duration 30s only (instead of 1 min or higher duration). Hence, the proposed method has a low computational cost and therefore it can be implemented in an embedded hardware device.
The key features of this study are as follows:
  • To the best of our knowledge, this is the first study to use the whole CAP database that includes 80 subjects with seven different sleep disorders (insomnia, bruxism, narcolepsy, NFLE, PLM, REM, RDB and SBD as well as normal subjects). We have used the highest number of epochs (80,667) in this study which is larger than most of the existing studies. In the existing literature [19,21,25], studies have used only a few healthy subjects.
  • A simple, fast and accurate automated sleep stage detection system is developed.
  • The proposed model has attained high classification performance for all 10 classification tasks considered in this study.
  • To the best of our knowledge this is the first study to perform sleep stage classification of all sleep disorders using the CAP sleep database.
  • As compared to previous studies on sleep stage classification, we have used more data containing sleep stage recordings of 80 subjects with 80,667 sleep stage epochs of 30 s duration each.
  • This is the first study to employ machine learning coupled with optimal wavelet filter bank for sleep scoring detection using EEG signals with a high sampling frequency of 512 Hz.
  • We have employed a new class of optimal wavelet filters to extract the norm-based features of EEG channels.
  • We have used only two EEG channels and extracted norm-based features which makes it simpler and computationally efficient.
The limitations of this are as follows:
  • Placement of EEG electrodes on the human skull is a complex task and sometimes it may even cause discomfort to the subjects. Hence, in this work, we have used only two electrodes.
  • We used the EEG signals of subjects sampled at 512 Hz sampling frequency, hence we had to eliminate the use of the other 28 subjects (not sampled at 512 Hz). Therefore, we finally used only 80 subjects for this study.
  • We obtained the least classification accuracy for S1 sleep stage as the number of data available are minimum using the unbalanced database. However, the performance accuracy of S1 sleep stage is comparable with other sleep stages using balanced database.
  • Computation of wavelet-based features takes more time than the ordinary statistical features. However, the same wavelet filter also helps to remove the noise.
  • The CAP database has been sleep scored according to the R & K criterion, in which sleep is classified into six stages. Therefore, in the proposed study, we have considered the six-class classification task. However, as per the American Academy of Sleep Medicine (AASM) guidelines for sleep scoring, stages S3 and S4 are combined into a single stage called N3. Thus, as per AASM guidelines, whole-night sleep is divided into five sleep stages: wakefulness (W), N1, N2, N3, and REM instead of six stages as defined by the R & K criterion. This limitation can be overcome by combining stages S3 and S4 into the new stage N3 and presenting the results as per AASM guidelines and not according to R & K rules.
The proposed model achieved high Cohen’s Kappa coefficient values (more than 0.65) for both unbalanced and balanced sleep datasets. In the future, we intend to evaluate the performance of our model with more sleep EEG data and install the developed model in the cloud to get accurate diagnosis of the type of sleep disorder immediately.

6. Conclusions

In this work, we have proposed an automated sleep stage classification system using two EEG channels: unipolar (C4-A1) and bipolar (F4-C4). We have used EEG signal recordings of 80 subjects consisting of six healthy subjects and 74 patients suffering from any one of seven sleep disorders, namely insomnia, bruxism, narcolepsy, NFLE, PLM, RBD and SDB. After segmenting each EEG signal into multiple 30 s epochs corresponding to six different classes (wake, S1, S2, S2, S4, REM), 5-level 1-D wavelet decomposition of each epoch is done. Then the norm-based features are extracted from each EEG channel. The duration of various sleep stages vary resulting in unbalanced data with unequal EEG epochs in six classes. In order to avoid bias and obtain better classification results, we performed data balancing using over-sampling and under-sampling techniques and obtained balanced data with almost equal epoch distribution among all six classes.
Our proposed method attained the maximum accuracy of 75.6% and Cohen’s Kappa coefficient of 0.6780 ± 0.0021 for unbalanced data while 85.1% accuracy with Cohen’s Kappa coefficient of 0.8214 for balanced data using the EBT classifier with ten-fold cross-validation strategy. The classification performance of the proposed model indicates that it can reliably classify the sleep stages using 30-s duration with single or dual channel EEG instead of using multichannel multimodal PSG.
As this method is based on only two EEG channels, practical setup is also easier to implement. It helps the sleep experts to devote more time and effort on sleep scoring. In the future, we intend the evaluate the performance of our developed model with more sleep EEG data and use it as cloud-based AI system to detect the sleep disorders immediately. In the future, we intend to extend our study on the CAP database to classify different sleep disorders such as insomnia and narcolepsy using EEG and ECG signals. We also plan to use our developed model for automated sleep stage classification using EOG and HRV signals.

Author Contributions

M.S.: Conceptualization, Methodology, Software, Writing- Original draft preparation, Visualization, Investigation, Editing; J.T.: Software, Validation, Reviewing and Editing; U.R.A.: Supervision, Reviewing and Editing. All authors have read and agreed to the published version of the manuscript.

Funding

We have not received any funding.

Institutional Review Board Statement

Not applicable. We do not have any review board at our Institute.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data used in this work is open-source and publicly available on PhysioNet.

Conflicts of Interest

All the authors confirm that they do not have any conflict of interest for the proposed work.

References

  1. Fletcher, J. Why Sleep is Essential for Health. 2019. Available online: https://www.medicalnewstoday.com/articles/325353/ (accessed on 25 April 2020).
  2. American Academy of Sleep Medicine. International Classification of Sleep Disorders. 2005, pp. 51–55. Available online: https://ci.nii.ac.jp/naid/20001061569/ (accessed on 3 May 2020).
  3. Thorpy, M.J. Classification of sleep disorders. Neurotherapeutics 2012, 9, 687–701. [Google Scholar] [CrossRef] [PubMed]
  4. Sharma, M.; Dhiman, H.S.; Acharya, U.R. Automatic identification of insomnia using optimal antisymmetric biorthogonal wavelet filter bank with ECG signals. Comput. Biol. Med. 2021, 104246. [Google Scholar] [CrossRef]
  5. Bhattacharya, D.; Sen, M.; Suri, J. Epidemiology of insomnia: A review of the global and Indian scenario. Ind. J. Sleep Med. 2013, 8, 100–110. [Google Scholar] [CrossRef]
  6. Murali, R.; Rangarajan, P.; Mounissamy, A. Bruxism: Conceptual discussion and review. J. Pharm. Bioallied Sci. 2015, 7, S265. [Google Scholar] [CrossRef] [PubMed]
  7. Glaros, A.G. Incidence of diurnal and nocturnal bruxism. J. Prosthet. Dent. 1981, 45, 545–549. [Google Scholar] [CrossRef]
  8. Bader, G.; Lavigne, G. Sleep bruxism: An overview of an oromandibular sleep movement disorder. Sleep Med. Rev. J. Ind. Prosthodont Soc. 2000, 10, 141–148. [Google Scholar] [CrossRef] [PubMed]
  9. Daniels, L.E. Narcolepsy1. Medicine 1934, 13, 1–122. [Google Scholar] [CrossRef]
  10. Nobili, L.; Proserpio, P.; Combi, R.; Provini, F.; Plazzi, G.; Bisulli, F.; Tassi, L.; Tinuper, P. Nocturnal frontal lobe epilepsy. Curr. Neurol. Neurosci. Rep. 2014, 14, 424. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  11. Scheffer, I.E.; Berkovic, S.; Bhatia, K.; Fish, D.R.; Marsden, C.D.; Lopes-Cendes, I.; Andermann, F.; Andermann, E.; Desbiens, R.; Cendes, F.; et al. Autosomal dominant frontal epilepsy misdiagnosed as sleep disorder. Lancet 1994, 343, 515–517. [Google Scholar] [CrossRef]
  12. Natarajan, R. Review of periodic limb movement and restless leg syndrome. J. Postgrad. Med. 2010, 56, 157. [Google Scholar] [CrossRef]
  13. Ohayon, M.M.; Caulet, M.; Priest, R.G. Violent behavior during sleep. J. Clin. Psychiatry 1997, 58, 369–376. [Google Scholar] [CrossRef]
  14. Boeve, B.F. REM sleep behavior disorder: Updated review of the core features, the REM sleep behavior disorder-neurodegenerative disease association, evolving concepts, controversies, and future directions. Ann. N. Y. Acad. Sci. 2010, 1184, 15–54. [Google Scholar] [CrossRef] [Green Version]
  15. Rechtschaffen, A. A Manual for Standardized Terminology, Techniques and Scoring System for Sleep Stages in Human Subjects. Brain Inf. Serv. 1968. Available online: https://ci.nii.ac.jp/naid/10027491188/ (accessed on 25 April 2020).
  16. Loh, H.W.; Ooi, C.P.; Vicnesh, J.; Oh, S.L.; Faust, O.; Gertych, A.; Acharya, U.R. Automated Detection of Sleep Stages Using Deep Learning Techniques: A Systematic Review of the Last Decade (2010–2020). Appl. Sci. 2020, 10, 8963. [Google Scholar] [CrossRef]
  17. Boostani, R.; Karimzadeh, F.; Nami, M. A comparative review on sleep stage classification methods in patients and healthy individuals. Comput. Methods Programs Biomed. 2017, 140, 77–91. [Google Scholar] [CrossRef]
  18. Zhu, G.; Li, Y.; Wen, P. Analysis and classification of sleep stages based on difference visibility graphs from a single-channel EEG signal. IEEE J. Biomed. Health Inform. 2014, 18, 1813–1821. [Google Scholar] [CrossRef] [PubMed]
  19. Kim, J.; Lee, J.; Shin, M. Sleep stage classification based on noise-reduced fractal property of heart rate variability. Procedia Comput. Sci. 2017, 116, 435–440. [Google Scholar] [CrossRef]
  20. Rajput, J.S.; Sharma, M.; Acharya, U.R. Hypertension Diagnosis Index for Discrimination of High-Risk Hypertension ECG Signals Using Optimal Orthogonal Wavelet Filter Bank. Int. J. Environ. Res. Public Health 2019, 16, 4068. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  21. Cui, Z.; Zheng, X.; Shao, X.; Cui, L. Automatic sleep stage classification based on convolutional neural network and fine-grained segments. Complexity 2018, 2018, 9248410. [Google Scholar] [CrossRef] [Green Version]
  22. Sharma, M.; Goyal, D.; Achuth, P.; Acharya, U.R. An accurate sleep stages classification system using a new class of optimally time-frequency localized three-band wavelet filter bank. Comput. Biol. Med. 2018, 98, 58–75. [Google Scholar] [CrossRef] [PubMed]
  23. Timplalexis, C.; Diamantaras, K.; Chouvarda, I. Classification of sleep stages for healthy subjects and patients with minor sleep disorders. In Proceedings of the 2019 IEEE 19th International Conference on Bioinformatics and Bioengineering (BIBE), Athens, Greece, 28–30 October 2019; pp. 344–351. [Google Scholar]
  24. Tripathy, R.K.; Ghosh, S.K.; Gajbhiye, P.; Acharya, U.R. Development of automated sleep stage classification system using multivariate projection-based fixed boundary empirical wavelet transform and entropy features extracted from multichannel EEG signals. Entropy 2020, 22, 1141. [Google Scholar] [CrossRef] [PubMed]
  25. Widasari, E.R.; Tanno, K.; Tamura, H. Automatic sleep disorders classification using ensemble of bagged tree based on sleep quality features. Electronics 2020, 9, 512. [Google Scholar] [CrossRef] [Green Version]
  26. Biswal, S.; Sun, H.; Goparaju, B.; Westover, M.B.; Sun, J.; Bianchi, M.T. Expert-level sleep scoring with deep neural networks. J. Am. Med Inform. Assoc. 2018, 25, 1643–1650. [Google Scholar] [CrossRef] [Green Version]
  27. Yildirim, O.; Baloglu, U.B.; Acharya, U.R. A deep learning model for automated sleep stages classification using psg signals. Int. J. Environ. Res. Public Health 2019, 16, 599. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  28. Terzano, M.G.; Parrino, L.; Sherieri, A.; Chervin, R.; Chokroverty, S.; Guilleminault, C.; Hirshkowitz, M.; Mahowald, M.; Moldofsky, H.; Rosa, A.; et al. Atlas, rules, and recording techniques for the scoring of cyclic alternating pattern (CAP) in human sleep. Sleep Med. 2001, 2, 537–553. [Google Scholar] [CrossRef]
  29. Goldberger, A.L.; Amaral, L.A.N.; Glass, L.; Hausdorff, J.M.; Ivanov, P.C.; Mark, R.G.; Mietus, J.E.; Moody, G.B.; Peng, C.K.; Stanley, H.E. PhysioBank, PhysioToolkit, and PhysioNet. Circulation 2000, 101, e215–e220. [Google Scholar] [CrossRef] [Green Version]
  30. Dhok, S.; Pimpalkhute, V.; Chandurkar, A.; Bhurane, A.A.; Sharma, M.; Acharya, U.R. Automated phase classification in cyclic alternating patterns in sleep stages using Wigner-Ville Distribution based features. Comput. Biol. Med. 2020, 119, 103691. [Google Scholar] [CrossRef]
  31. Hartmann, S.; Baumert, M. Automatic a-phase detection of cyclic alternating patterns in sleep using dynamic temporal information. IEEE Trans. Neural Syst. Rehabil. Eng. 2019, 27, 1695–1703. [Google Scholar] [CrossRef] [PubMed]
  32. Hartmann, S.; Baumert, M. Improved A-phase Detection of Cyclic Alternating Pattern Using Deep Learning. In Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, 23–27 July 2019; pp. 1842–1845. [Google Scholar]
  33. Mendonça, F.; Fred, A.; Mostafa, S.S.; Morgado-Dias, F.; Ravelo-García, A.G. Automatic detection of cyclic alternating pattern. Neural Comput. Appl. 2018, 1–11. [Google Scholar] [CrossRef]
  34. Mariani, S.; Bianchi, A.M.; Manfredini, E.; Rosso, V.; Mendez, M.O.; Parrino, L.; Matteucci, M.; Grassi, A.; Cerutti, S.; Terzano, M.G. Automatic detection of A phases of the Cyclic Alternating Pattern during sleep. In Proceedings of the 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology, Buenos Aires, Argentina, 31 August–4 September 2010; pp. 5085–5088. [Google Scholar]
  35. Machado, F.; Sales, F.; Bento, C.; Dourado, A.; Teixeira, C. Automatic identification of cyclic alternating pattern (CAP) sequences based on the teager energy operator. In Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 25–29 August 2015; pp. 5420–5423. [Google Scholar]
  36. CAP Sleep Database. 2012. Available online: https://archive.physionet.org/physiobank/database/capslpdb (accessed on 15 January 2020).
  37. Pedersen, N.R. Comparison of Subjective Sleep Quality and Daytime Sleepiness with Whole-Night Polysomnography Findings in Sleep Apnea Patients. 2019. Available online: https://zir.nsk.hr/islandora/object/mefst:687 (accessed on 20 May 2020).
  38. Sharma, M.; Agarwal, S.; Acharya, U.R. Application of an optimal class of antisymmetric wavelet filter banks for obstructive sleep apnea diagnosis using ECG signals. Comput. Biol. Med. 2018, 100, 100–113. [Google Scholar] [CrossRef]
  39. Sharma, M.; Achuth, P.; Pachori, R.B.; Gadre, V.M. A parametrization technique to design joint time–frequency optimized discrete-time biorthogonal wavelet bases. Signal Process. 2017, 135, 107–120. [Google Scholar] [CrossRef]
  40. Sharma, M.; Dhere, A.; Pachori, R.B.; Gadre, V.M. Optimal duration-bandwidth localized antisymmetric biorthogonal wavelet filters. Signal Process. 2017, 134, 87–99. [Google Scholar] [CrossRef]
  41. Sharma, M.; Pachori, R.B.; Acharya, U.R. A new approach to characterize epileptic seizures using analytic time-frequency flexible wavelet transform and fractal dimension. Pattern Recognit. Lett. 2017, 94, 172–179. [Google Scholar] [CrossRef]
  42. Sharma, M.; Vanmali, A.V.; Gadre, V.M. Construction of wavelets: Principles and practices. In Wavelets and Fractals in Earth System Sciences; Taylor & Francis CRC Press: Abingdon, UK, 2013; pp. 29–92. [Google Scholar]
  43. Sharma, M.; Tan, R.S.; Acharya, U.R. Automated heartbeat classification and detection of arrhythmia using optimal orthogonal wavelet filters. Inform. Med. Unlocked 2019, 100221. [Google Scholar] [CrossRef]
  44. Sharma, M.; Raval, M.; Acharya, U.R. A new approach to identify obstructive sleep apnea using an optimal orthogonal wavelet filter bank with ECG signals. Inform. Med. Unlocked 2019, 100170. [Google Scholar] [CrossRef]
  45. Strang, G.; Strang, G.; Strang, G.; Strang, G. Introduction to Linear Algebra; Wellesley-Cambridge Press: Wellesley, MA, USA, 1993; Volume 3. [Google Scholar]
  46. Vapnik, V. The Nature of Statistical Learning Theory; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2013. [Google Scholar]
  47. Wu, X.; Kumar, V.; Quinlan, J.R.; Ghosh, J.; Yang, Q.; Motoda, H.; McLachlan, G.J.; Ng, A.; Liu, B.; Philip, S.Y.; et al. Top 10 algorithms in data mining. Knowl. Inf. Syst. 2008, 14, 1–37. [Google Scholar] [CrossRef] [Green Version]
  48. Tolles, J.; Meurer, W.J. Logistic regression: Relating patient characteristics to outcomes. JAMA 2016, 316, 533–534. [Google Scholar] [CrossRef]
  49. Lewis, D.D. Naive (Bayes) at forty: The independence assumption in information retrieval. In European Conference on Machine Learning; Springer: Berlin/Heidelberg, Germany, 1998; pp. 4–15. [Google Scholar]
  50. Cortes, C.; Vapnik, V. Support-vector networks. Mach. Learn. 1995, 20, 273–297. [Google Scholar] [CrossRef]
  51. Friedman, J.H.; Bentley, J.L.; Finkel, R.A. An algorithm for finding best matches in logarithmic expected time. ACM Trans. Math. Softw. (TOMS) 1977, 3, 209–226. [Google Scholar] [CrossRef]
  52. Zhou, Z.H. Ensemble Methods: Foundations and Algorithms; CRC Press: Boca Raton, FL, USA, 2012. [Google Scholar]
  53. Li, S.Z.; Jain, A. (Eds.) LDA (Linear Discriminant Analysis). In Encyclopedia of Biometrics; Springer: Boston, MA, USA, 2009; p. 899. [Google Scholar] [CrossRef]
  54. Dietterich, T.G. Ensemble methods in machine learning. In International Workshop on Multiple Classifier Systems; Springer: Berlin/Heidelberg, Germany, 2000; pp. 1–15. [Google Scholar]
  55. Dunn, O.; Clark, V. Applied Statistics: Analysis of Variance and Regression; John Wiley and Sons: New York, NY, USA, 1974. [Google Scholar]
  56. Gilchrist, M.; Samuels, P. One-Way Analysis of Variance (ANOVA). In Statistics Teaching Resources; Loughborough University Mathematics Learning Support Centre and Coventry University Mathematics Support Centre: Maynooth, Ireland, 2014. [Google Scholar]
  57. Dodge, Y. The Concise Encyclopedia of Statistics; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2008. [Google Scholar]
  58. McHugh, M.L. Lessons in biostatistics. Biochem. Med. 2009, 19, 120–126. [Google Scholar] [CrossRef]
  59. Bhati, D.; Sharma, M.; Pachori, R.B.; Nair, S.S.; Gadre, V.M. Design of Time–Frequency Optimal Three-Band Wavelet Filter Banks with Unit Sobolev Regularity Using Frequency Domain Sampling. Circuits Syst. Signal Process. 2016, 35, 4501–4531. [Google Scholar] [CrossRef]
  60. Phoong, S.M.; Kim, C.W.; Vaidyanathan, P.; Ansari, R. A new class of two-channel biorthogonal filter banks and wavelet bases. IEEE Trans. Signal Process. 1995, 43, 649–665. [Google Scholar] [CrossRef] [Green Version]
  61. Tay, D.B.H. Two-stage, least squares design of biorthogonal filter banks. IEEE Proc. Vis. Image Signal Process. 2002, 149, 341–346. [Google Scholar] [CrossRef]
  62. Sharma, M.; Acharya, U.R. Analysis of knee-joint vibroarthographic signals using bandwidth-duration localized three-channel filter bank. Comput. Electr. Eng. 2018, 72, 191–202. [Google Scholar] [CrossRef]
  63. Sharma, M.; Acharya, U.R. Automated detection of schizophrenia using optimal wavelet-based l1 norm features extracted from single-channel EEG. Cogn. Neurodyn. 2021, 1–14. [Google Scholar] [CrossRef]
  64. Sharma, M.; Patel, S.; Acharya, U.R. Automated detection of abnormal EEG signals using localized wavelet filter banks. Pattern Recogn. Lett. 2020, 133, 188–194. [Google Scholar] [CrossRef]
  65. Sharma, M.; San Tan, R.; Acharya, U.R. A novel automated diagnostic system for classification of myocardial infarction ECG signals using an optimal biorthogonal filter bank. Comput. Biol. Med. 2018, 102, 341–356. [Google Scholar] [CrossRef] [PubMed]
  66. Sharma, M.; Patel, S.; Choudhary, S.; Acharya, U.R. Automated Detection of Sleep Stages Using Energy-Localized Orthogonal Wavelet Filter Banks. Arabian J. Sci. Eng. 2019. [Google Scholar] [CrossRef]
  67. Sharma, M.; Singh, S.; Kumar, A.; Tan, R.S.; Acharya, U.R. Automated detection of shockable and non-shockable arrhythmia using novel wavelet-based ECG features. Comput. Biol. Med. 2019, 103446. [Google Scholar] [CrossRef]
  68. Sharma, M.; Acharya, U.R. A new method to identify coronary artery disease with ECG signals and time-Frequency concentrated antisymmetric biorthogonal wavelet filter bank. Pattern Recognit. Lett. 2019, 125, 235–240. [Google Scholar] [CrossRef]
Figure 1. Flowchart of the proposed methodology.
Figure 1. Flowchart of the proposed methodology.
Ijerph 18 03087 g001
Figure 2. Tuning of hyper parameters of the ensemble of bagged trees (EBT) classifier: Mis-classification error vs. number of trees.
Figure 2. Tuning of hyper parameters of the ensemble of bagged trees (EBT) classifier: Mis-classification error vs. number of trees.
Ijerph 18 03087 g002
Figure 3. Results of ANOVA post-hoc test obtained using the whole database.
Figure 3. Results of ANOVA post-hoc test obtained using the whole database.
Ijerph 18 03087 g003
Figure 4. ROC curves obtained for the classifier whole database using the EBT with 10-fold CV.
Figure 4. ROC curves obtained for the classifier whole database using the EBT with 10-fold CV.
Ijerph 18 03087 g004
Table 1. Summary of the state-of-the-art automated sleep stage classification studies conducted.
Table 1. Summary of the state-of-the-art automated sleep stage classification studies conducted.
WorkDescriptionPerformance (%)
Kim et al. [19]
(2017)
  • Database: CAP
  • Subjects: 13 Healthy
  • Signals: ECG/HRV
  • Classification: 2-class [(a)W vs. Sleep; and (b) (S1 + S2) vs. (S3 + S4)]
  • Features: Detrended fluctuation analysis(DFA) alpha 1
  • Classifier: k-fold cross validation (k = 13)
  • Accuracy:
    (a):
    73.6%
    (b):
    72.3
Sharma et al. [22]
(2018)
  • Database: Sleep EDF
  • Subjects: 98 (Healthy)
  • Signals: EEG
  • Channel: Fpz-Cz, pz-Oz,C4-A1
  • Classification: 6-class [W vs. S1 vs. S2 vs. S3 vs. S4 vs. REM]
  • Features: Log energy, Signal fractal dimension, Signal sample entropy
  • Classifier: SVM
  • Accuracy: 91.5
Timplalexis et al. [23]
(2019)
  • Database: Sleep EDF
  • Subjects: 197 Healthy
  • Signals: EEG
  • Channel: Fpz-Cz, Pz-Oz
  • Classification: 5-class [W vs. N1 vs. N2 vs. N3 vs. REM]
  • Features: Mixture of time and frequency domain features
  • Classifier: EBT
  • Accuracy: 88.88
Tripathi et al. [24]
(2020)
  • Database: CAP
  • Subjects: 25 (6-H,7-Ins,1-Brux,1-SDB,10-RBD)
  • Signals: EEGs
  • Channel: F4-C4, C4-P4, P4-O2, C4-A1
  • Classification: 6-class (W vs. S1 vs. S2 vs. S3 vs. S4 vs. REM)
  • Features: Dispersion entropy and Bubble entropy
  • Classifier: Hybrid classifier
  • Accuracy: 71.68
Widasari et al. [25]
(2020)
  • Database: CAP
  • Subjects: 51 [16-H, 9-ins, 4-SDB, 22-RBD]
  • Signals: ECG
  • Classification: 4-class [W vs. (S1 + S2) vs. (S3 + S4) vs. REM]
  • Features: Sleep quality
  • Classifier: EBT
  • Accuracy: 86.27
Table 2. Description of the cyclic alternating pattern (CAP) sleep database.
Table 2. Description of the cyclic alternating pattern (CAP) sleep database.
Subject TypeSubjects AvailableRecordings Available @512 HzNumber-of EpochsMale & FemaleAge (in yrs) (Mean ± Std)
Healthy16660632 M & 4 F32 ± 4.89
Insomnia9785513 M & 5 F61.75 ± 10.20
Bruxism214271 M34
Narcolepsy5556142 M & 3 F31.6 ± 10.32
NFLE402726,88313 M & 14 F30.03 ± 10.53
PLM10975746 M & 3 F54.44 ± 6.37
RBD222222,67619 M & 3 F70.72 ± 6.23
SBD4328793 M69.33 ± 6.12
Total108808066748M & 32F48.25 ± 19.69
Table 3. Sleep stage-wise and subject-wise details of epoch distribution in the original unbalanced database.
Table 3. Sleep stage-wise and subject-wise details of epoch distribution in the original unbalanced database.
Sleep StageHealthySeven Different DisordersTotal
InsomniaBruxismNarcolepsyNFLEPLMRBDSBDEpochs(in %)
Wake445380144130331551332526649515,84119.64%
S1280223343011098266104826935194.36%
S221722456144170810,63027487446132428,62835.50%
S35736703947629879552880224880410.92%
S41184415995684108956250635210,18812.63%
REM140998667125849051317353021513,68716.97%
Total60638551427561426,883757422,676287980,667
Table 4. Sleep stage-wise and subject-wise details of epoch distribution in the created balanced data.
Table 4. Sleep stage-wise and subject-wise details of epoch distribution in the created balanced data.
Sleep StageHealthySeven Different DisordersTotal
InsomniaBruxismNarcolepsyNFLEPLMRBDSBDEpochs(in %)
Wake100014007193544801262377948013,40716.67%
S1100014007193544801262377948013,40716.67%
S2100014007193544801262377948013,40716.67%
S3100014007193544801262377948013,40716.67%
S4100014007193544801262377948013,40716.67%
REM100014007193544801262377948013,40716.67%
Total60008400426561026,880757222,674288080,442
Table 5. Statistical analysis using ANOVA for F4-C4 channel.
Table 5. Statistical analysis using ANOVA for F4-C4 channel.
FeatureSub-BandRankp-ValueW (Mean ± Std)S1 (Mean ± std)S2 (Mean ± Std)S3 (Mean ± Std)S4 (Mean ± Std)REM (Mean ± Std)
l 1 n o r m Sb-12020,656.14 ± 54,855.489281.65 ± 9331.5513,761.47 ± 9066.7918,321.05 ± 11,585.5928,405.90 ± 16,908.079989.34 ± 6288.26
Sb-210749.99 ± 1307.83296.04 ± 692.05197.74 ± 328.99162.27 ± 216.34154.04 ± 354.95167.73 ± 114.87
Sb-31602740.34 ± 4746.11877.57 ± 1053.98644.24 ± 718.22528.03 ± 452.74486.95 ± 602.20560.29 ± 450.23
Sb-41305857.94 ± 9138.122190.28 ± 1620.461749.38 ± 1380.681449.88 ±928.231277.17 ± 800.731796.56 ±1233.18
Sb-51407843.60 ± 10,155.933975.20 ± 2416.013764.24 ± 2709.473173.00 ± 1925.922813.16 ± 1394.383579.89 ± 2530.63
Sb-6406880.60 ± 9721.224182.79 ± 2983.744832.37 ± 3530.335016.90 ± 3619.345092.04 ± 3125.723765.75 ± 2241.55
l 2 n o r m Sb-1301368.82 ± 3627.26624.28 ± 896.76884.61 ± 663.451110.23 ± 718.851658.00 ± 980.41610.15 ± 488.75
Sb-271.13 × 10−12116.32 ± 62.5715.08 ± 130.575.61 ± 61.934.02 ± 47.444.37 ± 57.162.85 ± 4.35
Sb-315074.06 ± 136.7828.04 ± 80.1717.09 ± 48.8212.85 ± 35.0112.22 ± 48.5014.02 ± 28.81
Sb-4110204.88 ± 351.4776.59 ± 97.8356.15 ± 65.5245.08 ± 64.9639.13 ± 51.5355.86 ± 48.13
Sb-580363.07 ± 571.78181.13 ± 167.77165.59 ± 132.54138.25 ± 116.83122.02 ± 81.54152.98 ± 114.36
Sb-6100440.48 ± 803.03267.51 ± 282.30298.63 ± 234.43302.81 ± 239.25301.61 ± 193.20225.46 ± 137.81
l n o r m Sb-190279.62 ± 609.45139.41 ± 295.40198.69 ± 189.74218.39 ± 172.16287.81 ± 192.23117.48 ± 135.51
Sb-263.39 × 10−343.01 ± 31.626.05 ± 69.161.64 ± 34.011.16 ± 28.381.33 ± 30.190.30 ± 0.92
Sb-317011.51 ± 28.845.93 ± 37.012.69 ± 20.831.78 ± 13.421.85 ± 20.981.89 ± 5.85
Sb-45035.35 ± 76.9213.10 ± 39.597.90 ± 23.045.76 ± 20.354.83 ± 21.267.46 ± 13.76
Sb-512071.53 ± 157.2733.31 ± 74.2627.44 ± 42.1322.45 ± 39.6619.76 ± 29.9424.59 ± 33.29
Sb-618097.71 ± 247.8858.77 ± 126.5362.01 ± 68.4458.84 ± 75.5054.32 ± 47.0843.76 ± 37.97
Table 6. Statistical analysis using ANOVA for C4-A1 channel.
Table 6. Statistical analysis using ANOVA for C4-A1 channel.
FeatureSub-BandRankp-ValueW (Mean ± Std)S1 (Mean ± std)S2 (Mean ± Std)S3 (Mean ± Std)S4 (Mean ± Std)REM (Mean ± Std)
l 1 n o r m Sb-11036,981.35 ± 58,883.5123,963.81 ± 20,242.1632,003.79 ± 18,531.4841,632.86 ± 16,256.3363,699.18 ± 27,718.6725,232.93 ± 13,663.27
Sb-21401563.97 ± 1513.03852.80 ± 980.33534.78 ± 650.33437.19 ± 466.81345.76 ± 449.87358.28 ± 371.59
Sb-3505904.82 ± 5616.392981.21 ± 2803.201990.35 ± 2449.271665.68 ± 1614.531349.09 ± 1264.191361.41 ± 1437.16
Sb-413012,578.84 ± 11,387.836533.51 ± 4718.184690.19 ± 3691.484069.94 ± 2870.763403.56 ± 2202.584172.80 ± 2942.39
Sb-53015,775.64 ± 12,303.6710,209.32 ± 4514.268824.32 ± 4143.847721.53 ± 2683.566881.04 ± 2166.727998.54 ± 3353.39
Sb-66014,728.59 ± 11,745.0611,078.86 ± 5189.4111,424.13 ± 4806.4811,800.24 ± 3853.0411,895.95 ± 4454.359165.82 ± 3029.86
l 2 n o r m Sb-1402425.35 ± 3989.361578.19 ± 1751.272019.55 ± 1399.182501.36 ± 1138.423702.32 ± 1584.331554.12 ± 1183.25
Sb-27030.16 ± 64.4424.86 ± 126.9911.47 ± 62.728.30 ± 46.927.33 ± 57.066.42 ± 8.19
Sb-3160150.45 ± 159.7781.05 ± 106.2349.31 ± 73.7937.86 ± 50.8430.66 ± 53.6634.96 ± 47.16
Sb-420431.08 ± 428.95228.16 ± 210.81153.65 ± 150.42125.85 ± 119.98103.22 ± 83.31138.40 ± 126.72
Sb-5170719.14 ± 662.50463.36 ± 263.17391.92 ± 214.74337.04 ± 164.01299.96 ± 111.59349.51 ± 177.53
Sb-6150904.04 ± 887.55681.53 ± 364.61695.50 ± 313.77702.02 ± 251.02697.05 ± 262.48544.97 ± 195.84
l n o r m Sb-1180491.43 ± 757.98336.28 ± 510.28431.10 ± 370.31477.72 ± 303.09630.99 ± 313.80302.60 ± 355.54
Sb-2103.27 × 10 39 4.30 ± 31.837.18 ± 71.132.19 ± 33.771.39 ± 25.341.59 ± 29.970.78 ± 1.43
Sb-38020.73 ± 32.2712.47 ± 38.156.43 ± 20.304.29 ± 15.563.73 ± 23.685.22 ± 9.53
Sb-411069.69 ± 91.7237.35 ± 59.7022.08 ± 39.1315.64 ± 30.6811.96 ± 24.2821.98 ± 34.33
Sb-590136.69 ± 180.0784.12 ± 97.3066.58 ± 67.2954.39 ± 55.8547.25 ± 37.8261.45 ± 68.00
Sb-6120183.10 ± 263.80137.42 ± 131.19138.07 ± 96.04129.94 ± 79.24120.41 ± 61.44105.42 ± 74.07
Table 7. Summary of classification average accuracy obtained during each trial using the EBT classifier with 10-fold cross validation (CV).
Table 7. Summary of classification average accuracy obtained during each trial using the EBT classifier with 10-fold cross validation (CV).
Type of SubjectChannelTrial 1Trial 2Trial 3Trial 4Trial 5Mean ± Std
HealthyF4-C474.9671.8672.4671.5673.6672.9 ± 1.26
C4-A173.172.274.873.472.573.2 ± 0.91
F4-C4 + C4-A176.2677.6679.0679.6678.8678.3 ± 1.21
InsomniaF4-C486.7486.1486.1485.6485.8486.1 ± 0.37
C4-A186.7685.4685.4686.5685.2685.9 ± 0.63
F4-C4 + C4-A185.1486.2484.4484.5486.6485.4 ± 0.89
BruxismF4-C462.3464.8463.7463.9462.6463.5 ± 0.91
C4-A166.3464.0463.9466.2465.9465.3 ± 1.08
F4-C4 + C4-A167.7865.7865.4868.5865.8866.7 ± 1.24
NarcolepsyF4-C477.0678.4676.1677.1676.1677 ± 0.85
C4-A176.5278.0274.3276.2275.4276.1 ± 1.23
F4-C4 + C4-A180.277.977.979.880.779.3 ± 1.18
NFLEF4-C471.872.872.671.170.771.8 ± 0.82
C4-A172.1871.9871.7872.5873.9872.5 ± 0.79
F4-C4 + C4-A176.578.577.778.87677.5 ± 1.09
PLMF4-C473.8871.5874.2872.0874.6873.3 ± 1.24
C4-A174.774.674.976.573.874.9 ± 0.88
F4-C4 + C4-A177.2478.3478.1479.3476.9478 ± 0.85
RBDF4-C466.0667.6665.7664.5664.9665.8 ± 1.07
C4-A168.5666.8666.0667.7666.7667.2 ± 0.87
F4-C4 + C4-A171.3872.8870.3872.7872.0871.9 ± 0.93
SDBF4-C470.5268.1268.1269.7268.0268.9 ± 1.03
C4-A170.9472.3474.2471.5473.4472.5 ± 1.21
F4-C4 + C4-A174.0674.9673.9675.3673.1674.3 ± 0.78
Healthy + Unhealthy subjectsF4-C470.168.668.47169.969.6 ± 0.97
C4-A168.5871.7870.1870.2870.6870.3 ± 1.03
F4-C4 + C4-A173.3875.7876.0876.2875.9875.5 ± 1.07
Table 8. Performance of sleep stage classification obtained using the unbalanced dataset and EBT classifier with 10-fold CV.
Table 8. Performance of sleep stage classification obtained using the unbalanced dataset and EBT classifier with 10-fold CV.
Type of SubjectTotal Subjects AvailableRecordings Available @512 HzChannelAccuracy (%)No. of EpochsPrediction Speed (obs/s)Training Time (s)
Healthy166F4-C472.9606315,00025.23
C4-A173.217,00020.25
F4-C4 + C4-A178.317,00022.26
Insomnia97F4-C486.1855123,00022.73
C4-A185.923,00022.46
F4-C4 + C4-A185.413,00028.69
Bruxism21F4-C463.542715004.84
C4-A165.314004.88
F4-C4 + C4-A166.716005.02
Narcolepsy55F4-C477.0561411,00014.00
C4-A176.111,00014.45
F4-C4 + C4-A179.31300015.77
NFLE4027F4-C471.826,88321,00062.38
C4-A172.519,00063.17
F4-C4 + C4-A177.520,00078.32
PLM109F4-C473.3757414,00018.74
C4-A174.9970027.70
F4-C4 + C4-A178.016,00020.94
RBD2222F4-C465.822,67615,00063.79
C4-A167.218,00054.45
F4-C4 + C4-A171.912,000119.48
SDB43F4-C468.92879590010.72
C4-A172.5530012.45
F4-C4 + C4-A174.3610013.53
Healthy + Unhealthy subjects10880F4-C469.680,66718,000239.74
C4-A170.318,000239.80
F4-C4 + C4-A175.519,000286.92
Table 9. Confusion matrix corresponding to sleep stage classification of healthy subjects with unbalanced data using the EBT classifier with 10-fold CV.
Table 9. Confusion matrix corresponding to sleep stage classification of healthy subjects with unbalanced data using the EBT classifier with 10-fold CV.
CT-1: Healthy (Unbalanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake68.8%15.7%5.4%0.9%0.4%8.8%0.71
S121.4%42.5%23.2%0.0%0.7%12.1%0.44
S21.3%2.3%80.9%6.0%1.2%8.4%0.80
S30.9%0.2%28.6%56.9%12.6%0.9%0.59
S40.3%0.1%1.9%5.7%91.7%0.2%0.91
REM1.1%1.8%11.8%0.9%0.5%84.0%0.83
Cohen’s kappa (±kappa error) = 0.7212 ± 0.0069
Overall Accuracy = 78.3%
Table 10. Confusion matrix corresponding to sleep stage classification of healthy subjects with balanced data using the EBT classifier with 10-fold CV.
Table 10. Confusion matrix corresponding to sleep stage classification of healthy subjects with balanced data using the EBT classifier with 10-fold CV.
CT-1: Healthy (Balanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake97.7%1.4%0.5%0.1%0.1%0.2%0.96
S10.5%99.3%0.0%0.0%0.0%0.2%0.95
S22.3%5.0%64.1%12.5%2.3%13.8%0.73
S30.3%0.1%3.3%94.0%2.3%0.0%0.86
S40.2%0.1%1.4%9.8%88.5%0.0%0.92
REM2.1%5.0%8.0%1.5%0.9%82.5%0.83
Cohen’s kappa (±kappa error) = 0.8554 ± 0.0050
Overall Accuracy = 87.9%
Table 11. Confusion matrix corresponding to sleep stage classification of insomnia patients with balanced data using the EBT classifier with 10-fold CV.
Table 11. Confusion matrix corresponding to sleep stage classification of insomnia patients with balanced data using the EBT classifier with 10-fold CV.
CT-2: Insomnia (Unbalanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake96.2%0.7%2.6%0.1%0.1%0.4%0.93
S144.8%7.2%37.7%0.4%0.0%9.9%0.11
S28.9%0.6%81.6%3.6%0.4%4.9%0.82
S31.3%0.0%17.9%56.9%9.7%0.1%0.73
S40.0%0.0%2.2%14.5%83.4%0.0%0.82
REM4.7%0.1%13.8%0.1%0.1%81.2%0.82
Cohen’s kappa (±kappa error) = 0.7867 ± 0.0056
Overall Accuracy = 85.4%
Table 12. Confusion Matrix corresponding to sleep stage classification of insomnia patients with unbalanced data using the EBT classifier with 10-fold CV.
Table 12. Confusion Matrix corresponding to sleep stage classification of insomnia patients with unbalanced data using the EBT classifier with 10-fold CV.
CT-2: Insomnia (Balanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake92.1%4.0%2.7%0.2%0.3%0.6%0.93
S10.0%100%0.0%0.0%0.0%0.0%0.95
S25.6%4.5%74.1%7.1%0.7%7.9%0.82
S30.3%0.0%1.1%97.1%1.4%0.0%0.95
S40.0%0.0%0.0%0.0%100.0%0.0%0.99
REM1.1%1.1%3.9%0.1%0.1%93.6%0.93
Cohen’s kappa (±kappa error) = 0.9145 ± 0.0034
Overall Accuracy = 92.8%
Table 13. Confusion matrix corresponding to sleep stage classification of bruxism patients with unbalanced data using the EBT classifier with 10-fold CV.
Table 13. Confusion matrix corresponding to sleep stage classification of bruxism patients with unbalanced data using the EBT classifier with 10-fold CV.
CT-3: Bruxism (Unbalanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake54.5%15.9%18.2%6.8%2.3%2.3%0.48
S141.2%17.6%29.4%0.0%8.8%2.9%0.20
S26.9%5.6%75.0%6.3%2.8%3.5%0.72
S37.7%0.0%46.2%10.3%35.9%0.0%0.13
S44.0%4.0%6.1%5.1%80.8%0.0%0.80
REM0.0%0.0%11.9%0.0%0.0%88.1%0.89
Cohen’s kappa (±kappa error) = 0.5578 ± 0.0297
Overall Accuracy = 66.7%
Table 14. Confusion matrix corresponding to sleep stage classification of bruxism patients with balanced data using the EBT classifier with 10-fold CV.
Table 14. Confusion matrix corresponding to sleep stage classification of bruxism patients with balanced data using the EBT classifier with 10-fold CV.
CT-3: Bruxism (Balanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake90.1%8.5%0.0%0.0%0.0%1.4%0.89
S10.0%97.2%0.0%0.0%2.8%0.0%0.82
S28.5%19.7%45.1%14.1%5.6%7.0%0.59
S32.8%0.0%2.8%94.4%0.0%0.0%0.85
S41.4%9.9%4.2%12.7%71.8%0.0%0.79
REM0.0%1.4%1.4%0.0%1.4%95.8%0.94
Cohen’s kappa (±kappa error) = 0.7887 ± 0.0221
Overall Accuracy = 82.4%
Table 15. Confusion matrix corresponding to sleep stage classification of narcolepsy patients with unbalanced data using the EBT classifier with 10-fold CV.
Table 15. Confusion matrix corresponding to sleep stage classification of narcolepsy patients with unbalanced data using the EBT classifier with 10-fold CV.
CT-4: Narcolepsy (Unbalanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake91.0%1.7%4.3%0.1%0.2%2.8%0.87
S129.6%21.3%17.9%0.0%0.3%30.9%0.30
S23.9%1.2%84.0%3.4%0.6%7.0%0.80
S31.5%0.0%35.1%50.6%11.6%1.3%0.58
S40.7%0.0%5.1%9.2%85.0%0.0%0.86
REM5.5%2.0%10.1%0.1%0.0%82.4%0.81
Cohen’s kappa (±kappa error) = 0.7301 ± 0.0070
Overall Accuracy = 79.3%
Table 16. Confusion matrix corresponding to sleep stage classification of narcolepsy patients with balanced data using the EBT classifier with 10-fold CV.
Table 16. Confusion matrix corresponding to sleep stage classification of narcolepsy patients with balanced data using the EBT classifier with 10-fold CV.
CT-4: Narcolepsy (Balanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake87.0%5.1%3.0%0.1%0.2%4.6%0.89
S10.3%99.7%0.0%0.0%0.0%0.0%0.91
S23.5%5.7%71.7%10.9%1.0%7.3%0.78
S30.2%0.0%2.0%95.8%1.9%0.0%0.91
S40.1%0.1%0.4%3.1%96.3%0.0%0.97
REM5.0%8.6%7.0%0.3%0.1%79.0%0.83
Cohen’s kappa (±kappa error) = 0.8588 ± 0.0052
Overall Accuracy = 88.2%
Table 17. Confusion matrix corresponding to sleep stage classification of nocturnal frontal lobe epilepsy patients with unbalanced data using the EBT classifier with 10-fold CV.
Table 17. Confusion matrix corresponding to sleep stage classification of nocturnal frontal lobe epilepsy patients with unbalanced data using the EBT classifier with 10-fold CV.
CT-5: NFLE (Unbalanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake86.0%2.5%8.0%0.1%0.3%3.1%0.81
S142.3%23.5%18.6%0.3%0.0%15.3%0.34
S22.2%0.4%87.7%3.9%1.2%4.7%0.81
S30.2%0.0%50.3%34.1%15.1%0.2%0.43
S40.1%0.0%5.3%8.1%86.5%0.0%0.86
REM3.1%0.7%16.3%0.5%0.3%79.1%0.81
Cohen’s kappa (±kappa error) = 0.6914 ± 0.0035
Overall Accuracy = 77.5%
Table 18. Confusion matrix corresponding to sleep stage classification of Nocturnal frontal lobe epilepsy patients with balanced data using the EBT classifier with 10-fold CV.
Table 18. Confusion matrix corresponding to sleep stage classification of Nocturnal frontal lobe epilepsy patients with balanced data using the EBT classifier with 10-fold CV.
CT-5: NFLE (Balanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake93.5%3.3%1.4%0.1%0.1%1.6%0.93
S10.0%100.0%0.0%0.0%0.0%0.0%0.96
S23.2%2.2%68.8%15.1%1.9%8.9%0.73
S30.2%0.1%9.1%84.2%6.1%0.3%0.80
S40.1%0.0%1.6%9.%89.1%0.0%0.90
REM3.2%2.2%8.6%1.3%0.4%84.3%0.86
Cohen’s kappa (±kappa error) = 0.8395 ± 0.0025
Overall Accuracy = 86.6%
Table 19. Confusion matrix corresponding to sleep stage classification of periodic leg movement patients with unbalanced data using the EBT classifier with 10-fold CV.
Table 19. Confusion matrix corresponding to sleep stage classification of periodic leg movement patients with unbalanced data using the EBT classifier with 10-fold CV.
CT-6: PLM (Unbalanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake83.4%4.6%9.7%0.3%0.3%1.7%0.84
S132.3%22.6%25.2%0.4%0.0%19.5%0.28
S23.5%1.3%83.8%7.2%0.4%3.7%0.82
S30.5%0.0%26.0%59.8%13.2%0.5%0.61
S40.2%0.0%1.4%15.1%83.2%0.2%0.84
REM1.4%0.9%8.8%0.7%0.2%88.1%0.87
Cohen’s kappa (±kappa error) = 0.7296 ± 0.0061
Overall Accuracy = 78.0%
Table 20. Confusion matrix corresponding to sleep stage classification of periodic leg movement patients with balanced data using the EBT classifier with 10-fold CV.
Table 20. Confusion matrix corresponding to sleep stage classification of periodic leg movement patients with balanced data using the EBT classifier with 10-fold CV.
CT-6: PLM (Balanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake89.5%4.3%4.3%0.3%0.1%1.5%0.90
S10.0%100.0%0.0%0.0%0.0%0.0%0.96
S25.9%2.9%69.5%15.1%1.0%5.6%0.72
S30.6%0.1%11.4%80.0%7.6%0.3%0.78
S40.0%0.0%1.0%8.7%90.3%0.0%0.91
REM3.2%1.7%7.6%1.3%0.3%85.8%0.89
Cohen’s kappa (±kappa error) = 0.8301 ± 0.0048
Overall Accuracy = 85.8%
Table 21. Confusion matrix corresponding to sleep stage classification of REM behavioural disorder patients with unbalanced data using the EBT classifier with 10-fold CV.
Table 21. Confusion matrix corresponding to sleep stage classification of REM behavioural disorder patients with unbalanced data using the EBT classifier with 10-fold CV.
CT-7: RBD (Unbalanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake87.9%1.1%8.3%0.2%0.2%2.2%0.84
S129.0%36.2%24.2%0.4%0.0%10.2%0.47
S26.0%1.0%79.6%5.5%1.4%6.5%0.73
S31.4%0.0%31.6%51.6%12.3%3.2%0.56
S40.3%0.0%4.1%14.3%80.5%0.9%0.80
REM9.1%1.0%31.6%3.3%0.8%54.1%0.61
Cohen’s kappa (± kappa error) = 0.6372 ± 0.0039
Overall Accuracy = 71.9%
Table 22. Confusion Matrix corresponding to sleep stage classification of REM behavioural disorder patients with balanced data using the EBT classifier with 10-fold CV.
Table 22. Confusion Matrix corresponding to sleep stage classification of REM behavioural disorder patients with balanced data using the EBT classifier with 10-fold CV.
CT-7: RBD (Balanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake84.1%4.9%5.5%0.7%0.2%4.7%0.84
S10.1%99.8%0.1%0.0%0.0%0.1%0.94
S26.9%4.7%59.7%13.0%2.3%13.5%0.64
S30.8%0.1%8.8%79.9%7.9%2.5%0.78
S40.1%0.0%0.4%6.3%92.7%0.6%0.91
REM7.3%3.3%13.2%5.5%1.1%69.6%0.73
Cohen’s kappa (±kappa error) = 0.7715 ± 0.0031
Overall Accuracy = 81.0%
Table 23. Confusion matrix corresponding to sleep stage classification of sleep-breathing disorder patients with unbalanced data using the EBT classifier with 10-fold CV.
Table 23. Confusion matrix corresponding to sleep stage classification of sleep-breathing disorder patients with unbalanced data using the EBT classifier with 10-fold CV.
CT-8: SBD (Unbalanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake81.4%9.1%8.5%0.2%0.0%0.8%0.76
S134.9%29.4%34.2%0.0%0.4%1.1%0.35
S23.8%3.4%89.0%2.0%1.1%0.8%0.83
S30.0%0.0%60.3%22.8%17.0%0.0%0.31
S40.3%0.0%7.1%8.0%84.7%0.0%0.84
REM6.5%5.1%25.6%0.9%3.7%58.1%0.70
Cohen’s kappa (± kappa error) = 0.6276 ± 0.0117
Overall Accuracy = 74.3%
Table 24. Confusion matrix corresponding to sleep stage classification of sleep-breathing disorder patients with balanced data using the EBT classifier with 10-fold CV.
Table 24. Confusion matrix corresponding to sleep stage classification of sleep-breathing disorder patients with balanced data using the EBT classifier with 10-fold CV.
CT-8: SBD (Balanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake78.1%14.2%4.6%1.0%0.2%1.9%0.83
S15.2%91.0%2.3%0.0%0.8%0.0%0.85
S24.8%8.8%67.3%12.7%2.3%4.2%0.75
S30.0%0.0%2.9%95.4%1.7%0.0%0.88
S40.4%0.2%0.8%5.8%92.5%0.2%0.94
REM0.4%0.4%1.3%0.4%0.4%97.1%0.95
Cohen’s kappa (±kappa error) = 0.8429 ± 0.0075
Overall Accuracy = 86.9%
Table 25. Confusion matrix corresponding to sleep stage classification of all disorders combined with unbalanced data using the EBT classifier with 10-fold CV.
Table 25. Confusion matrix corresponding to sleep stage classification of all disorders combined with unbalanced data using the EBT classifier with 10-fold CV.
CT-9: All Disordered (Unbalanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake88.9%1.3%7.5%0.1%0.1%2.1%0.84
S139.7%21.4%24.2%0.1%0.1%14.5%0.31
S24.6%0.6%85.4%3.6%1.0%4.9%0.78
S31.1%0.0%42.5%42.1%12.9%1.3%0.50
S40.1%0.0%6.0%10.2%83.4%0.2%0.84
REM6.0%0.9%22.3%1.4%0.3%69.1%0.74
Cohen’s kappa (± kappa error) = 0.6780 ± 0.0021
Overall Accuracy = 75.6%
Table 26. Confusion matrix corresponding to sleep stage classification of all disordered patients with balanced data using the EBT classifier with 10-fold CV.
Table 26. Confusion matrix corresponding to sleep stage classification of all disordered patients with balanced data using the EBT classifier with 10-fold CV.
CT-9: All disordered (Balanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake87.7%4.9%3.8%0.6%0.2%2.9%0.87
S10.1%99.8%0.1%0.0%0.0%0.1%0.94
S26.7%3.3%65.6%12.8%1.4%10.2%0.71
S30.6%0.0%8.0%85.4%5.1%0.9%0.82
S40.1%0.1%0.9%6.9%91.9%0.1%0.92
REM5.7%3.5%10.0%2.9%0.4%77.5%0.81
Cohen’s kappa (± kappa error) = 0.8482 ± 0.1667
Overall Accuracy = 84.8%
Table 27. Confusion matrix corresponding to sleep stage classification of all subjects combined (healthy + seven disordered) with unbalanced data using the EBT classifier with 10-fold CV.
Table 27. Confusion matrix corresponding to sleep stage classification of all subjects combined (healthy + seven disordered) with unbalanced data using the EBT classifier with 10-fold CV.
CT-10: All Subjects Combined (Unbalanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake87.8%1.5%7.9%0.2%0.1%2.4%0.83
S139.2%21.4%24.2%0.1%0.1%15.0%0.31
S24.7%0.6%84.4%3.7%1.1%5.5%0.78
S31.2%0.0%43.4%41.1%13.1%1.2%0.49
S40.2%0.0%5.9%9.6%84.0%0.2%0.84
REM5.9%0.8%22.5%1.3%0.4%69.1%0.73
Cohen’s kappa (± kappa error) = 0.6697 ± 0.0020
Overall Accuracy = 75.5%
Table 28. Confusion matrix corresponding to sleep stage classification of all patients combined with balanced data using the EBT classifier with 10-fold CV.
Table 28. Confusion matrix corresponding to sleep stage classification of all patients combined with balanced data using the EBT classifier with 10-fold CV.
CT-10: All Subjects Combined (Balanced Data)
True ClassPredicted ClassF1 Score
WakeS1S2S3S4R
Wake88.1%5.0%3.5%0.4%0.1%2.9%0.88
S10.0%100%0.0%0.0%0.0%0.0%0.94
S26.2%3.0%66.5%12.5%1.4%10.4%0.70
S30.6%0.1%6.9%87.5%4.4%0.6%0.84
S40.1%0.0%0.8%6.6%92.3%0.1%0.93
REM5.5%3.2%10%2.8%0.5%78%0.81
Cohen’s kappa (± kappa error) = 0.8214 ± 0.0015
Overall Accuracy = 85.1%
Table 29. Sleep stage-wise epochs and accuracy (before balancing) using the EBT classifier with 10-fold CV.
Table 29. Sleep stage-wise epochs and accuracy (before balancing) using the EBT classifier with 10-fold CV.
Sleep StageAll Subjects CombinedHealthy GroupSleep-Disordered Group
EpochsEpochs (in %)EpochEpochs (in %)EpochEpochs (in %)
Wake15,84119.644457.3415,39620.65
S135194.362804.6232394.34
S228,62835.50217235.8226,45635.48
S3880410.925739.45823111.04
S410,18812.63118419.53900412.07
REM13,68716.97140923.2412,27816.45
Total80,667100.006063100.0074,604100.00
Accuracy75%78.8%75.6%
Table 30. Sleep stage-wise epochs and accuracy (after balancing) using the EBT classifier with 10-fold CV.
Table 30. Sleep stage-wise epochs and accuracy (after balancing) using the EBT classifier with 10-fold CV.
Sleep StageAll Subjects CombinedHealthy GroupDisordered-Group
EpochsEpochs (in %)EpochEpochs (in %)EpochEpochs (in %)
Wake14,00016.66100016.6612,50016.66
S114,00016.66100016.6612,50016.66
S214,00016.66100016.6612,50016.66
S314,00016.66100016.6612,50016.66
S414,00016.66100016.6612,50016.66
REM14,00016.66100016.6612,50016.66
Total84,000100600010075,000100
Accuracy85.1%87.9%84.8%
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Sharma, M.; Tiwari, J.; Acharya, U.R. Automatic Sleep-Stage Scoring in Healthy and Sleep Disorder Patients Using Optimal Wavelet Filter Bank Technique with EEG Signals. Int. J. Environ. Res. Public Health 2021, 18, 3087. https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph18063087

AMA Style

Sharma M, Tiwari J, Acharya UR. Automatic Sleep-Stage Scoring in Healthy and Sleep Disorder Patients Using Optimal Wavelet Filter Bank Technique with EEG Signals. International Journal of Environmental Research and Public Health. 2021; 18(6):3087. https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph18063087

Chicago/Turabian Style

Sharma, Manish, Jainendra Tiwari, and U. Rajendra Acharya. 2021. "Automatic Sleep-Stage Scoring in Healthy and Sleep Disorder Patients Using Optimal Wavelet Filter Bank Technique with EEG Signals" International Journal of Environmental Research and Public Health 18, no. 6: 3087. https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph18063087

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop