Next Article in Journal
Health Benefits of Physical Activity Related to an Urban Riverside Regeneration
Next Article in Special Issue
Health Information Engagement Factors in Malaysia: A Content Analysis of Facebook Use by the Ministry of Health in 2016 and 2017
Previous Article in Journal
Current and Emerging Disaster Risks Perceptions in Oceania: Key Stakeholders Recommendations for Disaster Management and Resilience Building
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Smartphone Sensors for Monitoring Cancer-Related Quality of Life: App Design, EORTC QLQ-C30 Mapping and Feasibility Study in Healthy Subjects

by
Sabina Asensio-Cuesta
1,*,
Ángel Sánchez-García
1,
J. Alberto Conejero
2,
Carlos Saez
1,
Alejandro Rivero-Rodriguez
3 and
Juan M. García-Gómez
1
1
Instituto de Tecnologías de la Información y Comunicaciones (ITACA), Universitat Politècnica de València, Camino de Vera s/n, 46022 Valencia, Spain
2
Instituto Universitario de Matemática Pura y Aplicada, Universitat Politècnica de València, Camino de Vera s/n, 46022 Valencia, Spain
3
Salumedia Tecnologías, 41011 Sevilla, Spain
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2019, 16(3), 461; https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph16030461
Submission received: 18 December 2018 / Revised: 29 January 2019 / Accepted: 31 January 2019 / Published: 5 February 2019
(This article belongs to the Special Issue E-Health Services)

Abstract

:
Quality of life (QoL) indicators are now being adopted as clinical outcomes in clinical trials on cancer treatments. Technology-free daily monitoring of patients is complicated, time-consuming and expensive due to the need for vast amounts of resources and personnel. The alternative method of using the patients’ own phones could reduce the burden of continuous monitoring of cancer patients in clinical trials. This paper proposes monitoring the patients’ QoL by gathering data from their own phones. We considered that the continuous multiparametric acquisition of movement, location, phone calls, conversations and data use could be employed to simultaneously monitor their physical, psychological, social and environmental aspects. An open access phone app was developed (Human Dynamics Reporting Service (HDRS)) to implement this approach. We here propose a novel mapping between the standardized QoL items for these patients, the European Organization for the Research and Treatment of Cancer Quality of Life Questionnaire (EORTC QLQ-C30) and define HDRS monitoring indicators. A pilot study with university volunteers verified the plausibility of detecting human activity indicators directly related to QoL.

1. Introduction

Smartphones currently represent a significant and growing presence in people’s daily lives worldwide. The number of phone users in the world is forecast to grow from 2.1 billion in 2016 to around 2.5 billion in 2019. Android, with 80% of all phones sales, leads the market. In contrast, about 15% of all the phones sold to end customers have iOS as their operating system [1]. The widespread use of phones offers new opportunities for the development of mobile apps in areas such as cancer healthcare [2].
Many phones are now equipped with various sensors, including accelerometers, Wi-Fi, light and temperature sensors, gyroscopes, barometers, etc. These sensors have become rich data sources for measuring various aspects of the user’s daily life. The typical physical activities include walking, jogging, sitting, etc. Due to their unobtrusiveness, low or no installation cost, and being easy to use, phones are becoming the leading platform for human activity recognition [3].
The most popular sensors are those related to user location and allow the provision of mobile services that are tailored to the user’s location, namely location-based services [4]. Other sensors, like accelerometers, have been used in activity recognition tools, within the popular area of activity trackers and fitness apps [5].
The above examples are related to general-purpose applications that facilitate and improve the user’s life in a certain way, but they are not medical applications, nor are they widely used by healthcare professionals. According to Higgins [6], healthcare providers can benefit from the development of those mobile services based on sensor data; phone apps can be used to monitor users’ health or fitness parameters, by using built-in sensors such as the accelerometer, microphone, speaker, and camera. For instance, this type of application can be used by healthcare professionals to monitor patients between visits [6]. Further, researchers are now tapping into using smartphones for medical diagnostics such as [7,8].
The present trend is towards applications which can be used to collect data on patients’ quality of life (QoL) [9]. Studies have been published on the feasibility of using sensor data to monitor mobile phone users’ QoL. A recent study showed the feasibility of using passively collected mobile phone keyboard metadata to predict signs of manic depression via clinician-administered rating scales [10]. This could be promising for cancer patients, due to the prevalence of depression in this group [11].
In this context we developed the Human Dynamics Reporting Service (HDRS) and here propose it as a mHealth solution to assist cancer patients. The system is based on the multiparametric continuous acquisition of movement, location, phone calls, conversations and use of data to simultaneously monitor the physical, psychological, social and environmental activity of patients undergoing oncological treatment. These features are incorporated in the phone application known as HDRS. HDRS monitoring indicators were also defined and mapped with standardized QoL items for cancer patients, the European Organization for the Research and Treatment of Cancer Quality of Life Questionnaire (EORTC QLQ-C30). A pilot study was conducted on university volunteers to validate the detection of physical and social activity indicators directly related to QoL.

Other New Approaches in Monitoring Cancer Patients’ Quality of Life

The World Health Organization defines QoL as a multi-dimensional concept formed by the physical, psychological, social and environmental domains [12]. Chronic diseases such as cancer have a strong influence on both physical health and QoL, which together comprise the actual concept of health-related quality of life (HRQoL), the complete state of physical, social, and psychological functioning [13]. The evidence that QoL is a prognostic variable for cancer patient survival has led to its increased routine incorporation into clinical research [14], so that improving survival rates and preserving the patient’s HRQoL [15] has become a major goal in the treatment of cancer.
Patient-reported outcomes (PROs) play an increasingly important role in complementing clinical outcome parameters in measuring HRQoL. In fact, PROs are currently the accepted gold standard for collecting data about patients’ subjective perception of their own state of health [16]. One of the most significant PROs for HRQoL measurement in cancer patients is provided by the European Organization for the Research and Treatment of Cancer (EORTC) [17], the quality of life questionnaire QLQ-C30 version 3.0 (EORTC, Brussels, Belgium) [18].
As a result of technological advances, apart from the phone sensing applications already mentioned, other approaches have been proposed to assist patient’s health, such as electronic patient-reported outcomes (e–PRO), wearable fitness trackers and action-monitoring or actigraphy. These approaches facilitate the daily monitoring of patients’ health and could reduce the current barrier to continuous monitoring of cancer patients in clinical trials. The new actigraphy devices in particular provide non-intrusive and non-stigmatizing monitoring of outpatients for weeks or even months and facilitate treatment outcome measure during daily activities [19].
Applications such as e–PRO installed on the patients’ own phones can be employed to collect QoL data remotely [9]. Systems like e–PRO are used in oncology clinical care because of their ability to: (i) enhance clinical care by flagging essential symptoms and saving clinician’s time; (ii) improve the availability of standardized methods for creating and implementing PROs in clinics; and (iii) facilitate the access of user-friendly platforms for patient self-reporting, such as tablets, computers and automated telephone surveys [20]. Recently, Wallwiener et al. (2017) [21] evaluated an e–PRO version of the PRO EORTC QLQ-C30 and concluded that it was reliable for patients with breast cancer, showing a high correlation in almost all questions (and on many scales).
New developments in wearable activity monitors mean they can now collect objective patient activity data in a non-obtrusive manner [22]. Continuous physical and physiological monitoring in any environment would shorten hospital stays, improve both the recovery and reliability of diagnosis and improve patients’ QoL. Apart from physiological measurements, the daily physical activity of chronic patients reflects the quality of their daily lives [23].

2. Materials and Methods

In this section we describe both the HDRS application and the pilot study designed to test its feasibility as a potential tool for monitoring cancer patients, through movements, location, conversations, and data using sensors in their phones. First, we give the technical data of HDRS and explain the workflow of the web application in charge of registering users and saving the data obtained. We then describe the participants and data collection (definitions of subjects’ characteristics and prediction variables). A hypothesized association between HDRS monitoring variables and the constructed scales of the EORTC QLQ-C30 Version 3.0 [18] is also given.

2.1. Mobile Application Human Dynamics Reporting Service (HDRS)

This section gathers information on the application requirements functionality and workflow and explains the workflow of the web application in charge of registering users and saving the data obtained from them.

2.1.1. Requirements

In order to obtain the necessary physical and social information for the functioning of the application, it is necessary to have access to the following sensors:
  • Accelerometer: obtains the data related to the user’s movements.
  • Microphone: obtains the data related to the sound surrounding the user.
  • In order to run the application correctly, the phone must have an Android operative system with the sensors mentioned above, plus Wi-Fi + 4G to obtain the data related to location, and also a sim card for calls.

2.1.2. Application Workflow

We here describe the different steps in the HDRS workflow:
  • Agreement: A screen with the study conditions is displayed to the user before starting. This is the electronic adaptation of the "participant informed consent model" provided by our university, Universitat Politècnica de València (UPV). If the user does not accept the conditions by clicking on the accept button, he will not be able to go to the Boot screen.
  • Boot: The app can be booted in two ways:
    When it is initialized and the user is not registered, the registration form is shown.
    When the app is booted and the user is already registered, the agreement menu is shown.
  • Registration Form: asks the user for the following information (Figure 1):
    Gender: Male or female.
    Weight (kg): The weight of the user in kg.
    Height (cm): The height of the user in cm.
    Country: Country of residence.
    Occupation: The daily physical activity of the user depending on his occupation.
    Level of activity/exercise: The workout activity related to the user’s activities.
  • Permissions: The following permissions are required for running the application properly:
    Enabling the app to run in the background.
    Enabling microphone use.
    Access to call register.
    Access phone location.
    Acceptance of Agreement shown when the application is booted.
  • Utilities: We included the following functions to facilitate data acquisition and processing:
    Background: When the app is running in the background, a notification is shown for the user to stop the application, synchronize data to the server, or tag an event.
    Tagging an event: When the user is accessing the tag event action from the main menu, a message dialog is displayed to add the description of the current event (Figure 2).
    Synchronize data: When the user is accessing data sync from the main menu, a message is shown when data is sent and another to confirm end of sync.
    Closing the app: When closing the app from the menu, the service will finish and the application will close.

2.1.3. Web Application

The web application functions are: (1) registering users on the database; (2) storing user data obtained from the application; and (3) showing the users’ table.
The web service is built in Ruby, using the Sinatra framework to build the web application to operate with the database and users’ data.
The registration process is made on the application, and the values are sent to be stored in a database on the web. The information obtained by the sensors is also sent to the web application. When it is received, this information is stored in the Google Drive folder created for this app use.

2.2. Data Colletion

Ethical approval was obtained for the study from the Ethical Committee of the Universitat Politècnica de València (UPV, Ethical Code: P8_12_11_2018). The subjects were volunteers from the Instituto de Tecnologías de la Información y Comunicaciones (ITACA) recruited face-to-face and by email invitations. All the participants gave their consent via the app.

2.2.1. Participants

The participants received a brief explanation of the study objectives and the particular purpose of the HDRS application. They were also taught how to use the app functions; agreement, registration data, event tagging, and data synchronization. A total of five volunteers participated in the study.
The average age of the participants was 36.2, with a maximum of 45 and minimum of 21.

2.2.2. Sensor Data Collection

The following sensors were used to collect data for our purpose:
  • The accelerometer obtained the linear acceleration of the device with a vector of three components (using the XYZ axis) excluding the value of gravity to give more realistic results related to movements. The values of each axis were measured in m/s2. When the data was saved to a CSV file, it computed the amount of movement by calculating the magnitude of the vector. The final result is measured in m/s2.
  • Internet usage is the amount of data received and sent through the device measured in Megabyte. The time started when the application was booted.
  • The microphone captured the frequency of the sound and calculated the average frequency of 100 sound samples obtained in 5 s to detect conversation. If the average sound frequency in one minute was greater than 50 Hz, then the value given was 1. If there was no noise it was saved as a low frequency near 0 or −1.
  • Position calculated the location of the user using Wi-Fi and cell tower networks. The values may have had an offset of 25 m and were saved as latitude and longitude.
  • Calls sensor registered calls in and out.
The sampling frequencies considered for the sensors were the following:
  • Accelerometer: The number of movements was calculated every 60 s.
  • Internet usage: The amount of data sent and received was calculated every 60 s.
  • Microphone: To detect conversations it recorded 5 s of audio, obtaining 100 samples of sound frequency. This measurement was repeated every 60 s.
  • Location: Since the position depended on the API used, the minimum sampling time was 60 s, but may have been longer.
  • Phone calls: Calls in and out were calculated every 60 s.

2.2.3. Defined HDRS App Monitoring Indicators

A total of six monitoring indicators were defined and calculated from the data collection sensors:
  • Amount of movement (m/s2), provided by the accelerometer sensor. Movements per hour were calculated by summing all the samples in a minute by the user’s accelerometer sensor.
  • Distance from a specific place (km) by the location sensor. The distance travelled represented the number of meters travelled per hour by the user by summing all the samples in a minute.
  • Talk indicator was provided by the microphone. We assigned the value "1" if a conversation or other sources of sound were obtained and “0” if not, in minutes per hour.
  • Data traffic (sent data + received) Mb provided by the smartphone. The mobile data traffic indicator represented the number of megabytes consumed per hour and was calculated as the sum of the received and sent data in Mb every minute.
  • Number of phone calls (number of calls/min) made + received in a minute.
  • User activity tag events.

2.2.4. Data acquisition protocol

Sensor data from the participants concerning the accelerometer, internet usage, microphone, location and phone calls was acquired for a period of seven days (24 h) with the acquisition frequency defined in Section 2.2.2 (mostly every 60 s). The participants sent the data obtained by the sensors to the web application once a day by the app´s synchronize data function (Section 2.1.2). On reception, this information was stored in the Google Drive folder created for the use of the app. The data could be accessed and downloaded for calculating defined HDRS app monitoring indicators with the user list URL (Section 2.2.3).
The participants who recorded less than three days of 24 h data acquisition from all the sensors were discarded from the pilot study.

3. Results

This section gives the results obtained from the pilot study with the HDRS application. First, we show each monitoring indicator from the data acquired by the HDRS app from the different sensors and then indicate the related EORTC QLQ-C30 questionnaire items. We also point out that other information on the user’s physical, cognitive, and social aspects can be obtained from them.
We also give a summary of the proposed mapping between HDRS monitoring indicators and items in the EORTC QLQ-C30, followed by the results of HDRS app usability test with the participants.

3.1. User Movements

Figure 3 shows a graph of the user indicator values monitored in a week. The increased number of driving, walking and working-out movements and the decreased activities such as working, eating, or watching TV can be seen thanks to the event tags entered. Figure 4 shows differences between participants in the movement indicator in one day. One of the participants was excluded from this figure due to a problem with the accelerometer and missing data. It should be noted that in general, lower movement values were recorded at night (22 pm to 7 am), while they increased during the rest of the day.
The user movements computed weekly can be applied to assess the following EORTC QLQ-C30 physical and role functioning items:
  • Strenuous physical activities (Item 1 of EORTC QLQ-C30). This item is associated with a high number of movement values. In Figure 3 working out is indicated and the corresponding increase in movements.
  • Long/short walk (Items 2 and 3 of EORTC QLQ-C30). Here, the amount of movement would be higher than in sedentary activities such as watching TV.
  • Need to stay in bed (Item 4 of EORTC QLQ-C30) or sitting at home. In this case, the number of movements will be very few, close to nil.
  • Limited in work or other daily activities; limited in hobbies/leisure time activities (Items 6 and 7 in EORTC QLQ-C30). Fewer movements are assumed than in regular activities.

3.2. Distance Traveled

Figure 5 shows all the distances travelled by a user in a week. The hours when the device is kept still are not shown on the chart. Longer distances were registered when driving or walking. Figure 6 shows the differences between the participants in distance travelled in one day. Late at night no distances were registered. In general, long distances in one hour were due to vehicle transfers.
Apart from the registered tagged events, the user’s speed could be used to determine if the subject was walking or running. For example, the mean comfortable gait speed ranged from 127.2 cm/s for women in their seventies to 146.2 cm/s for men in their forties. The mean maximum gait speed ranged from 174.9 cm/s for women in their seventies to 253.3 cm/s for men in their twenties [24].
The distance-travelled weekly indicator could also be associated with the following EORTC QLQ-C30 physical and role functioning items:
  • Strenuous physical activities (Item 1 of EORTC QLQ-C30). A long-distance walk or run could be considered as strenuous physical activity.
  • Long/short walk (Item 2 and 3 of EORTC QLQ-C30). Tags added by the user were useful to identify when the user was walking, while the distance-travelled indicator could distinguish short from long walks.
  • Need to stay in bed (Items 4 of EORTC QLQ-C30). This indicator, combined with data from other sources, may be associated with no distance travelled.
  • Limited work or other daily activities; limited hobbies/leisure time activities (Items 6 and 7 in EORTC QLQ-C30) were associated with a significant decrease in the distance travelled compared with other reference periods of regular activity.

3.3. Mobile Data Traffic and Conversation

Figure 7 shows the data traffic (Mb) and conversations of a user in a week. Conversation is frequent when eating, working, walking, and watching TV, showing the relationship between conversation and social activities, although filters are needed to differentiate between conversation and sounds from the TV or the environment. Figure 8 shows the differences between the participants’ data traffic in a day. In general, more data traffic is registered in the early afternoon (2–3 PM) and night (8 PM–11 PM). These periods could be associated with rest activities between 4 am and 8 am (central sleeping hours), when no data traffic was recorded.
The mobile data traffic indicator could help to identify the user’s activity if compared with standard phone consumption when viewing a video, listening to music, writing or talking in a forum, social networking, etc. This should also include data downloaded from a Wi-Fi connection.
The mobile data traffic and talk indicators (weekly) could be associated with social and cognitive functioning EORTC QLQ-C30 items:
  • Cognitive functioning (Item 20 of EORTC QLQ-C30 scales). This was considered for determining focus/concentration difficulties. High hourly consumption of Mb may indicate that the user was concentrating on an activity such as writing, using apps, in social media, searching, reading, chatting and sharing photos, video, watching, etc.
  • Social functioning (Items 26 and 27 of EORTC QLQ-C30). This could be used to find out if the physical condition and its relation to the medical treatment have interfered with the user’s family life and social activities. Regular social functioning should be associated with frequent conversations and data traffic, mainly from checking and participating in social networks.
  • Limited work or other daily activities; limited hobbies/leisure time activities (Items 6 and 7 of EORTC QLQ-C30). These items were associated with reduced data traffic and talk frequency compared with the user´s periods of reference with a regular activity.

3.4. Number of Phone Calls

The number of calls in and out was considered a poor indicator. Phone calls have now been replaced by internet calls or chat messages. In the pilot study the mean number of calls was 3.4. While two users registered zero calls in a week, others registered four, six and seven.

3.5. Tag Events

A difference was detected between the event tags and the expected records of the sensors. The users indicated that they had forgotten to tag but did so once the action had started or even when it had finished.

3.6. Summary of Mappings between HDRS Monitoring Indicators and Items Of The EORTC QLQ-C30

The EORTC QLQ-C30 (Version 3.0) [18] questionnaire is composed of both multi-item scales and single-item measures. These include five functional scales, three symptom scales, a global health status/QoL scale, and six single items. Each of the multi-item scales includes a different set of items, with no item appearing in more than one scale. All the scales and single-item scores range from 0 to 100. A high scale score represents a higher response level. Thus, a high score for a functional scale represents a high/healthy level of functioning, a high score for the global health status/QoL represents a high QoL, but a high score for a symptom scale/item represents a high level of symptomatology/problems. The principle for scoring these scales is normalized as follows; firstly, by estimating the average of the items that contribute to the scale (this is the raw score); and secondly, by using a linear transformation to standardize the raw score, so that scores range from 0 to 100. A higher score represents a higher (“better”) level of functioning, or a higher (“worse”) level of symptoms [25].
The associations between HDRS monitoring indicators and items of the EORTC QLQ-C30 (Version 3.0) questionnaire are summarized in Table 1.

3.7. Calculating QoL Global Score from HDFS Monitoring Indicators

To obtain a global QoL score from the defined HDRS monitoring indicators, the following procedure was proposed according to the EORTC QLQ-C30 questionnaire [26]:
First, convert monitoring indicators associated with QoL items in a range between one and four to indicate frequency of occurrence. Such items are scored in EORTC QLQ-C30 questionnaire as; (1) “not at all”, (2) “a little” (3) “quite a bit” (4) “very much”. This was necessary to be able to validate the app comparing QoL scores obtained from it with the EORTC QLQ-C30 questionnaire scores. In the EORTC QLQ-C30 questionnaire each item has four response alternatives, except for the global health-status/quality of life scale, which has response options ranging from (1) “very poor” to (7) “excellent”. Then calculate the Global QoL Score according to Equation (1):
Global QoL Score = (Functional Scales + Symptom Scales + Global Health Status)/NS,
with NS, the total number of scales;
Functional scales = PFS + RFS + EFS + CFS + SFS
Symptom scales = FAS + NAS + PAS + DYS + IMS + APS + COS + DIS + FIS
where PFS is the physical functioning score (movement, distance indicators). RFS is the role functioning score (movement, distance indicators). EFS is the emotional functioning score. CFS is the cognitive functioning score (data traffic indicator). SFS is the social functioning score (talk, data traffic, call indicators). FAS is the fatigue score. NAS is the nausea and vomiting score. PAS is the pain score. DYS is the dyspnea score. IMS is the insomnia score (talk, data traffic, call indicators). APS is the appetite loss score. COS is the constipation score. DIS is the diarrhea score. FIS is the financial difficulties score.
The principle for scoring functional and symptom scales are:
  • Estimate the average of the items (app indicators) that contribute to the scale; this is the raw score.
    Raw Score (RS) = (I1 + … + In)/n
  • Use a linear transformation to standardize the raw score, so that scores range from 0 to 100; a higher score represents a higher (“better”) level of functioning, or a higher (“worse”) level of symptoms.
    Functional Scales: Score = 1 − [(RS − 1)/range] × 100
    Symptom scales/items and Global health status/QoL: Score = [(RS − 1)/range] × 100
Thus, a high score for a functional scale represents a high/healthy level of functioning, a high score for the global health status/QoL represents a high QoL, but a high score for a symptom scale/item represents a high level of symptomatology/problems.
It should be mentioned that future versions of the app will include symptoms and emotional function tagging and their corresponding app indicators. Global health status questions will also be included to calculate the global health status.

3.8. Usability Assessment (System Usability Scale Questionnaire)

To assess the usability of the HDRS app, the participants in the pilot study responded to the system usability scale questionnaire (SUS). SUS calculated a single score representing a measure of the overall usability of the system. To calculate the SUS score, we first added the score contributions from each item. Each item’s score contribution ranged from zero to four. For items 1, 3, 5, 7, and 9 the score contribution was the scale position minus one. For items 2, 4, 6, 8 and 10, the contribution is five minus the scale position. Finally, we multiplied the sum of the scores by 2.5 to normalize the final score between 0 and 100 [26].
Figure 9 shows USU scores by participant; only one participant has a score of 65, under the threshold of 68 points (50th percentile) [27], which represents an acceptable usability level. The USU mean score is 75, considered as good. However, in these preliminary results we are aware that the low sample size is a limitation of this usability study. Further usability improvements and assessments are planned.

4. Discussion

In a one week period, the HDRS application was shown by some users to be a feasible tool for monitoring user activities associated with QoL via their smartphone sensors. However, they reported problems regarding battery and memory consumption, forgetting tagging events and data synchronization. Two technical incidents were also registered; missing user registered data and pauses in the sensor data registry due to low batteries.
In this work preliminary relationships were set up between items associated with cancer patients’ QoL and the variables monitored by HDRS, although it is necessary to extend the study to validate these hypotheses. Future studies would include the evaluation of the QoL of patients undergoing oncological treatment by the EORTC QLQ-C30 questionnaire, HDRS monitoring for a week, as well as a subsequent analysis of the correlations between QoL items and HDRS indicators.
Apart from the EORTC QLQ-C30 questionnaire, other indicators related to QoL could be considered and linked with the HDRS app’s monitoring data and indicators, such as patterns of daily/weekly physical activity, regularity of physical activity, types of physical activity (walking, cycling, running), use of means of transportation, patterns of daily/weekly social activity, sleep periods, etc. In this way, nonlinear models can be considered for describing these patterns and for contributing to the improvement of cancer treatments [28].
In relation to the variables associated with user movement and distance travelled, it is important to highlight their relationship with the user´s physical activity. They are important because physical activity is perceived to be an effective non-pharmacological therapy in cancer patients by relieving the distress caused by physical and psychological symptoms. A growing body of evidence supports the idea that increasing physical activity provides important benefits by promoting psychological and physical well-being in cancer patients. Engaging in physical activity is related to improved HRQoL among breast cancer survivors [29]. Wearable fitness trackers, such as Fitbit, which measure daily steps may be useful tools to evaluate and help cancer patients’ treatments [30]. Bade et al. (2018) [31] recently showed the feasibility of using a Fitbit accelerometer to measure step counts in advanced-stage lung cancer patients. Correlations with high QoL, low depression, and low symptom scores with the more active patients were found. Such results can be relevant for both the physical and psychological domains of the QoL concept.
It is also possible to estimate user sleeping periods from HDRS data since the registers of HDRS indicators diminish to almost null during these periods. It is well known that sleep problems are a common cause of reduced QoL [32]. Different clinical issues can cause sleep problems, and often physical activity can be used to alleviate the symptoms [33]. Another critical factor, which is also linked to physical activity and sleep quality, is cancer-related fatigue, unless it is caused by mental health issues such as anxiety and distress. To monitor these aspects, it is common to use a set of methods collectively referred to as ecological momentary assessments (EMA) [34]. These are often complemented by wearable devices for tracking physical activity and sleep objectively [35].
On the other hand, while the promise of mHealth is that we can leverage the power and ubiquity of mobile and cloud technologies to monitor and manage side effects and treatment outside the clinical setting, it is essential to be attentive to usability and to keep the intended users, tasks and considered environments in mind [36]. The results obtained show the HDRS application utility as a potential tool for monitoring user activity. Although the tests carried out in the study are satisfactory from the perspective of the tool´s functional capacity, the future use of the app by cancer treatment patients requires an in-depth study of fundamental aspects such as user-centered design and usability. Adapting the app’s interaction to the specific characteristics and needs of cancer patients is one of the main prerequisites for developing a useful and effective mHealth application. At present, there are several proposals to help developers and researchers in the development, integration, and evaluation of mobile health services that support cancer patients in managing their health-related issues [2,37].
In addition to usability, many factors can influence how data is understood and can be used for making decisions. In the case of mobile and wearable health data, we face the additional challenges that in most cases the data can be sparse, hard to analyze and most healthcare professionals are not familiar with its use. Data-driven cancer care might require the development of tools looking into the whole concept of exposome informatics [38]. Another related challenge for the accomplishment of this vision is the semantic interoperability between clinical data and PROs/lifestyle data that can be tackled by the use of semantic health technologies to better capture the context of the patient using mobile and wearable technologies. In fact, the creation of ontologies to support knowledge-inferring from mobile and wearable devices has already been considered [39].

5. Conclusions

In this study we assessed the feasibility of monitoring the quality of life of cancer patients by phone sensors in a direct approach due to the need to monitor QoL as a clinical outcome in clinical trials for new cancer treatments. Most of the SoA (Service-Oriented Architectures) is now done using expensive/specific devices. Although the EORTC QLQ-C30 questionnaire has been previously adapted to the smartphone app format [40,41], and applied to assess the effects of a smartphone app intervention in QoL of patients with cancer [42], none of the previous research involved using the phone´s sensors to estimate QoL as defined in the EORTC QLQ-C30 questionnaire. Our approach was to map phone sensors to the EORTC QLQ-C3, which would bring a massive amount of technology within the reach of clinical trials, giving access to clinical outcomes of a large number of patients. Our study; (1) shows the feasibility of capturing relevant QoL information, (2) offers research groups an open access phone application for human dynamics reporting, (3) proposes the novel mapping between standardized QoL items for cancer patients (EORTC QLQ-C30) and HDRS monitoring indicators. The HDRS app is able to monitor users’ daily/weekly activities through indicators of their movements, distance travelled, conversations and data traffic. It also allows users to report their own activity by using the event-tagging app functionality. In addition, a graphic representation was developed for the indicators to facilitate their analysis.
The next step in this research program will define the specific method of converting phone sensor data to individual QoL item values defined in the EORTC QLQ-C30 and mapped in the present work. It will involve the acquirement of at least 200 samples following our smartphone sensor approach for evaluating QoL and the estimation of QoL items by multiple linear/non-linear regression of smartphone sensor data. Alternatively, we will calculate QoL items following a rule-based approach, e.g., Item 2 EORTC QLQ-C30: “Do you have any trouble taking a long walk?” could be calculated by following the steps: (1) Calculate the distance travelled in meters daily (dd) from the latitude and longitude records collected every 60 s by the location sensor. The Harvesine equation [43] would be used to calculate the value of the distance travelled. (2) Count the number of times the distance travelled daily is greater than or equal to the reference value of ‘long walk’ in meters according to EORTC QLQ-C30 in a week. If dd = 0, record value item 4 (“very much”); if dd = 1 item record value 3 (“remove a bit”); dd = 2 record value item 2 ("a little"); dd ≥ 3 record value item 1 (“not at all”). These values will be adjusted after experimentation and clinical judgment. A pilot study involving patients undergoing oncology will also be required to test the feasibility of HDRS monitoring and validate the proposed QoL indicators.
Moreover future work will also address the improvement of app design by including automatic recognition of user activities (walking, running, consulting, chatting, etc.), improving the app’s performance and the graphical interface under usability criteria, defining a set of activity tags to be selected by the user instead of writing text and a tag reminder pop-up, periodic auto-sending of recorded data, definition of new mobile sensor indicators associated with cancer patients’ QoL, the integration of graphs of QoL indicators into the app, and their adaptation to iOS.
Finally, we are aware that the study is limited as regards the size of the participant’s sample and the length of the data acquisition period. Although the results are promising, larger samples will be needed to validate the technical issues and validate the defined indicators.

Author Contributions

Funding acquisition, J.M.G.-G.; investigation, S.A.-C., Á.S.-G., J.A.C., C.S., A.R.-R. and J.M.G.-G.; software, Á.S.-G.; supervision, S.A.-C. and J.M.G.-G.; writing—original draft, S.A.-C., Á.S.-G., J.A.C., C.S., A.R.-R. and J.M.G.-G.

Funding

Funding for this study was provided by the authors’ various departments, and partially by the CrowdHealth Project (Collective Wisdom Driving Public Health Policies (727560)) and the MTS4up project (DPI2016-80054-R).

Acknowledgments

The authors gratefully acknowledge the support provided by Luis Fernandez–Luque of the Qatar Computing Research Institute and expert in digital health.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Number of Smartphone Users Worldwide from 2014 to 2020 (in Billions). Available online: https://0-www-statista-com.brum.beds.ac.uk/statistics/330695/number-of-smartphone-users-worldwide/ (accessed on 1 October 2018).
  2. Mirkovic, J.; Kaufman, D.R.; Ruland, C.M. Supporting cancer patients in illness management: Usability evaluation of a mobile app. J. Med. Internet Res. 2014, 2, e33. [Google Scholar] [CrossRef] [PubMed]
  3. Su, X.; Tong, H.; Ji, P. Activity recognition with smartphone sensors. Tsinghua Sci. Technol. 2014, 19, 235–249. [Google Scholar] [CrossRef]
  4. Schmitz Weiss, A. Exploring news apps and location-based services on the smartphone. J. Mass Commun. Q. 2013, 90, 435–456. [Google Scholar] [CrossRef]
  5. Silva, P.A.; Holden, K.; Nii, A. Smartphones, smart seniors, but not-so-smart apps: A heuristic evaluation of fitness apps. In Proceedings of the International Conference on Augmented Cognition, Toronto, ON, Canada, 17–22 July 2014; Springer: Cham, Switzerland, 2014; pp. 347–358. [Google Scholar]
  6. Higgins, J.P. Smartphone applications for patients’ health and fitness. Am. J. Med. 2016, 129, 11–19. [Google Scholar] [CrossRef] [PubMed]
  7. Rivenson, Y.; Ceylan Koydemir, H.; Wang, H.; Wei, Z.; Ren, Z.; Günaydın, H.; Ozcan, A. Deep Learning Enhanced Mobile-Phone Microscopy. ACS Photonics 2018, 5, 2354–2364. [Google Scholar] [CrossRef]
  8. Priye, A.; Ball, C.; Meagher, R. Colorimetric-Luminance Readout for Quantitative Analysis of Fluorescence Signals with a Smartphone CMOS Sensor. Anal. Chem. 2018, 90, 12385–12389. [Google Scholar] [CrossRef] [PubMed]
  9. Measuring Quality of Life for Cancer Patients: Where Are We Today and Where Are We Headed Tomorrow? Available online: http://blog.mdsol.com/measuring-quality-of-life-for-cancer-patients-where-are-we-today-and-where-are-we-headed-tomorrow/ (accessed on 20 August 2018).
  10. Zulueta, J.; Piscitello, A.; Rasic, M.; Easter, R.; Babu, P.; Langenecker, S.A.; McInnis, M.; Ajilore, O.; Nelson, P.C.; Ryan, K.; et al. Predicting Mood Disturbance Severity with Mobile Phone Keystroke Metadata: A BiAffect Digital Phenotyping Study. J. Med. Internet Res. 2018, 20, e241. [Google Scholar] [CrossRef]
  11. Caruso, R.; Giulia Nanni, M.; Riba, M.B.; Sabato, S.; Grassi, L. Depressive Spectrum Disorders in Cancer: Diagnostic Issues and Intervention. A Critical Review. Curr. Psychiatry Rep. 2017, 19, 33. [Google Scholar] [CrossRef]
  12. Harper, A.; Power, M.; The WHOQOL Group. Development of the World Health Organization WHOQOL-BREF quality of life assessment. Psychol. Med. 1998, 28, 551–558. [Google Scholar]
  13. Sosnowski, R.; Kulpa, M.; Ziętalewicz, U.; Wolski, J.K.; Nowakowski, R.; Bakuła, R.; Demkow, T. Basic issues concerning health-related quality of life. Cent. Eur. J. Urol. 2017, 70, 206–211. [Google Scholar] [CrossRef]
  14. Sloan, J.A. Metrics to Assess Quality of Life After Management of Early-Stage Lung Cancer. Cancer J. 2011, 17, 63–67. [Google Scholar] [CrossRef] [PubMed]
  15. Bordoni, R.; Ciardiello, F.; vonPawel, J.; Cortinovis, D.; Karagiannis, T.; Ballinger, M.; Sandler, A.; Yu, W.; He, P.; Matheny, C.; et al. Patient-Reported Outcomes in OAK: A Phase III Study of Atezolizumab Versus Docetaxel in Advanced Non–Small-cell Lung Cancer. Clin. Lung Cancer 2018, 19, 441–449.e4. [Google Scholar] [CrossRef] [PubMed]
  16. Hartkopf, A.D.; Graf, J.; Simoes, E.; Keilmann, L.; Sickenberger, N.; Gass, P.; Wallwiener, D.; Matthies, L.; Taran, F.A.; Lux, M.P.; et al. Electronic-Based Patient-Reported Outcomes: Willingness, Needs, and Barriers in Adjuvant and Metastatic Breast Cancer Patients. JMIR Cancer 2017, 3, e11. [Google Scholar] [CrossRef] [PubMed]
  17. Aaronson, N.K.; Ahmedzai, S.; Bergman, B.; Bullinger, M.; Cull, A.; Duez, N.J.; Filiberti, A.; Flechtner, H.; Fleishman, S.B.; de Haes, J.C.; et al. The European Organization for Research and Treatment of Cancer QLQ-C30: A quality-of-life instrument for use in international clinical trials in oncology. J. Natl. Cancer Inst. 1993, 85, 365–376. [Google Scholar] [CrossRef] [PubMed]
  18. Bjordal, K.; de Graeff, A.; Fayers, P.M.; Hammerlid, E.; van Pottelsberghe, C.; Curran, D.; Ahlner-Elmqvist, M.; Maher, E.J.; Meyza, J.W.; Brédart, A.; et al. A 12 country field study of the EORTC QLQ-C30 (version 3.0) and the head and neck cancer specific module (The EORTC QLQ-H&N35). Eur. J. Cancer Care 2000, 36, 1796–1807. [Google Scholar]
  19. Fuster-Garcia, E.; Bresó, A.; Miranda, J.M.; García-Gómez, J.M. Actigraphy Pattern Analysis for Outpatient Monitoring. Methods Mol. Biol. 2015, 1246, 3–17. [Google Scholar] [CrossRef]
  20. Bennett, A.V.; Jensen, R.E.; Basch, E. Electronic patient-reported outcome systems in oncology clinical practice. CA Cancer J. Clin. 2012, 62, 337–347. [Google Scholar] [CrossRef] [PubMed]
  21. Wallwiener, M.; Matthies, L.; Simoes, E.; Keilmann, L.; Hartkopf, A.D.; Sokolov, A.N.; Walter, C.B.; Sickenberger, N.; Wallwiener, S.; Feisst, M.; et al. Reliability of an e-PRO Tool of EORTC QLQ-C30 for Measurement of Health-Related Quality of Life in Patients with Breast Cancer: Prospective Randomized Trial. J. Med. Internet Res. 2017, 19, e322. [Google Scholar] [CrossRef]
  22. Gresham, G.; Hendifar, A.E.; Spiegel, B.; Neeman, E.; Tuli, R.; Rimel, B.J.; Figlin, R.A.; Meinert, C.L.; Piantadosi, S.; Shinde, A.M. Wearable activity monitor to assess performance status and predict clinical outcomes in advanced cancer patients. NPJ Digit. Med. 2018, 1, 27. [Google Scholar] [CrossRef]
  23. Avci, A.; Bosch, S.; Marin Perianu, M.; Marin Perianu, R.; Havinga, P.J.M. Activity Recognition Using Inertial Sensing for Healthcare, Wellbeing and Sports Applications: A Survey. In Proceedings of the 23th International Conference on Architecture of Computing Systems, Hannover, Germany, 22–23 February 2010. [Google Scholar]
  24. Bohannon, R.W. Comfortable and maximum walking speed of adults aged 20—79 years: Reference values and determinants. Age Ageing 1997, 26. [Google Scholar] [CrossRef]
  25. Fayers, P.M.; Aaronson, N.K.; Bjordal, K.; Groenvold, M.; Curran, D.; Bottomley, A.; On behalf of the EORTC Quality of Life Group. The EORTC QLQ-C30 Scoring Manual, 3rd ed.; European Organisation for Research and Treatment of Cancer: Brussels, Belgium, 2001. [Google Scholar]
  26. Brooke, J. SUS: A quick and dirty usability scale. In Usability Evaluation in Industry; Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, L., Eds.; Taylor & Francis: London, UK, 1996; pp. 189–194. [Google Scholar]
  27. Bangor, A.; Kortum, P.; Miller, J.A. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. J. Usabil. Stud. 2009, 4, 114–123. [Google Scholar]
  28. Pérez-García, V.M.; Fitzpatrick, S.; Pérez-Romasanta, L.A.; Pesic, M.; Schucht, P.; Arana, S.; Sánchez-Gómez, P. Applied mathematics and nonlinear sciences in the war on cancer. Appl. Math. Nonlinear Sci. 2016, 1, 423–436. [Google Scholar] [CrossRef]
  29. Shin, W.K.; Song, S.; Jung, S.Y.; Lee, E.; Kim, Z.; Moon, H.G.; Noh, D.Y.; Lee, J.E. The association between physical activity and health-related quality of life among breast cancer survivors. Health Qual. Life Outcomes 2017, 15, 132. [Google Scholar] [CrossRef]
  30. UT Southwestern Medical Center. Wearable Fitness Monitors Useful in Cancer Treatment, Study Finds. ScienceDaily, 2018. Available online: www.sciencedaily.com/releases/2018/05/180501130856.htm (accessed on 20 August 2018).
  31. Bade, B.C.; Brooks, M.C.; Nietert, S.B.; Ulmer, A.; Thomas, D.D.; Nietert, P.J.; Silvestri, G.A. Assessing the Correlation Between Physical Activity and Quality of Life in Advanced Lung Cancer. Integr. Cancer Ther. 2018, 17, 73–79. [Google Scholar] [CrossRef] [PubMed]
  32. Fortner, B.V.; Stepanski, E.J.; Wang, S.C.; Kasprowicz, S.; Durrence, H.H. Sleep and Quality of Life in Breast Cancer Patients. J. Pain Symptom Manag. 2002, 24, 471–480. [Google Scholar] [CrossRef]
  33. Mishra, S.I.; Scherer, R.W.; Snyder, C.; Geigle, P.; Gotay, C. Are exercise programs effective for improving health-related quality of life among cancer survivors? A systematic review and meta-analysis. Oncol. Nurs. Forum 2014, 41, E326–E342. [Google Scholar] [CrossRef] [PubMed]
  34. Ratcliff, C.G.; Lam, C.Y.; Arun, B.; Valero, V.; Cohen, L. Ecological momentary assessment of sleep, symptoms, and mood during chemotherapy for breast cancer. Psycho-Oncology 2014, 23, 1220–1228. [Google Scholar] [CrossRef] [PubMed]
  35. Cox, S.M.; Lane, A.; Volchenboum, S.L. Use of wearable, mobile, and sensor technology in cancer clinical trials. JCO Clin. Cancer Inform. 2018, 2, 1–11. [Google Scholar] [CrossRef] [PubMed]
  36. Brown, W.; Yen, P.; Rojas, M.; Schnal, R. Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) technology. J. Biomed. Inform. 2013, 46, 1080–1087. [Google Scholar] [CrossRef]
  37. Darlow, S.; Wen, K. Development testing of mobile health interventions for cancer patient self-management: A review. Health Inform. J. 2016, 22, 633. [Google Scholar] [CrossRef] [PubMed]
  38. Martin Sanchez, F.; Gray, K.; Bellazzi, R.; Lopez-Campos, G. Exposome informatics: Considerations for the design of future biomedical research information systems. J. Am. Med. Inform. Assoc. 2013, 21, 386–390. [Google Scholar] [CrossRef] [PubMed]
  39. Kim, H.H.; Lee, S.Y.; Baik, S.Y.; Kim, J.H. MELLO: Medical lifelog ontology for data terms from self-tracking and lifelog devices. Int. J. Med. Inform. 2015, 84, 1099–1110. [Google Scholar] [CrossRef] [PubMed]
  40. Lamber, P.; Mitterer, M.; Napolitano, L.; Ricci, F.; Zini, F. Surveying patients with smart devices. In Proceedings of the 25th IEEE International Symposium on Computer-Based Medical Systems (CBMS), Sao Carlos, Brazil, 20–22 June 2012; pp. 1–4. [Google Scholar]
  41. Kessel, K.A.; Vogel, M.M.E.; Alles, A.; Dobiasch, S.; Fischer, H.; Combs, S.E. Mobile App Delivery of the EORTC QLQ-C30 Questionnaire to Assess Health-Related Quality of Life in Oncological Patients: Usability Study. JMIR mHealth uHealth 2018, 6, e45. [Google Scholar] [CrossRef] [PubMed]
  42. Elsbernd, A.; Hjerming, M.; Visler, C.; Hjalgrim, L.L.; Niemann, C.U.; Boisen, K.; Pappot, H. Cocreated Smartphone App to Improve the Quality of Life of Adolescents and Young Adults with Cancer (Kræftværket): Protocol for a Quantitative and Qualitative Evaluation. JMIR Res. Protoc. 2018, 7, e10098. [Google Scholar] [CrossRef] [PubMed]
  43. Van Brummelen, G. Heavenly Mathematics: The Forgotten Art of Spherical Trigonometry; Princeton University Press: Princeton, NJ, USA, 2012; ISBN 9780691148922. [Google Scholar]
Figure 1. Mobile application Human Dynamics Reporting Service (HDRS) registration forms.
Figure 1. Mobile application Human Dynamics Reporting Service (HDRS) registration forms.
Ijerph 16 00461 g001
Figure 2. Mobile application HDRS tag event.
Figure 2. Mobile application HDRS tag event.
Ijerph 16 00461 g002
Figure 3. Amount of movement per hour and event tags in a five day HDRS monitoring period.
Figure 3. Amount of movement per hour and event tags in a five day HDRS monitoring period.
Ijerph 16 00461 g003
Figure 4. Normalized number of movements of participants in a one day HDRS monitoring period.
Figure 4. Normalized number of movements of participants in a one day HDRS monitoring period.
Ijerph 16 00461 g004
Figure 5. Distance travelled by the user (m/h) and event tags in a week of HDRS monitoring.
Figure 5. Distance travelled by the user (m/h) and event tags in a week of HDRS monitoring.
Ijerph 16 00461 g005
Figure 6. Distance travelled by participants (m) in a day of HDRS monitoring.
Figure 6. Distance travelled by participants (m) in a day of HDRS monitoring.
Ijerph 16 00461 g006
Figure 7. Data traffic (Mb) and time of talking or listening to sounds (minutes) by HDRS monitoring.
Figure 7. Data traffic (Mb) and time of talking or listening to sounds (minutes) by HDRS monitoring.
Ijerph 16 00461 g007
Figure 8. Traffic (Mb) in a day.
Figure 8. Traffic (Mb) in a day.
Ijerph 16 00461 g008
Figure 9. Graph of the results of the system usability scale (SUS) questionnaire. All the results exceed the threshold of 68 points (50th percentile) and represent an acceptable level of perceived usability of the system [27].
Figure 9. Graph of the results of the system usability scale (SUS) questionnaire. All the results exceed the threshold of 68 points (50th percentile) and represent an acceptable level of perceived usability of the system [27].
Ijerph 16 00461 g009
Table 1. Preliminary relationships between the European Organization for the Research and Treatment of Cancer Quality of Life Questionnaire EORTC QLQ-C30 (Version 3.0) items and the Human Dynamics Reporting Service (HDRS) monitoring indicators.
Table 1. Preliminary relationships between the European Organization for the Research and Treatment of Cancer Quality of Life Questionnaire EORTC QLQ-C30 (Version 3.0) items and the Human Dynamics Reporting Service (HDRS) monitoring indicators.
EORTC QLQ-C30Smartphone Data Sources
Constructed Scales
(Period: During the Past Week)
Version 3.0
Item Numbers
Amount of Movement (m/s2)Distance from a Specific Place (Km)Talk (0,1)Data Taffic (MB)The Number of Phone Calls (calls/min)
Global health status/QoL
Global health status/QoL29, 30
Functional scales
Physical functioning (strenuous activities, long/short walk; need to stay in bed; help with eating; dressing; washing or using the toilet)1 to 5
Role functioning (limited in work or other daily activities; limited in Hobbies/leisure time activities)6, 7
Emotional functioning (feel tense/worry/irritable/depressed)21 to 24
Cognitive functioning (difficulty in concentrating on things/difficulty remembering things)20, 25
Social functioning (physical condition or medical treatment interfered with your family life/social activities)26, 27
Symptom scales/items
Fatigue (need to rest; felt weak; tired)10, 12, 18
Nausea and vomiting (nauseated; vomited)14, 15
Pain (felt pain/pain interfere with your daily activities)9, 19
Dyspnoea (short of breath)8
Insomnia (trouble sleeping)11
Appetite loss (lacked appetite)13
Constipation (constipated)16
Diarrhoea (had diarrhoea)17
Financial difficulties (physical condition or medical treatment caused you financial difficulties)28

Share and Cite

MDPI and ACS Style

Asensio-Cuesta, S.; Sánchez-García, Á.; Conejero, J.A.; Saez, C.; Rivero-Rodriguez, A.; García-Gómez, J.M. Smartphone Sensors for Monitoring Cancer-Related Quality of Life: App Design, EORTC QLQ-C30 Mapping and Feasibility Study in Healthy Subjects. Int. J. Environ. Res. Public Health 2019, 16, 461. https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph16030461

AMA Style

Asensio-Cuesta S, Sánchez-García Á, Conejero JA, Saez C, Rivero-Rodriguez A, García-Gómez JM. Smartphone Sensors for Monitoring Cancer-Related Quality of Life: App Design, EORTC QLQ-C30 Mapping and Feasibility Study in Healthy Subjects. International Journal of Environmental Research and Public Health. 2019; 16(3):461. https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph16030461

Chicago/Turabian Style

Asensio-Cuesta, Sabina, Ángel Sánchez-García, J. Alberto Conejero, Carlos Saez, Alejandro Rivero-Rodriguez, and Juan M. García-Gómez. 2019. "Smartphone Sensors for Monitoring Cancer-Related Quality of Life: App Design, EORTC QLQ-C30 Mapping and Feasibility Study in Healthy Subjects" International Journal of Environmental Research and Public Health 16, no. 3: 461. https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph16030461

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop