http://orcid.org/0000-0002-0547-2168
Figures
Abstract
Background
The general problem is lack of inter-professional collaboration and the way private primary care responds to manage chronic diseases in Malaysia. Absence of prescription review, inadequate patient education, the highest percentage of prescribing errors and half of the chronic disease patients are nonadherent. Medicines are the most common and life long used interventions in chronic diseases. Hence, the need to manage medicine in chronic diseases becomes obligatory. As both general practitioner and community pharmacist can dispense medications, this has resulted in a business rivalry. There is a need to build consensus among various healthcare stakeholders for a collaborative medication therapy management model (CMTM) where community pharmacist has an active role in chronic care.
Method
This study utilized modified e-Delphi method to build consensus. A validated e-Delphi survey was administered to a purposive sample of 29 experts. Consensus was pre-defined to be the point where >85% of the experts fall in either agree or strongly agree category for each statement. The inter-expert agreement was computed in both rounds using Intra-class correlation coefficient and Kendall's W. Delphi operates in an iterative fashion till there comes stability in responses. At the end of each round, experts were provided aggregate response, their own response and choice to change their response in the light of aggregate response.
Results
Response rate was 70.73% and 100% in 1st and 2nd round, respectively. Consensus was achieved on 119/132 statements which mainly referred to the need, structural and regulatory aspects of CMTM model in Malaysia. However, there were some flashpoints on dispensing separation and means to finance this model. Stability in response of experts was achieved after 2nd round; hence, no next round was executed.
Conclusion
Overall, the study findings witnessed the expert panel’s support for the CMTM model. Study helped to sketch CMTM model and facilitated development of some recommendations to the authorities which may help to formulate a policy to bring CPs under a working relationship with GPs. Hence, this study should be taken as a call for redefining of the roles of CPs and GPs in Malaysia.
Citation: Mubarak N, Hatah E, Aris MAM, Shafie AA, Zin CS (2019) Consensus among healthcare stakeholders on a collaborative medication therapy management model for chronic diseases in Malaysia; A Delphi study. PLoS ONE 14(5): e0216563. https://doi.org/10.1371/journal.pone.0216563
Editor: Christophe Leroyer, Universite de Bretagne Occidentale, FRANCE
Received: January 15, 2019; Accepted: April 23, 2019; Published: May 10, 2019
Copyright: © 2019 Mubarak et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files. However, it is a PhD LEVEL project, so after the thesis is published, it will be in the university repository.
Funding: CSZ was supported by a research grant from The Ministry of Education Malaysia (Fundamental Research Grant Scheme, FRGS 15-195-0436).
Competing interests: The authors have declared that no competing interests exist.
Background
Chronic diseases pose a huge burden on the Malaysian healthcare system. In Malaysia, primary care developments are slow to offer an efficient chronic care delivery for growingly ageing population [1–5]. World Health Organization (WHO) country profile reveals an alarming burden of chronic diseases in Malaysia where 73% of all deaths were related to chronic diseases [6,7]. Malaysia is among the top ten countries in the world in terms of prevalence of diabetes, for instance, National Health and Morbidity Survey-2015 revealed that Malaysia has highest number of diabetic patients (3.5 million) in the region [3]. ″National survey on the use of medicines by Malaysian consumers-2015″ revealed 70.8% respondents were not satisfied by the current level of counselling in the clinics and expressed the dire need of additional education and counselling to manage their medicine and disease state [8]. Furthermore, recent research has highlighted rampant prevalence of inappropriate prescribing in nursing homes and private clinics in Malaysia [9,10]. A systematic review of 17 studies has placed Malaysia on top in South East Asia with the highest percentage (34%) of prescribing errors [11]. These medications errors increase incidence of adverse drug related admissions in hospitals and contribute a significant burden on healthcare system in Malaysia [12].
The general problem is the way private primary care network functions to manage chronic diseases. Numerous studies have highlighted number of apprehensions on the GPs’ practice as a single care provider where GPs diagnose, dispense, educate, counsel all alone or through a medical assistant or a nurse, which is even worst as they have never been trained in medicines as compared to a pharmacist [13,14]. As mentioned, GP can also dispense medications (which itself constitutes a conflict of interest and reason of overprescribing), there is hardly any prescription reviewed by the community pharmacists (CPs) who are at a conveniently accessible position for public. Malpractices have also been documented in the literature on the side of CPs, such as handling complex medical conditions at pharmacy without referral to a GP, dispense medications which require a prescription, and perform dose adjustments or change of the drug therapy without communicating to a GP.
Thus, there are four main problems in Malaysia, which demand and advocate a shift in the healthcare policy to make a system to optimize and rationalize the medicine use in chronic care, these are:
- over and inappropriate prescribing in private GPs clinic [15] and absence of prescription review in private clinics [16–18].
- lack of patient education and knowledge about how to use or store a medication appropriately, or how to avoid or recognise an adverse effect of a medication or what could be possible interactions with other medications or supplements or traditional medicine a patient is taking, [8,19].
- almost 50% chronic disease nonadherent patients [14] due to absence of any adherence support service in private sector [8,20].
- high prevalence of polypharmacy in older adults with chronic diseases [10].
Medicines are the most common and life long used interventions in chronic diseases. Hence, the need to manage medicine in chronic diseases becomes obligatory. However, it requires an effective medicine management model based on a defined system which dually monitors rational prescribing, quality use of medicines and ensures optimal patient education and adherence support in primary care. If there is a system of medicine management, CPs may utilize their medicine expertise to rationalize the use of medicines among chronic disease patients. Prescription review and adherence support offer a platform which can engage CP and GP in collaborative practice to manage medications. The collaboration between GP and CP have successfully resolved many potential drug related problems and have a proven trajectory of improvements in health outcomes in developed countries [21–25]. However, simply replicating a model which was successful in some developed country might not result the same in Malaysia, as each country has its own ground realities. Thus, studies should first seek opinions and aims to build consensus of all the relevant stakeholders to explore all dimensions and then move ahead. This requires, as a first step, open communication between stakeholders to bring them on same page to optimize patient care.
In this situation, a consensus building approach involving relevant stakeholders or experts would be of significant importance as it ensures that everyone’s interests have been considered along with all the distinct perspectives, concerns and inputs [26,27]. Delphi method represents one of the consensus building methods which has been regarded as a structured communication of experts who can offer valuable contribution in order to resolve a complex issue, and thus used as an efficient tool to help decision making, conflict management and identifying priorities, barriers, concepts and best practices [28].
Research objectives
The overall objective of this study was to seek consensus among different healthcare stakeholders for a ″collaborative medication therapy management″ (CMTM) model for chronic diseases in Malaysia which may involve CPs and GPs in an active collaboration for management of medicines and diseases.
The specific objectives of this Delphi study were to:
- To identify the perspectives of an expert panel, comprised of healthcare stockholders/experts, on current situation and need of collaboration between CP and GP for a ‘collaborative medication therapy management’ (CMTM) model for chronic diseases in Malaysia.
- To seek and gauge level of consensus on various aspects of CMTM model (e.g., the regulatory requirements, perceived barriers, administrative aspects and means of financing).
- To measure level of consensus among experts on proposed solutions to various flashpoints/problems between different stakeholders on various aspects of CMTM model.
- To proffer consensus based recommendations to relevant stakeholders in Malaysia on way forward to CMTM.
- To draw a working sketch of the CMTM model based on the consensus of healthcare experts in Malaysia.
Methods
This study was granted ethical approval by the Medical Research Ethical Committee, Ministry of Health, Malaysia (ref: KKM/NIHSEC/P16-1632, NMRR-16-1775-32273) and was also approved by the Research Ethics Committee, International Islamic University Malaysia (ref: IIUM/308/C/1/G1527859).
Survey time frame
The time frame for two rounds of Delphi surveys spanned from August, 2017 to February, 2018 (including 1st round data analysis and; two and three weeks extension in deadline date to respond to survey for 1st and 2nd round, respectively).
Modified e-Delphi survey
This study utilized a modified e-Delphi method, conducted and reported in accordance with the latest guidelines for conducting [29] and reporting [30] a Delphi research. The flow chart of this modified Delphi study is depicted in Fig 1.
CP = community pharmacist, GP = general practitioner, CMTM = Collaborative Medication Therapy Management, IQR = interquartile range, ICC = intra-class correlation coefficient, QP = QuestionPro, MS = Microsoft.
Reasons to choose Delphi method
The research problem under investigation was narrowly defined in Malaysian context. There is a high temperature on the issue in hand, where both stakeholders (GP and CP) hold competing interests (business rivalry as both can dispense because of absence of law for dispensing separation). GPs do not trust clinical skills of CPs while CPs are disappointed being underutilized in patient care. Delphi is appropriate in this situation which involves many stakeholders or there exists a little communication among the stakeholders. Delphi method is free from the bias of dominant individuals who hold superior positions because it does not require experts to meet physically [31] and thus experts remain anonymous and can take part in the survey on their ease (asynchronous participation). Modified Delphi further adds unique advantage to the study for instance, the survey instrument is informed by an extensive review of the literature and thus provide an authentic foundation for 1st round questionnaire [32]. Finally, non-response is generally low in Delphi as pre-research commitments are taken [27,33–35].
Expert selection
In Delphi context, experts are ″informed individuals, specialists, and those with knowledge about a specific subject ″ [36]. The expert panel was selected across Malaysia. For experts’ selection, we adopted the step by step approach as described in literature [35]. In the first step, this approach involves decision about the categories of experts to be included in the panel based on a pre-set qualifying criteria, followed by identification of names of experts (purposive sampling) and asked initial experts to nominate other experts who might were up to criteria of the study (preparation of knowledge resource nomination worksheet). Finally, nominated experts were ranked based on availability and invited experts through an email. Criteria applied to select the experts for this study (Section A in S2 Appendix) were based on the guidelines [37].
To represent GPs, CPs and Nurses, experts were invited from Ministry of higher education, Ministry of health, various professional associations and councils of these professions, such as Federation of private medical practitioners’ association, Malaysian medical council, Malaysian medical association, Malaysian primary care network, Malaysian pharmaceutical society, Malaysian community pharmacy guild and Malaysian nurses’ association. Thus, a heterogenous expert panel was constituted.
A heterogenous (different experts) panel gives more credibility and acceptance than a homogenous panel, because range of viewpoints may be taken, and thus may cover all the possible aspects of issue in hand. For a heterogenous panel, 5–10 experts per group is the optimal number [38]. As the aim of Delphi survey is not to represent a population, hence, Delphi does not use random sampling to recruit a panel of experts, as unlike conventional survey which generally holds an aim of representativeness [39,40]. To avoid any bias, researcher should set and strictly stick to a pre-defined inclusion criteria. Finally, the results of the study would be considered compromised, if the response rate of the experts falls below 75% during rounds [41].
Experts were included from professional bodies or organizations of GPs and CPs because the proposed CMTM model specifically looking for collaboration between these two stakeholders. We added nurses in this study as they are involved in health care delivery at all point of patient care.
Experts were initially approached via email and later by a phone call. They were explained in detail about the study, Delphi method, their responsibilities as a participant in the survey, the time needed to complete the survey and a commitment to complete at least two rounds of the study. They were provided the consent forms and study information sheet which they need to return through email, if they agreed to participate voluntarily. Fig 2 depicts expert selection process.
MREC = Medical Research & Ethics Committee, KRNW = Knowledge Resource Nomination Worksheet, GP = general practitioner, CP = community pharmacist, MoH = Ministry of Health, MHE = Ministry of Higher Education, MPS = Malaysian Pharmaceutical Society, FMS = Family Medicine Specialist, MMC = Malaysian Medical Council, MMA = Malaysian Medical Association.
Ensuring optimal response rate.
Optimal response rate was ensured by following the recommendations of famous Delphi expert, Okoli [35] which included sending experts, three humble reminders, individualized thank you note, announcing an honorarium of 100 Malaysian Ringgit ($ 24.30), and to hold face to face interviews of the experts between 1st and 2nd round which definitely built rapport and created a personal bond with the experts. We also extended the date of submission by one week and three weeks for 1st and 2nd round, respectively, to maximize the response and to accommodate all the experts.
Survey instrument
Survey development through literature review.
The instrument of modified Delphi survey was characterized by directly presenting close ended, pre-determined, well-constructed items (statements) to the expert panel for rating or ranking. The items in modified Delphi survey instrument were informed mainly by the literature review where similar collaboration between CP and GP (in different developed countries) were cited. The detailed search strategy and the articles (which were used to draft the items/statements in Delphi survey) are presented in Section A to C in S1 Appendix. The statements were modified in Malaysian context and assembled in the form of a theme to cover a specific aspect of CMTM model. In draft version, there were nine themes which covered nine specific aspect of CMTM model.
Survey instrument validation.
The draft version was sent as an email attachment to 6 reviewers (3 CP, 3 GP; not part of panel of experts) to provide feedback for the face and content validity. All the six reviewers were PhD in their respective disciplines and have vast experience of survey instrument making and designs. The reviewers were requested to rate all the statements of the survey using a 4-point Likert scale from ″highly relevant″ to ″non-relevant″. An empty column was also provided in Word document for any comments to improve the statements. Experts were asked to comment, delete or edit any item. The modifications made were incorporated accordingly to bring improvements in the survey instrument. This review of the draft enhanced the clarity in phrasing, eliminated any leading or double barrel questions and improved relevance of the questionnaire.
We used, content validity index/universal agreement method to measure content validity as recommended in the literature [42]. Section D in S1 Appendix-I depicts the detailed version of all the review process and presents all 138 statements into four categories based on reviewers’ recommendations i.e., statements modified, not-modified, added or deleted. Any statement with a CVI less than 0.7 was omitted. Finally, the survey emerged with 132 statements assembled in 11 themes.
Web version of the survey (online version).
QuestionPro
After the process of face and content validity on the Word document, the final version of survey instrument was uploaded on the World Wide Web by using an online, flexible and secure survey tool ″QuestionPro″ (QP). QP generates a separate code for each expert to make sure one response per expert. The feature of real time data analysis (analysis as soon as any expert fill the survey) in QP saved a lot of time. QP gave freedom to export the survey results in various downloadable formats for further data analysis for instance, as Statistical Package for Social Scientists (SPSS) file, file for Excel (XML), etc. Furthermore, another unique feature of QP was ″spotlight report″ which assisted experts to view their own response in comparison with collective response of all experts in the panel after the completion of each round. Before sending to the expert panel, weblink of survey was sent to three of the six reviewers to complete the full survey online and provide feedback about the efficiency of the online version, report any missing or non-functional link or unexpected error and to check the type of data received for downloading. There were few problems reported by the experts on the online version and after a formal go ahead the final online version was sent to experts.
Sections of the survey instrument
The final web survey had 4 sections as it appeared online in the sequence given below:
- A brief introduction of the study; with background and objectives clearly stated.
- Instructions for experts on ″how to fill the survey″, such as if an expert missed to give response over any statement, the QP was programmed to show a “validity error” on the screen mentioning the missing area highlighted in red to be easily identified by the expert. Similarly, at the end of each theme, QP gave three options to the experts to: ″provide any comment (qualitative feedback) ″, ″Save and continue later″ (to complete the survey in multiple sitting) or ″Next″ to move to the next theme in the survey. In case of confusion or ambiguity expert would just click on the terminology or word given, a pop-up window would appear at the top right side of the webpage and had complete definition and context in which the term was utilized in the survey. These instructions supported the participants to complete the survey with ease and clarity.
- Demographic details of experts.
- Survey statements (arranged in 11 themes). The fourth part was the main section of the survey. Out of 11 themes, 6 themes (theme 1, 2, 3, 7, 8 and 10) had statements to be rated on a five points Likert-scale (from strongly disagree to strongly agree) while, 5 themes (theme 4, 5, 6, 9 and 11) had statements to be ranked based on the priority or feasibility of various aspects of CMTM model in Malaysian setting.
Complete survey is attached as Section E in S1 Appendix.
Finally, survey instrument was sent to all experts in the form of an openable weblink, embedded in a separate email generated through QP. The link when clicked by the expert opened on QP interface.
The Delphi process including survey instrument making is given in Fig 1.
Defining consensus
It is a fact that a few problems can achieve an absolute 100% agreement among experts. Generally, the consensus is defined by an arbitrary percentage of experts who agree to a statement in a survey, but that percentage must be pre-defined and declared in the survey instrument before the start of the survey [43].
Literature defined consensus as the "gathering of individual evaluations around a median response, with minimal divergence" [44–46]. Another aspect to cross check the point of consensus is the ″stability″ in responses of experts between two consecutive rounds, which must be ensured before concluding iteration in Delphi process [45].
The percentage of agreement is a measure of how many experts agree to a statement and is simply calculated by the number of experts rating ″agree or strongly agree″ to a given statement, divided by the total number of experts. However, for inter-expert agreement, there are various statistical measure, such as median, IQR, Kendall’s W, and ICC. It is recommended to use combination of statistical parameters to report a consensus and not merely relying on the percentage of experts who choose ″agree″ or ″strongly agree″ [47].
The final level of consensus would be measured at the end of 2nd round. The consensus was decided to be accomplished if a statement would receive sum of the percentage of ″agree″ ″strongly agree″ or ″disagree″ ″strongly disagree″ rating by ≥85% of the experts. Thus,
- ≥85% high consensus
- 80% to 84% moderate consensus
- 75% to 79% low consensus
- ≤74% poor or no consensus
For the ranking statements the Kendall’s coefficient of concordance (Kendall’s W), was used to measure the extent of consensus among the experts, in addition to mean rank.
It is important to know when to stop iterations (rounds). Too early stop may contribute to meaningless results and too long would make the process tedious and bring factor of fatigue as bias. Thus, as per literature guidelines, two parameters were planned which would act as indicators to stop the Delphi process in this study [40], they were:
- When the pre-defined consensus level was achieved.
- When there was an evidence of no change of response of experts in two consecutive rounds as defined by Wilcoxon signed rank test of stability.
2nd round’s survey instrument
The instrument for 2nd round followed the same administrative procedures. It was identical to 1st round’s instrument in terms of statements, however, with some additional features provided in MS Word document, these were:
- Instructions to complete the survey in the 2nd round.
- Expert’s own response on the given statement in 1st round.
- Collective group response (response of 29 experts) for each items of the survey, in terms of percentage of people who were in favour (agree/strongly agree) or against (disagree/strongly disagree) a statement in the survey.
Thus, the objective of the 2nd round was the re-evaluation of the statements of instrument by the experts in the light of aggregate response, and reasoning of other experts in the group. It would be sent to all the same experts through an email attachment. At the end of 2nd round, the statements where pre-defined level of consensus could not be achieved would be omitted.
Data analysis
As Delphi operates in an iterative fashion (rounds), data were collected, analysed and shared repeatedly with the experts to review in each round [34]. Literature does not pose hard and fast rules on the exact number of rounds which should be carried out but offers two general stop criteria for iteration as mentioned earlier i.e., stability in response of experts between two consecutive rounds and achieving pre-defined consensus level. If 1st round’s questionnaire is based on literature review, two rounds are sufficient [28,48,49].
Data would be analysed through various statistical operations which are discussed in following sections:
Quantitative data.
All the statistical operations were performed using the Statistical package for social science (SPSS, version 23.0) by exporting the data files of QP in SPSS.
Rating statements
For rating statements in the survey, Likert scale is generally considered as ordinal, hence, tests were applied accordingly. The results were interpreted through descriptive statistics i.e., median, inter quartile range (IQR). Median measures central tendency of the ratings of the experts while IQR is used to check the dispersion. The reason to choose median and IQR was, both are not affected by extreme values and use of both operations are favoured in literature [50].
Ranking statements
For the ranking themes (where experts were asked to rank or prioritize certain items based on feasibility in Malaysia), the mean and priority rank along with median and IQR were computed for each item. QP would automatically compute the mean rank, also called whole rank based on the ranks given by all 29 experts for each item (statement). This helped in creating the order of the statements based on their priority for each theme. Lower the mean rank, the higher the priority rank. Another test was computed for ranking statements i.e., Kendall’s W (coefficient of concordance) test, which calculated agreement among raters for these statements. Different values of W, for instance, less than 0.3, 0.3 to 0.5, 0.5 to 0.7 and 0.7 to 0.9 would be interpreted as weak, moderate, good and strong agreement, respectively.
Intra-class Correlation Coefficient (ICC)
ICC, two-way mixed model (Type A) test was applied over all the themes (both rating and ranking) individually to check the absolute agreement among the experts. This test is suitable for use when the multiple scores or responses of a respondent must be tested to check agreement among raters. We chose absolute agreement instead of individual agreement during the computation of this test as described in [51]. The interpretation of ICC values are as follows: less than 0.3, 0.3 to 0.5, 0.5 to 0.7 and 0.7 to 0.9 indicate as weak, moderate, good and strong agreement, respectively.
Finally, after the completion of both rounds, to authenticate the decision of consensus, Wilcoxon signed rank test was executed on all items of the survey to compute stability of response of experts between two rounds. The null hypothesis (H0) was ″there is no difference in response of respondents between the two rounds″, while the alternative hypothesis (HA) was ″there is a difference in response of respondents between the two rounds″.
Hence, when p > 0.05, null hypothesis would be accepted and there would be no difference between the Delphi rounds.
Qualitative data and narrative synthesis.
The comments made by the experts (if any) would be narrated and utilized to inform the interview guide in the next phase of the study. These comments are provided in Section B in S2 Appendix.
Planning to overcome the limitations of Delphi
Many of the limitations of Delphi, as narrated in literature [52], were managed through a pre-planning which countered various kind of biases in the current study.
Section C in S2 Appendix describes these limitations and how we planned to tackle them.
Results
Experts
Response and completion rate.
Response and completion rate of both rounds of Delphi for all expert categories is detailed in Table 1.
Demographics of expert panel.
The expert panel represents a wide coverage across Malaysia, and experts participated from almost all states as depicted in the Fig 3.
There were equal number of male and female experts in the panel. The median number of years of experience of the experts was 24 years with a range (17–39). The demographic data are provided in Table 2, Fig 3 and Section D in S2 Appendix.
Survey instrument
Validity of the instrument.
Average item level Content Validity Index (CVI) was 0.92, while, Scale level CVI/Universal agreement was computed to be 0.83. Hence, achieved an excellent agreement for both item and scale level validity index.
Delphi rounds.
The average time taken by the experts to complete the 1st round of survey was 44 minutes and 10 seconds. For themes with rating scale (strongly disagree to strongly agree), in the final round (2nd round) consensus was achieved on:
- 72% (n = 13/18) statements of theme-1,
- 78% (n = 7/9) statements of theme-2,
- 100% (n = 16/16) statements of theme-3,
- 69% (n = 9/13) of theme-7,
- 50% (n = 3/6) statements of theme-8 and
- 82% (n = 28/34) statements of theme-10.
The theme-4,5,6,9 and 11 were not related to Likert scale but were ranking in nature and hence their computation of consensus was carried out through Kendall’s W.
The consensus level among experts for rating statements in 1st and 2nd round based on the percentage of agreement and the stability in responses in two consecutive rounds are given in Table 3. Similarly, consensus level based on median, IQR and Kendall’s W for ranking statements is given in Table 4. Finally, the inter-expert agreement computed through ICC is in the Table 5. The process of reaching consensus for rating and ranking statements are separately depicted in Section E in S2 Appendix, respectively. The statements over which consensus was not achieved after 2nd round are given in Section F in S2 Appendix.
Theme-1.
Theme-1 of the Delphi survey aimed to map the consensus of experts on the need, current situation, and potential disadvantages of lack of CP, GP collaboration and proposed CMTM model for chronic diseases. It had 18 statements, 13/18 statements, reached consensus in 1stround, while, 3 out of the 5 conflicted statements achieved consensus in 2nd round whereas 2 statements failed to reach consensus at this point. The ICC value of 0.62 (<0.001) showed good agreement among experts for both rounds for this theme.
Theme-2.
Theme-2 evaluated experts’ perspective on the potential roles and responsibilities of CP, if they were to provide a CMTM service in collaboration with GP. This theme involved 9 statements, out of which experts had consensus on 7 in 1st round. The 2 conflicted statements touched the pre-defined consensus level in 2nd round. The ICC result showed good agreement among experts for theme 2 with a value of 0.5 (<0.001) for both rounds.
Theme-3.
Theme-3 sought expert consensus on potential impact of CMTM service. The theme consisted of 16 statements, out of which 13 statements reached consensus in 1st round, while, 3 statements received consensus in 2nd round. The ICC values pointed to a poor agreement among experts for theme 3 with a value of 0.19 and 0.13 (<0.001) for 1st and 2nd round, respectively.
Theme-4.
Theme-4 involved 10 ranking statements pertaining to GPs’ perceived barriers on way to collaboration with CPs. The highest rank (most relevant barrier) was obtained by the statement ″CP-GP collaboration is a threat to GP’s job″, which gained a mean rank 3.03 at the end of 2nd round. As the scale used was set to mark highest rank at 1 and lowest rank at 10, any statement, with the lowest mean rank, would indicate the highest priority. The 2nd highest rank barrier was the statement ″CPs’ interventions will be projected as challenge for GP’s clinical decisions″ which receive a mean rank value of 4.07 at the end of 2nd round. Similarly, the third highest ranked barrier was related to concerns regarding the jurisdiction violation, which received a mean rank value of 4.69 at the end of 2nd round. However, the least ranked barrier with a mean rank value of 7.83 after round 2, was ″Malaysia does not have enough CPs to cater population healthcare needs″. The lowest priority or relevance to this barrier indicated that experts are fully aware of the current situation in terms of number of CPs in Malaysia and did not consider it as a barrier anymore. The ICC value of 0.87 and 0.86 (<0.001) denoted excellent agreement among experts for theme 4.
Theme-5.
Theme-5 also utilized ranking statements to identify the CP’s perceived barriers for a CMTM model in Malaysia. Lack of dispensing separation was the highest ranked barrier, which received a mean rank value of 1.76. The lowest ranked barrier by experts, from CPs’ view point was ″CPs are comfortable with their current roles″ which received the mean rank value of 6.55 after 2nd round and pointed that experts were fully aware of the awakening or realization in pharmacist community in Malaysia about their potential for extended roles. The ICC depicted excellent agreement among experts for this theme with a value of 0.98 (<0.001).
Theme-6.
Theme-6 highlighted consumer’s perceived barrier if there would be CP’s provided medicine management services in Malaysia. The highest ranked barrier from the consumer’s point of view was ″Consumers are still GP-centred″ with a mean rank of 1.93, while the lowest ranked barrier was ″Consumers will not trust CPs″ which received a mean rank of 3.34. This ranking by experts witnessed that public perception is not a top barrier in Malaysia. Excellent agreement among experts was witnessed by the ICC for this theme with a value of 0.90 (<0.001).
Theme-7.
This theme was characterized by 13 statements related to administrative or regulatory aspects of CMTM model. By the end of 2nd round 9/13 statements reached consensus level. The ICC test resulted in excellent agreement among experts for this theme with a value of 0.82 (<0.001).
Theme-8.
In theme-8, various chronic diseases (which could be benefitted by CMTM model) were enlisted. Consensus was achieved on3 diseases which were predicted to be benefitted most significantly by CP-GP collaboration as per Malaysian population health care needs, named as hypertension, asthma/ COPD and diabetes. The ICC value authenticated an excellent agreement among experts for this theme with a value of 0.96 (<0.001).
Theme-9.
Theme-9 asked experts to rank various chronic disease in order of priority for Malaysian healthcare system. Diabetes, hypertension and asthma were ranked as 1st, 2nd and 3rd priority respectively in Malaysia. The ICC value was in range of excellent agreement among experts for this theme with a value of 0.99 (<0.001).
Theme-10.
Theme-10 sought experts’ opinion on proposed solutions for various barriers and problem CMTM model may encounter during its course. Solutions for problem-1, 2, 3, 4 and 5 received consensus of the experts in the 1st round which did not change by the end of 2nd round.
For problem-6, 4 out of 5 solutions reached consensus at the end of study, however, solution ″burden of additional consultation fee for CMTM services may be minimized by Government subsidies″ could not reach consensus, which hinted that experts in Malaysia did not consider government subsidies as a viable financing options to compensate CP for the CMTM service.
Both solutions offered for problem-7 received experts’ consensus by the end of 2nd round. Problem-8, the solution ″collaboration between CP and GP can be achieved even without dispensing separation as it does not matter where a patient is getting medicines because at the end, he would be seeing a CP″ failed to reach consensus level, as only 37.93% of experts agreed to this. This is very important as it means majority of the experts were of view that collaboration between CP and GP, without dispensing separation would not make sense and would be functionally meaningless. This finding was quite important for this study. The ICC test depicted excellent agreement among experts for this theme with a value of 0.87 (<0.001).
Theme-11.
Theme-11 enlisted various means of financing the CMTM model and asked experts to rank the most applicable means of compensation or remuneration of CPs. UHC was the top ranked mean of financing CMTM model experts believe would be suitable for a mean of 2.24, third party payer was at the 2nd rank with a mean of 2.45 and ″direct billing″ was at the3rd rank with a mean of 3.34. However, the lowest rank was given to ″Incident to service″ with a mean of 6.76. The ICC test yielded excellent agreement among experts for this theme with a value of 0.98 (<0.001).
Discussion
This study deployed a modified Delphi method to present a consensus based recommendations to restructure the healthcare system at primary care level towards a more collaborative working model involving CP and GP for chronic disease management in Malaysia.
The study findings revealed significant recognition by the three group of stakeholders on the need for a CMTM model. Experts in Malaysia had complete realization of the current situation of medicine misadventures, lack of adherence, no prescription review system in private primary care especially for chronic diseases, and thus there existed high level of consensus on proper education and adherence support for chronic disease patients through an effective prescription review system in private setting and connecting all health professionals to one another through a national electronic record system where CP is viewed as reliable partner for medication management, documentation and follow up in chronic disease.
Experts also find priority areas of collaboration where need of collaboration is crucial and CMTM model may significantly benefit the Malaysian population i.e., hypertension, diabetes and asthma as top three chronic disease.
Generally, there was high level of consensus for most of the statements pertaining to structural, functional and regulatory aspects of CMTM. However, for the statements where consensus was not achieved among the experts (conflicted statements) there could be many reasons, for instance, the heterogeneity within the panel (we had quite a heterogenous panel) and nature of certain issues where high conflicts exist driven by either political rivalry or some financial conflict of interest. For such cases, expecting a high level of consensus (more than 80% agreement) would not be realistic. For example, study established that experts representing GPs hold high consensus level on the need of collaboration between CP and GP for managing chronic diseases therapy, but when next statements linked this collaboration with dispensing separation, they see it through a different lens. Thus, statement which linked collaboration with dispensing separation did not have high level of consensus but hold a moderate consensus. However, still more than 60% experts were of the view that collaboration without dispensing separation would not yield optimal results.
To authenticate the findings of this Delphi study, consensus was measured after confirmation of stability in the response of experts between two rounds. There were insignificant changes (Wilcoxon signed rank test results) in the responses of experts between 1st and 2nd round which means that any next round would not contribute significantly on the level of consensus. The time taken by the experts is a potential indicator of the diligence and interest of experts with which they filled the survey. The response rate in 2nd round (100%) warranted that attrition was not a limitation of this study.
Our findings are in line with the results of a study carried out in Malaysia to understand the GP’s perspective about the possible extended roles of CP in patient care [53]. This study concluded a favourable response from GPs on the extended roles of CPs in patient care. It also highlighted GP’s perceived barriers in collaboration and their doubts on the clinical skills of CPs. Our study took a step further and utilized a unique method to involve experts’ panel to first gauge the current level of consensus among different healthcare stakeholders in Malaysia for active involvement of CPs in medication therapy management of chronic diseases and then engaged experts to offer consensus based solutions to various GP’s perceived problems or barriers mentioned in previous research [53] which hamper an effective collaboration between CP and GP. For example, the apprehension of GPs on the clinical skills of CPs was mentioned as a barrier in collaboration in the mentioned study. To address this apprehension, expert panel offered a consensus based solution of clinical skills enhancing course for all CPs as an essential requirement for accreditation of CPs (who intend to offer CMTM services).
On the same node, another survey based research attempted to collect GP’s views on various possible roles a CP might offer in medicine management if there is an extension in roles of CPs in Malaysia [54]. The objective was to evaluate which roles of CP are viewed positively by the GPs. With a 73.4% response rate, more than 50% of GPs were positive for CP’s roles, such as, advice on medicine, patient education and counselling, referring patient to CP in case of any drug misadventure. However, GPs have issues with separation of dispensing and prescription and change in the therapy of medication. Our study looked the same problem through a different angle and collected diverse perspectives on different roles of CP in CMTM (patient care). However, the fundamental difference is the choice of sample. Our sample was a balanced mix of experts from GPs, CPs, and Nurses. Nonetheless, our study can add to the findings that the acceptance of CP’s different roles, has been increased at least in experts’ circles, however, dispensing separation remains a burning issue. Our study finds, GPs are positive for collaboration with CPs, however, 37% experts (purely from GPs side) still believe collaboration without dispensing separation would be the only acceptable options for GPs.
The results of our study are broadly comparable to a recent international Delphi study published with a focus to push chronic care forward through collaboration among healthcare professionals in Abu Dhabi [55]. The study objectives aimed to inform the UAE’s 2021 agenda to design a world class healthcare system by setting priorities and identifying barrier in delivery of optimal chronic care in Abu Dhabi through experts’ consensus. Like our results, they also identified lack of adherence support, proper patient education, monitoring, documentation and follow up through a centralized record system as the top barriers in continuity of chronic care. The top priority in chronic care was the reorganization of healthcare system to a more patient centred approach where every healthcare provider has the access to patient information through a centralized system (an electronic record). Similar was the case in United Arab Emirates where a modified Delphi study evaluated medicine management practices for the elderly and concluded on the need of an effective collaboration between pharmacist and GP [56]. These recent updates implied even in the developing countries like Abu Dhabi or United Arab Emirates, there is an awakening to respond to the burden of chronic disease through the concept of collaborative care.
Recommendations
This study offered potential strategies as way forward to address the problem of lack of collaboration between CP and GP in Malaysia. Based on study findings and experts’ consensus salient recommendations are being proposed for Ministry of health, Ministry of education and leadership of CPs and GPs in Malaysia.
For the Ministry of Health
- Ministry should seek measures to utilize CP’s potentials in delivering patient-centred care through CMTM service.
- The government should initiate CMTM for at least one of the top three priority chronic diseases i.e., hypertension, diabetes, and asthma/COPD where experts have consensus that CMTM model would significantly contribute.
- There should be a regulatory check on the collaborative practice and it must involve Ministry of health, Malaysian Pharmaceutical Society, Malaysian Medical association and other relevant stakeholders. The regulatory body would devise continuous professional development’s points, conduct exams and issue license to practitioner CPs or consultant pharmacist to work under protocols for a specific chronic disease.
- An electronic national prescription database system should be developed under the auspices of Ministry of health to store the prescription records of all the chronic disease patients.
- This electronic database should be made assessable to CP licensed to collaborate with GP for a specific chronic disease.
- Both CP and GP should be compensated for CMTM service. The most feasible way to finance or remunerate the CP and GP through this service is UHC or third-party payers. Thus, government should make it compulsory for its citizen a have a health insurance.
- There should be a pilot study to run CMTM model to test its potential advantages, and if proven favourable, it should be gradually implemented in big cities in Malaysia to avoid any setback.
- To promote public awareness about the importance of CMTM, government should run a national level campaign, to explain the advantages of CMTM model and advertise slogans, such as ″Know your medicine by asking your pharmacist″.
For community pharmacists’ leadership
- Community pharmacy needs to improve its infrastructure to offer such kind of collaborative services, for instance database system to store prescriptions’ record, such as computer, server, data storage software and internet.
- CPs should improve their clinical knowledge and skills for specific chronic diseases. Thus, before starting the service, CP should officially get a mandatory accredited training/diploma/course on CMTM service for a specific chronic disease (asthma, diabetes, hypertension).
- CP should improve their communication and documentation and must communicate all interventions to GP on a structured CP’s interventions form.
For General Practitioners’ leadership
- GPs should recognise the needs of the aging population and importance of inter-professional collaboration in patient care in Malaysia.
- GPs should gradually adopt to a collaborative model of chronic care. In Malaysia, CP and GP should work together in managing chronic disease(s).
- Collaboration would not be optimal without dispensing separation thus; dispensing separation should be executed at least in major cities to break the ice for this collaboration.
For the Ministry of Education
- Inter-professional education should be practically implemented in all medical and pharmacy colleges in Malaysia, so as the students understand each other role since start of their career.
Blue print of proposed CMTM model
The study aimed to draw a working sketch of a CMTM model, including, identifying the required resources, training, skills, accreditation, regulatory and infrastructure needs and prioritizing the common areas for collaborative practice through experts’ consensus. Based on the finding of this study we have proposed a theoretical framework for the CMTM model and is given in Fig 4.
MMA = Malaysian Medical Association, MoH = Ministry of Health, MPS = Malaysian Pharmaceutical Society, CMTM = Collaborative Medication Therapy Management, CP = community pharmacy, GP = general practitioner, UHC = Universal Health Coverage, CPD = Continuous Professional Development.
Conclusion
Overall, the study findings witnessed the expert panel’s support for the CMTM model for chronic care. It also explored and prioritized the issues in collaborative practice between CP and GP. The findings of the study helped to propose a working sketch of CMTM model and facilitated development of some recommendations to the authorities in Malaysia which may help to formulate a policy to bring CPs under a working relationship with GPs. Hence, this study should be taken as a call for redefining of the roles of CPs and GPs involved in primary care for chronic disease management. Undoubtedly CPs are an untapped national resource, which must be given the due role and there is acceptance of this perspective at least in the expert panel as it is the right time to share the care in Malaysia.
Supporting information
S1 Appendix.
This appendix includes section A to E containing database search flowchart, search strategy, examples of articles which laid foundation of survey, survey instrument validation and complete Delphi survey.
https://doi.org/10.1371/journal.pone.0216563.s001
(DOCX)
S2 Appendix.
This appendix includes section A to F containing tables regarding criteria of experts, qualitative comments, limitations, affiliations of experts and conflicted statements; and figures representing rating and ranking statements.
https://doi.org/10.1371/journal.pone.0216563.s002
(DOCX)
S1 File. This is a compressed file containing SPSS output files of median, IQR, Wilcoxon signed rank test and Kendall’s W test for both rounds.
https://doi.org/10.1371/journal.pone.0216563.s003
(ZIP)
S2 File. This is a compressed file containing SPSS output files of ICC test of both rounds (theme-wise).
https://doi.org/10.1371/journal.pone.0216563.s004
(ZIP)
Acknowledgments
We thank Professor Dr Khalid Ahmad, Punjab University, Lahore, Pakistan, for the advice on various statistical operations.
We are also immensely grateful to Prof. Chitu Okali, for continuous guidance on the Delphi survey.
References
- 1.
Jaafar S, Mohd Noh K, Othman NH, Healy J. Malaysia Health System Review. Vol. 3, Health System in Transition,. 2013.
- 2.
WHO. Non Communicable Diseases Progress Monitor 2017 [Internet]. 2017. Available from: http://apps.who.int/iris/bitstream/handle/10665/258940/9789241513029-eng.pdf;jsessionid=51F94AFBC0E219F486A7EDBCE26B5DF3?sequence=1%0Ahttp://apps.who.int/iris/bitstream/handle/10665/258940/9789241513029-eng.pdf?sequence=1
- 3.
MOH. National Health and Morbidity Survey 2015 (NHMS 2015). Vol. II: Non-Communicable Diseases, Risk Factors & Other Health Problems. Vol. II, Ministry of health. 2015. 1–291 p.
- 4.
Global age watch M. Ageing and Health, Malaysia. Vol. 2013. 2013.
- 5. Sebastian A, Alzain MA, Asweto CO, Mahara G, Guo X, Song M, et al. The Malaysian health care system: Ecology, plans, and reforms. Fam Med Community Heal [Internet]. 2016;4(3):19–29. Available from: http://www.ingentaconnect.com/content/10.15212/FMCH.2016.0101
- 6.
WHO County Profile Malaysia. World Health Organization—Noncommunicable Disease (NCD) Country Profile, 2014. World Heal Organ. 2014;1.
- 7.
MoH M. Health Facts 2017 (Reference data for 2016). Ministry of Health Malaysia, Health Informatics Centre Planning and Development Division. 2017.
- 8.
Mohamad Azmi H, Fahad S, Mohd Dziehan M, Che Pun B, Salmiah MA, Siti Fauziah A, et al. A National Survey on the Use of Medicines (NSUM) By Malaysian. 2016.
- 9. Chen LL, Tangiisuran B, Shafie AA, Hassali MAA. Evaluation of potentially inappropriate medications among older residents of Malaysian nursing homes. Int J Clin Pharm. 2012;34(4):596–603. pmid:22622593
- 10. Lim LM, McStea M, Chung WW, Azmi NN, Aziz SAA, Alwi S, et al. Prevalence, risk factors and health outcomes associated with polypharmacy among urban community-dwelling older adults in multiethnic Malaysia. PLoS One. 2017;12(3):1–18.
- 11. Salmasi S, Khan TM, Hong YH, Ming LC, Wong TW. Medication errors in the Southeast Asian countries: A systematic review. PLoS One. 2015;10(9):1–19.
- 12. Karuppannan M, Nee T, Thong W, Boardman H, Ali S. The prevalence of adverse drug event-related admissions at a local hospital in Malaysia. Arch Pharm Pract [Internet]. 2013;4(December):160. Available from: http://www.archivepp.com/text.asp?2013/4/4/160/123224
- 13. Ramli; ASWLP. Rejuvenating Chronic Disease Management in Malaysian Private General Practice–a Global Perspective. Malaysian Fam Physician. 2010;5(1):49–52.
- 14. Ramli A, Ahmad NS, Paraidathathu T. Medication adherence among hypertensive patients of primary health clinics in Malaysia. Patient Prefer Adherence. 2012;6:613–22. pmid:22969292
- 15. Sellappans R, Lai PSM, Ng CJ. Challenges faced by primary care physicians when prescribing for patients with chronic diseases in a teaching hospital in Malaysia: a qualitative study. BMJ Open. 2015 Aug;5(8):e007817. pmid:26316648
- 16. Dawood OT, Hassali MA, Saleem F. A qualitative study exploring medicines use pattern and practice among general public in Malaysia. Pharm Pract (Granada). 2016;14(2):1–7.
- 17. Ibrahim MIM. Antibiotics Dispensing for URTIs by Community Pharmacists and General Medical Practitioners in Penang, Malaysia: A Comparative Study using Simulated Patients. J Clin Diagnostic Res [Internet]. 2014;119–23. Available from: http://www.jcdr.net/article_fulltext.asp?issn=0973-709x&year=2014&volume=8&issue=1&page=119&issn=0973-709x&id=3923
- 18. Saw PS, Nissen LM, Freeman C, Wong PS, Mak V. Health care consumers’ perspectives on pharmacist integration into private general practitioner clinics in Malaysia: A qualitative study. Patient Prefer Adherence. 2015;9:467–77. pmid:25834411
- 19. Mohd-Tahir N-A, Paraidathathu T, Li S-C. Quality use of medicine in a developing economy: Measures to overcome challenges in the Malaysian healthcare system. SAGE Open Med [Internet]. 2015;3(February 2016):205031211559686. Available from: http://journals.sagepub.com/doi/10.1177/2050312115596864
- 20. Ting CY, Lee KS, Tan RT-H, Ang WC, Ming LC. An Exploratory Study on the Consumers Use of Medicine in the State of Sarawak, Malaysia. J Young Pharm [Internet]. 2016;8(4):477–82. Available from: http://www.jyoungpharm.org/article/911
- 21. Mossialos E, Courtin E, Naci H, Benrimoj S, Bouvy M, Farris K, et al. From “retailers” to health care providers: Transforming the role of community pharmacists in chronic disease management. Health Policy (New York) [Internet]. 2015;119(5):628–39. Available from: http://dx.doi.org/10.1016/j.healthpol.2015.02.007
- 22. Latif A, Pollock K, Boardman HF. The contribution of the Medicines Use Review (MUR) consultation to counseling practice in community pharmacies. Patient Educ Couns [Internet]. 2011;83(3):336–44. Available from: pmid:21621943
- 23. Malet-Larrea A, Garcia-Cardenas V, Saez-Benito L, Benrimoj S, Calvo B, Goyenechea E. Cost-effectiveness of professional pharmacy services in community pharmacy: a systematic review. Expert Rev Pharmacoeconomics Outcomes Res [Internet]. 2016;16(6):747–58. Available from: http://dx.doi.org/10.1080/14737167.2016.1259071
- 24. McMillan SS, Wheeler AJ, Sav A, King MA, Whitty JA, Kendall E, et al. Community pharmacy in Australia: A health hub destination of the future. Res Soc Adm Pharm [Internet]. 2013;9(6):863–75. Available from: http://dx.doi.org/10.1016/j.sapharm.2012.11.003
- 25. Messerli M, Blozik E, Vriends N, Hersberger KE. Impact of a community pharmacist-led medication review on medicines use in patients on polypharmacy—a prospective randomised controlled trial. BMC Health Serv Res [Internet]. 2016;16:145. Available from: pmid:27108410
- 26. Franco-Trigo L, Hossain LN, Durks D, Fam D, Inglis SC, Benrimoj SI, et al. Stakeholder analysis for the development of a community pharmacy service aimed at preventing cardiovascular disease. Res Soc Adm Pharm [Internet]. 2016; Available from: http://dx.doi.org/10.1016/j.sapharm.2016.06.009
- 27. Geist MR. Using the Delphi method to engage stakeholders: A comparison of two studies. Eval Program Plann. 2010;33(2):147–54. pmid:19581002
- 28. Landeta J. Current validity of the Delphi method in social sciences. Technol Forecast Soc Change. 2006;73(5):467–82.
- 29. Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol [Internet]. 2014;67(4):401–9. Available from: pmid:24581294
- 30. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: A systematic review. PLoS One. 2011;6(6).
- 31. Fletcher AJ, Marchildon GP. Using the delphi method for qualitative, participatory action research in health leadership. Int J Qual Methods. 2014;13(1):1–18.
- 32. Avella JR. Delphi panels: Research design, procedures, advantages, and challenges [Internet]. Vol. 11, International Journal of Doctoral Studies. 2016 [cited 2019 Jan 7]. Available from: http://www.informingscience.org/Publications/3561
- 33. Grisham T. The Delphi technique: a method for testing complex and multifaceted topics. Int J Manag Proj Bus. 2009;2(1):112–30.
- 34. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm. 2016;38(3):655–62. pmid:26846316
- 35. Okoli C, Pawlowski SD. The Delphi method as a research tool: An example, design considerations and applications. Inf Manag. 2004;42(1):15–29.
- 36. Trevelyan EG, Robinson N. Delphi methodology in health research: How to do it? Eur J Integr Med [Internet]. 2015;7(4):423–8. Available from: http://dx.doi.org/10.1016/j.eujim.2015.07.002
- 37. Robinson P, Oades LG, Caputi P. Conceptualising and measuring mental fitness: A Delphi study. Int J Wellbeing [Internet]. 2015;5(1):53–73. Available from: http://www.internationaljournalofwellbeing.org/index.php/ijow/article/view/256/443
- 38. Iqbal S, Pipon-Young L. The Delphi Method. Br Psychol Soc. 2009;22(7):598–600.
- 39. Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: Application of bootstrap data expansion. BMC Med Res Methodol. 2005;5:1–12. pmid:15636638
- 40. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs [Internet]. 2000;32(4):1008–15. Available from: http://doi.wiley.com/10.1046/j.1365-2648.2000.t01-1-01567.x pmid:11095242
- 41. Warner LA. Using the Delphi Technique to Achieve Consensus: A Tool for Guiding Extension Programs 1. Agric Educ Commun. 2017;1–5.
- 42. Polit DF, Beck CT. The Content Validity Index: Are You Sure You Know What’s Being Reported? Critique and Recommendations. Res Nurs Health [Internet]. 2006 [cited 2018 Dec 17];29:489–97. Available from: www.interscience.wiley.com pmid:16977646
- 43. Li Y, Ehiri J, Hu D, Zhang Y, Wang Q, Zhang S, et al. Framework of behavioral indicators for outcome evaluation of TB health promotion: A Delphi study of TB suspects and Tb patients. BMC Infect Dis. 2014;14(1):1–14.
- 44. Sandrey M. A. & Bulger SM. The Delphi method: an approach for facilitating evidence based practice in athletic training. Athl Train Educ J [Internet]. 2008;3(4):135–142. Available from: www.nataej.irg.
- 45.
Keenyey S, Hasson F, McKenna H. The Delphi Technique in Nursing and Health Research. A John Wiley & Sons, Ltd., Publication; 2011. 1–210 p.
- 46. Ross S, Metcalf A, Bulger SM, Housner LD. Modified delphi investigation of motor development and learning in physical education teacher education. Res Q Exerc Sport [Internet]. 2014;85(3):316–29. Available from: pmid:25141085
- 47. von der Gracht HA. Consensus measurement in Delphi studies. Review and implications for future quality assurance. Technol Forecast Soc Change. 2012;79(8):1525–36.
- 48. Shariff NJ. Utilizing the Delphi Survey Approach: A Review. J Nurs Care [Internet]. 2015;04(03). Available from: https://www.omicsgroup.org/journals/utilizing-the-delphi-survey-approach-a-review—2167-1168-1000246.php?aid=51677
- 49. Thangaratinam S, Redman CW. The Delphi technique. Obstet Gynaecol [Internet]. 2005;7(2):120–5. Available from: http://doi.wiley.com/10.1576/toag.7.2.120.27071
- 50. Habibi A, Sarafrazi A, Izadyar S. Delphi technique theoretical framework in qualitative research. Int J Eng Sci [Internet]. 2014;3(4):8–13. Available from: http://www.academia.edu/download/36610746/B03404008013.pdf
- 51. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med [Internet]. 2016;15(2):155–63. Available from: pmid:27330520
- 52. Day J, Bobeva M. A generic tookit for the successful management of Delphi studies. Electron J Bus Res Methodol. 2005;3(2):103–16.
- 53. A, Hassali M AA, A, Shafie A, Saeed M. PROFESSIONAL TRAINING AND ROLES OF COMMUNITY PHARMACISTS IN MALAYSIA: VIEWS FROM GENERAL MEDICAL PRACTITIONERS. Malaysian Fam Physician. 2009;4(2):71–6.
- 54. Sarriff A, Nordin N, Ahmad Hassali MA. Extending the Roles of Community Pharmacists: Views from General Medical Practitioners. Med J Malaysia. 2012;67(6):577–81. pmid:23770948
- 55. Paulo MS, Loney T, Lapao L V. A pushing chronic care forward in Abu Dhabi by identifying priorities and addressing barriers: a modified Delphi technique. BMJ Open [Internet]. 2018 Jun 4 [cited 2018 Dec 17];8(6):e020189. Available from: pmid:29866724
- 56. Al Shemeili S, Klein S, Strath A, Fares S, Stewart D. A modified Delphi study of structures and processes related to medicines management for elderly hospitalised patients in the United Arab Emirates. J Eval Clin Pract. 2016;22(5):781–91. pmid:27060885