Figures
Abstract
Background
Currently, zoonoses account for 58% to 61% of all communicable diseases causing illness in humans globally and up to 75% of emerging human pathogens. Although the impact of zoonoses on animal health and public health in North America is significant, there has been no published research involving health professionals on the prioritization of zoonoses in this region.
Methodology/Principal Findings
We used conjoint analysis (CA), a well-established quantitative method in market research, to identify the relative importance of 21 key characteristics of zoonotic diseases for their prioritization in Canada and the US. Relative importance weights from the CA were used to develop a point-scoring system to derive a recommended list of zoonoses for prioritization in Canada and the US. Study participants with a background in epidemiology, public health, medical sciences, veterinary sciences and infectious disease research were recruited to complete the online survey (707 from Canada and 764 from the US). Hierarchical Bayes models were fitted to the survey data to derive CA-weighted scores for disease criteria. Scores were applied to 62 zoonotic diseases to rank diseases in order of priority.
Conclusions/Significance
We present the first zoonoses prioritization exercise involving health professionals in North America. Our previous study indicated individuals with no prior knowledge in infectious diseases were capable of producing meaningful results with acceptable model fits (79.4%). This study suggests health professionals with some knowledge in infectious diseases were capable of producing meaningful results with better-fitted models than the general public (83.7% and 84.2%). Despite more similarities in demographics and model fit between the combined public and combined professional groups, there was more uniformity across priority lists between the Canadian public and Canadian professionals and between the US public and US professionals. Our study suggests that CA can be used as a potential tool for the prioritization of zoonoses.
Citation: Ng V, Sargeant JM (2013) A Quantitative Approach to the Prioritization of Zoonotic Diseases in North America: A Health Professionals’ Perspective. PLoS ONE 8(8): e72172. https://doi.org/10.1371/journal.pone.0072172
Editor: Niko Speybroeck, Université Catholique de Louvain, Belgium
Received: October 22, 2012; Accepted: July 11, 2013; Published: August 21, 2013
Copyright: © 2013 Ng, Sargeant. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: Funding was provided from a Canadian Institute of Health Research (CIHR) Institute of Population and Public Health/Public Health Agency of Canada Applied Public Health Research Chair awarded to JMS. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Zoonotic diseases are diseases that are naturally transmitted between vertebrate animals and humans [1], [2]. Currently, it is estimated that zoonoses account for 58% to 61% of all communicable diseases causing illness in humans globally [3], [4] and up to 75% of emerging human pathogens [3]–[5]. Although zoonotic outbreaks are a significant burden of disease in North America, for example, outbreaks of West Nile virus, SARS, H1N1 influenza and Lyme disease in the past decade [6]–[9], there are limited resources available for their control and prevention making it necessary to prioritize diseases in order to allocate resources to those with the greatest impact. While there is consensus on the need to prioritize zoonoses, there are numerous challenges to the process. These include, the difficulty in comparing the overall public health impact of zoonoses when they vary greatly in incidence, clinical manifestations, control measures, transmission potential and socio-economic impact in humans and animals [10], [11]; the multiple stakeholders involved each with their own prioritization objectives and beliefs [12]; and the lack of agreement in prioritization methodologies [13]–[15]. These challenges limit the ability to establish a universally accepted priority list for zoonoses. Nonetheless, a number of studies have attempted to prioritize communicable diseases [13], [16]–[21] and more recently, zoonotic diseases [22]–[26], though the studies on zoonotic diseases have all been conducted in Europe.
Progress has been made towards the use of quantitative approaches to address the complexity of disease prioritization and to overcome constraints in traditional methods [13], [14], [21], [24], [25]. Although methodological approaches differ, prioritization typically follows a series of steps: (1) selecting a group of diseases/pathogens to prioritize; (2) identifying a list of measurable criteria to assess diseases/pathogens; (3) defining a range of levels for each criterion; (4) determining the relative importance for each level by assigning a weight or score; (5) assigning weights and/or scores by matching the level of each disease criterion to the select group of diseases/pathogens; (6) aggregating weights/scores to produce an overall score for each disease/pathogen; and (7) ranking diseases/pathogens by their overall score to derive a priority list. Current prioritization methods, including recent quantitative approaches, are limited by the requirement to produce arbitrary scores and subjective weights for disease criteria and their levels (step 4), these are typically derived from a simplified linear point-scoring system without weights [16]–[20] or by an expert Delphi panel who assign weights that are applied to a linear point-scoring system [13], [14], [21], [24], [26]. This self-explicated approach to deriving scores and weights can introduce subjective bias into the prioritization exercise. Further, the matrix approach to assigning scores and weights to each disease criterion separately makes the assumption that disease criteria are independent.
A novel quantitative approach to overcoming these specific limitations is Conjoint Analysis (CA). CA is a market research technique used in exploring consumer preferences [27]. It is gaining recognition in the last decade for its use in eliciting preferences in the healthcare setting [28]–[31]. The principle behind CA is that a product (goods or service) can be described by a set of characteristics and the extent to which an individual values a product is determined by the level of each of those characteristics and the combination of those characteristics together [28], [29], [32]. A CA study presents individuals with competing products containing both desirable and undesirable characteristics and forces the individual to state a preference, usually as a choice between products. In doing so, individuals make a trade-off between the desirable and undesirable characteristics in the products through their choices revealing the true value of each characteristic relative to each other. This preference elicitation method overcomes the need to assign arbitrary scores and subjective weights as relative weighted scores for each characteristic and their levels are derived from the choice data. Additionally, this approach forces individuals to consider multiple characteristics together; thus, criteria are not assumed to be independent from each other.
In the context of zoonoses, a disease can be treated as a product described by a set of disease criteria (characteristics), and the value of the disease can be determined by the level of each criteria and the combination of those criteria. While similar methods to CA such as Maximum Difference Scaling (MaxDiff) are available [32], zoonotic diseases are often complex requiring an understanding of preferences under a range of multiple characteristics and levels, thus the use of CA, that allows for the exploration of inter-relationships across numerous characteristics and levels, was considered a more appropriate tool. An additional benefit in using CA in disease prioritization is that by presenting zoonoses as a set of disease characteristics without identifying diseases, individuals are forced to prioritize based on science, eliminating potential biases associated with disease names. These can include biases arising from prioritizing diseases on the basis of professional awareness and/or personal gain, or from the potential fear of a disease name compared to a lesser-known disease. Finally, by presenting respondents with all the information to prioritize diseases, CA allows for wide social participation; this can include experts who may not be familiar with the full range of diseases. CA is similar to the methodology used in a recent study on the prioritization of emerging zoonoses in The Netherlands [25], although a different mathematical approach was used.
Previously, we presented on the novel use of CA to develop a point scoring system for disease criteria considered important in determining priority amongst individuals from the general public [33]. The current study describes the results of individuals identified as ‘health professionals’ - individuals with a background in medical and veterinary sciences, public health, epidemiology and infectious diseases. This paper will also compare the results between the public groups and the professional groups. The primary objective of this study is to present on the methodological approach used to prioritize zoonoses in North America, a secondary objective is to define the most important zoonoses in North America as identified using this approach.
Materials and Methods
Study Participants
The Research Ethics Board at the University of Guelph approved all aspects of this study. The target study participants were: epidemiologists, public health practitioners and policymakers in the human and animal health disciplines at the local, provincial/state and national level, academic and practicing physicians and veterinarians, infectious disease researchers, human and animal health laboratory microbiologists, pathologists and technicians and registered nurses. Thus, the study group included individuals in relevant professional disciplines that would provide some prior knowledge of infectious diseases.
Participants from both Canada and the US were recruited through email invitation. Email searches were conducted for academic and practicing physicians and veterinarians and public health representatives and policymakers at the local, provincial/state and federal level. Direct email invitations were endorsed and sent by professional associations. Web advertisements were placed in relevant publications and some individuals were recruited in person at provincial and national conferences (see S1 for full listing). An additional 125 Canadian physicians were recruited online using a healthcare panel through Research Now™ [34]; these were groups of pre-screened Canadian physicians who had expressed a willingness to participate in online surveys. Reminder emails were sent a week after the initial email invitation and three weeks after the initial email invitation. Recruitment commenced in November 2010 and was completed in January 2012. Sample size calculations were made using Sawtooth Software SSI Web v7 [35]; a minimum of 500 professionals per country was needed for this study.
Surveys were collected online and anonymously. All participants acknowledged an informed consent assuring confidentiality and the option to withdraw from participation without penalty. Sawtooth Software SSI Web v7 [35] was used to screen participants through a series of demographic questions prior to survey commencement. Participants were disqualified if they did not reside in North America or were not employed in one of the following fields: epidemiology, public health, medical sciences, veterinary sciences, infectious disease research, laboratory technician or nursing.
Survey Development
The methods for criteria identification, disease selection, literature review, defining levels for disease criteria, survey development and administration were described previously [33]. Briefly, six focus groups were conducted using the nominal group technique to identify the disease criteria in the study [12]. A total of 21 criteria were selected to inform the CA experimental design. Each of these selected criteria could be quantitatively measured with scientific data in the literature [33]. A total of 62 existing and emerging zoonotic and enteric diseases was selected for the study on the basis of being nationally or internationally notifiable or identified as a priority [33]. Diseases exhibiting multiple forms (for example, acute/chronic, latent/active) were divided into separate syndromes and approximate proportions were assigned to each syndrome as informed by the literature. There were 117 separate disease syndromes identified from the 62 diseases. A literature search for each criterion for each disease syndrome was conducted, searches included websites of reputable human and animal health organizations, reference textbooks and PubMed catalogued peer-reviewed publications [33]. Criterion levels were defined according to the range exhibited in the literature with three or four levels assigned to each criterion.
Survey Instrument
Due to the large number of criteria, a partial-profile choice-based conjoint (CBC) survey was developed comprising 14 choice tasks [36], [37]; each choice task presented participants with five disease combinations containing varying levels of 5 of the 21 criteria using an orthogonal experimental design [33]. Participants were asked to select one zoonosis to prioritize for their control and prevention in either Canada or the US (Figure 1). Definitions for the technical terms ‘case-fatality’ (proportion of deaths over the total number of cases) and ‘disease incidence’ (number of new cases of diseases over the last five years in the population) were provided to ensure study participants understood the presented disease criteria. Disease criteria and levels varied between choice tasks and the ordering of the presentation of disease criteria within each choice task was randomized to reduce ordering bias. Two additional fixed choice tasks were included to test the reliability of responses by identifying respondents who did not understand the choice task process and/or fatigue responders. Fixed choice task 1 presented one zoonosis with the highest incidence in humans (10,000 cases), most severe illness in humans (severe clinical symptoms), highest transmission potential between humans (high), highest case-fatality in humans (80%) and the most costly economic burden in humans ($10,000 per sick individual). In comparison, the remaining four zoonoses contained a combination of lower and less severe criteria levels. Fixed choice task 2 presented one zoonosis with the most severe illness in animals (severe clinical symptoms), highest case-fatality in animals (80%), most costly socioeconomic burden in trade in animals (high cost such as culling of herds or destroying infected crops/produce), longest duration of illness in animals (chronic illness or permanent deficits) and rapid change in disease trend in the human population (new emerging disease, rapid increase over the last five years). In comparison, the remaining four zoonoses contained a combination of lower and less severe criteria levels. The fixed choice tasks were also randomized to reduce ordering bias and as an additional measure of reliability. Sawtooth Software CBC module v7 [38] was used to create 300 survey versions, each version presented an efficient experiment design using a balance overlap approach with balanced levels across choice tasks and an orthogonal design [39]. The D-efficiency of the experimental design was 908.13326 relative to a full-orthogonal design with a standard error of <0.05 for each criterion level. Sawtooth Software SSI Web v7 [35] was used to randomly assign a survey version to each study participant. Surveys were offered in English, French and Spanish.
As multiple survey versions were administered randomly to each person, a different combination of disease criteria and levels was presented to study participants. The ordering of the presentation of disease criteria within each choice task was randomized to reduce ordering bias. (*Canadian participants were asked to prioritize for Canada while US participants were asked to prioritize for the US.).
Data Analysis
Hierarchical Bayes (HB) was used to compute parameter estimates (weighted scores) from the CA survey choice data [40]. Sawtooth Software CBC/HB v5.2.8 [2] was used to estimate individual-level parameter estimates (β). The program combined Bayes theorem with a Monte Carlo Markov Chain (MCMC) procedure and the Metropolis/Hasting algorithm to iteratively update parameter estimates drawn from an upper-level model (prior) and a lower-level model (posterior) [40]. The HB algorithm estimates the average parameter estimates for the entire population (prior) and then uses the respondent’s individual data (posterior) to determine how each respondent differed from the population mean. The algorithm will then adjust each respondent’s parameter estimates so that they reflect an optimal mix of the individual respondent choices and the population mean. The optimal mix is determined by the amount of data provided by each respondent (posterior) and the amount of variance in the population mean (prior). The greater the prior variance (the distribution or differences amongst the population), the less Bayesian shrinkage is applied to the mean and the more individuals are allowed to vary such that their parameter estimates provide better individual-level fit to the individual-level responses) [41]. A total of 30,000 preliminary iterations were computed before convergence was observed (graphically and on observing stable goodness-of-fit measures including root-likelihood and variance) and an additional 30,000 iterations were computed per respondent to estimate final parameters. Final parameter estimates (β) are presented as zero-centered standardized utility values by setting the average range of the parameter values of all disease criteria to 100.
Part-worth utility values (β) represent the relative influence each criterion level had on respondent choices with higher values indicating a stronger influence on choice [32]. As part-worth utility values were calculated from choice task sets presenting disease combinations containing varying levels of multiple disease criteria, interaction effects between disease criteria and levels were accounted for in the calculation of part-worth utility values. Twenty-one part-worth utility values, one for each disease criterion, were assigned to the 117 separate disease syndromes by matching the level of each disease criterion to those of disease syndromes [33]. Part-worth utility values were summed up in proportion to the relative frequency of each syndrome within a disease to derive an overall score for each of the 62 diseases. The overall scores were used to rank-order diseases; the higher the score, the higher the ranking on the priority list. As part-worth utility values are interval data, the overall scores cannot be directly compared both within and between countries [32]. Instead, we compare the difference in disease ranking as an overall measure of proximity of diseases both within and between countries. The summed part-worth values approach was used instead of market simulations to apply CA-derived scores to a set of diseases to explore a method comparable to current traditional prioritization methods [14], [19]–[21], [24]–[26].
To estimate the influence of each criterion collectively, importance scores were calculated, for each respondent, as a percentage by dividing the difference in range between the highest and lowest level part-worth utility values by the sum of all part-worth utility value ranges across all criteria, and then averaging those importance scores across respondents. The larger the difference between the levels in a criterion, the higher the importance score and the stronger the influence the criterion had on the decision to prioritize [32]. Individual-level part-worth utility values and importance scores were calculated directly using Sawtooth Software SMRT v4.22.0 [42]. The standard error across individual part-worth utility values was used to quantify uncertainty in the part-worth utility values. T-statistics were derived by dividing the mean difference in range in part-worth utility values across each criterion by the standard error of the differences to test each disease criterion for statistical significance in the final model. Overall scores for diseases were calculated as the summation of the mean (population-level) part-worth utility values across the 21 disease criteria. As the overall scores were derived from population-level utility values, uncertainty in the overall scores were quantified by incorporating the standard error across individual part-worth utility values and calculated as the summation of the upper and lower confidence interval part-worth utility values, respectively, across the 21 disease criteria. As part-worth utility values are interval data, we compare changes in disease ranks rather than changes in the overall scores to assess uncertainty.
We used Sawtooth Software CBC/HB v5.2.8 [2] to generate goodness-of-fit measures for the individual-level HB models; these included a percent certainty fit and a root likelihood (RLH). Both of these measures are calculated from the probability of each respondent choosing as they did on each choice task using a logit model estimated with each respondent’s part-worth utility values [40]. The percent certainty and RLH both reflect how well the current model fit is in comparison to a chance model and a perfect model.
Chi Square and Fisher’s exact tests were used to compare the demographic, professional background and survey characteristics of study participants to their respective national populations and to make comparisons by country. The Mann-Whitney test was used to explore differences in completion time by country. National population data for gender, age, region and education were obtained from Statistics Canada [43], [44] and the US Census Bureau [45], [46]. Unpaired t-tests, F-tests and Welch’s t-tests were used to explore differences in standardized importance scores and part-worth utility values between Canada and the US. We used t-tests to compare differences in importance scores between the public groups presented in our previous paper [33] and the professional groups presented in this paper, and ANOVA to compare differences between the four groups (Canadian public, Canadian professionals, US public and US professionals). Spearman rank correlation was used to compare disease priority ranks between groups. Friedman’s ANOVA was used to assess uncertainty in the final models on disease priority ranks.
Results
Survey and Demographic Characteristics
The response rate for survey completion could not be calculated because the total number of respondents approached at conferences and through advertisement in publications could not be ascertained. Approximately 12,076 Canadians and 33,992 Americans were recruited via direct email invitation, however, as individuals may have been affiliated with one or more professional associations, duplicate posting likely occurred and the exact number of participants approached is unknown. As surveys were collected anonymously, we could not determine survey responses by recruitment mode.
A total of 928 Canadian and 998 US surveys were completed in 13 months and 3 months, respectively, of these, 62 Canadian and 55 US surveys were removed due to not fulfilling the eligibility requirements of being human health or animal health professionals. The majority of Canadian surveys were completed in English (95.5%) while the remaining surveys were completed in French (4.5%). All US surveys were completed in English despite an option to complete the survey in Spanish. Participants passed the survey if all 14 choice task sets were completed and the correct diseases were selected for both fixed choice task set. The fixed choice task questions were designed to present an obvious “best answer” to serve as a screening tool to identify participants who did not understand the choice task process. The Canadian pass rate was 81.6% (707) while the US pass rate was 81.0% (764); there was no significant difference in the pass rate between countries (χ2 = 0.1148, p = 0.735), nor was there significant difference in the pass rate between Canadian surveys completed in English or French (χ2 = 0.1262, p = 0.722). Although the fixed choice task sets were introduced to eliminate bias from unreliable respondents, these results suggest that respondents were no more likely to pass or fail (and thus introduce bias) from a particular country or language of completion. The median completion time for passed surveys was 26.9 minutes in Canada and 28.1 minutes in the US; there was no significant difference in the completion time between countries (p = 0.515). There were 1,471 completed and passed surveys available for analysis in this study.
There were minor differences between the study populations and their respective national populations by gender, age and geography (Table 1). The most notable difference was a higher educated population in both study populations compared to their respective national populations (p<0.001 for both countries). This was expected as the study populations were recruited for their professional background rather than to obtain a representative sample of the national populations. Differences were also observed between the Canadian and US study populations by gender and education (Table 2); there were more female study participants in the US than in Canada (61.2% vs. 54.0%), more high school graduates or less (3.5% vs. 0.6%), Bachelor’s degrees (14.7% vs. 13.5%), Master’s degrees (18.6% vs. 13.6%) and Doctorate degrees (22.7% vs. 20.1%) in the US than in Canada, and more Diplomas, trade or college degrees (3.6% vs. 0.5%) and Professional degrees (48.6% vs. 40.0%) in Canada than in the US.
The Canadian and US study populations differed significantly in professional disciplines, individuals identifying as human health or animal health professionals and by workplace of employment (Table 3). The Canadian study population comprised of more physicians and professionals in the medical sciences (19.9% vs. 10.1%), veterinarians and professionals in the veterinary sciences (34.1% vs. 29.2%) and animal health laboratory technicians (2.0% vs. 0.8%) while the US study population comprised of more epidemiologists (16.0% vs. 10.5%), public health professionals (20.9% vs. 16.4%), infectious disease researchers (8.9% vs. 5.7%), human disease laboratory technicians (1.8% vs. 0.7%) and nurses (4.7% vs. 2.5%). There were more individuals who self-identified as animal health professionals in the Canadian study population, conversely, more individuals self-identified as human health professionals or both human and animal health professionals in the US study population. The Canadian study population included a higher number of professionals working for the government or at hospitals/clinics while the US study population included a higher number of professionals working in academia and from industry. Despite differences in professional background characteristics between Canada and the US, the study populations each reflected a good representation of professional disciplines, animal and human health professionals and workplace of employment within country (Table 3). Consistent between countries was the years in employment with the majority of study participants indicating over 10 years of work experience in their profession (62.5% in Canada, 62.8% in the US).
Model Fit
The Canadian model had a percent certainty fit of 83.7% and a root likelihood (RLH) of 0.77, the US model had a percent certainty fit of 84.2% and a root likelihood (RLH) of 0.78. The expected percent certainty for a chance model is 0% and a perfect model is 100% while the expected RLH for a chance model is 0.2 (one divided by five disease combinations per task) and a perfect model is 1.0 [40]. While the models in this study do not represent a perfect model, the models are certainly above satisfactory producing robust part-worth utility values.
Disease Criteria Importance Scores and Part-worth Utility Values
The importance scores for disease criteria indicate the degree to which each criterion contributed to the decision to prioritize (Table 4). Human-related criteria contributed more to the decision to prioritize than corresponding animal-related criteria with each of the eight matching criteria exhibiting this trend in both countries. The four transmission potential criteria contributed in the following order of preference in both countries: animal-human, human-human, animal-animal and human-animal; thus also revealing a stronger preference for human-related criteria over animal-related criteria. While the contribution of each disease criterion in the decision to prioritize differed with varying degree of importance, ranging from 1.32% to 9.06% (Table 4), each criterion as a whole was statistically significant (p<0.05) in the final model for both countries (Table 5).
Although differences were observed between countries, both groups considered incidence of the disease in the last five years in humans and case-fatality in humans to be the most influential criteria in the decision to prioritize zoonoses (Table 4). Similarly, both groups considered high-risk groups in animals to be the least influential criteria in the decision to prioritize zoonoses. An additional eight disease criteria were ranked equally between the countries (transmission potential from animals to humans, case-fatality in animals, economic and social burden on trade in animals, transmission potential between humans, transmission potential between animals, efficacy of control measures in humans, severity of disease in animals and duration of illness in animals) while the remaining criteria differed only by a maximum of two ranked positions (severity of the disease in humans, disease trend in the last five years in humans, disease incidence in the last five years in animals, economic burden in humans, duration of illness in humans, disease trend in the last five years in animals, transmission potential from humans to animals, efficacy of control measures in animals, how much is known scientifically about the disease and high risk groups in humans) indicating a general consensus between the two countries on the contribution of the 21 disease criteria in the decision to prioritize zoonoses.
The part-worth utility values (β) indicate the relative influence each level had on respondent choices with higher values representing a stronger degree of influence on choice. The mean part-worth utility values and the upper and lower uncertainty estimates of mean part-worth utilities values are presented in Table 5. The wider the range in part-worth utility values between the lowest levels and highest levels within each criterion, the more influence the criterion had on the decision to prioritize. There were nine disease criteria in which the importance score differed significantly between countries (Table 4); the utility trends for these criteria can be broadly summarized as follows:
Canadian professionals were more strongly influenced by severity of disease in humans, economic burden in humans, duration of illness in humans, case-fatality in animals and high-risk groups in animals (p<0.0024 for all, Table 4). US professionals were more strongly influenced by incidence of the disease in the last five years in humans, incidence of the disease in the last five years in animals, disease trend in the last five years and economic and social burden on trade in animals (p<0.0024 for all, Table 4). Despite the differences in importance scores, there was agreement between countries on the levels of least importance (lowest part-worth utility values) and levels of highest importance (highest part-worth utility values) for each of these nine disease criteria, with incremental increases in the part-worth utility values for the levels in between (Table 5). While the strength of preference in disease criteria importance scores and part-worth utility values differed between countries (Tables 4 and 5), there was general agreement in the contribution of disease criteria in the decision to prioritize zoonoses. The difference may be due to a difference in the perceived threat of disease and disease characteristics by country.
Disease Priority Lists
Table 6 presents the final ranking of diseases derived from their overall CA scores. The range in the overall scores by diseases differed between Canada and the US and correlates with the part-worth utility values derived from the country-specific models (Table 5). Canadians considered rabies to be the most important zoonoses to prioritize, followed by Nipah virus encephalitis, H1N1 influenza, variant Creutzfeldt-Jakob disease and listeriosis. These were also the top five priority diseases in the US, ranked in different order. There was also consensus between the bottom five diseases on the priority list with three of the five least important diseases appearing in both priority lists. Although differences were observed in disease rankings between countries, the majority of diseases (76%) were within ten ranked positions of each other indicating general agreement in disease ranks between countries (Spearman’s rho = 0.8356, p = 0.000).
As we included a broad group of zoonotic diseases, many of these diseases may not be relevant for specific stakeholder groups, for example, there may be groups only interested in vector-borne diseases, food-borne and enteric diseases, exotic diseases, endemic diseases or diseases affecting certain commodity groups. The priority list can be further broken down into these sub-groups and the priority diseases by subgroup include: vector-borne diseases (leishmaniasis, Chagas disease and the plague in Canada; babesiosis, anaplasmosis and the plague in the US), food-borne and enteric diseases (variant Creutzfeldt-Jakob disease, listeriosis and botulism in Canada; variant Creutzfeldt-Jakob disease, listeriosis and paralytic shellfish poisoning in the US), exotic diseases (Nipah virus encephalitis, Ebola virus haemorrhagic fever and Marburg haemorrhagic fever for both Canada and the US) and endemic diseases (rabies, H1N1 influenza and variant Creutzfeldt-Jakob disease in Canada; variant Creutzfeldt-Jakob disease, rabies and H1N1 influenza in the US) (Table 6).
Diseases of high priority exhibited high incidence (H/A), high case-fatality (H/A), severe symptoms in humans, an increasing or emerging trend (H/A), high socioeconomic burden (H/A), prolonged duration of illness in humans and high transmission potential from animals to humans. However, it was not necessary to exhibit each of these characteristics to be identified as a priority (for example, H1N1 influenza and babesiosis both have low case-fatality rates while Nipah virus encephalitis, Ebola virus hemorrhagic fever and Marburg hemorrhagic fever do not occur naturally in North America). Diseases of low priority generally included rare diseases or diseases with a large proportion of asymptomatic cases (H/A), low case-fatality (H/A), mild symptoms in humans, stable or decreasing trend (H/A), low socioeconomic burden (H/A), short duration of illness in humans and low transmission potential from animals to humans.
Canadians considered leishmaniasis, Hendra virus, salmonellosis, cryptosporidiosis and Crimean-Congo hemorrhagic fever of higher priority than Americans (difference of 12 ranked positions or more, Table 6). Conversely, Americans considered babesiosis, anaplasmosis, paralytic shellfish poisoning, coccidioidomycosis, cysticercosis and Rocky Mountain spotted fever of higher priority than Canadians (difference of 12 ranked positions or more, Table 6). This can be explained by regional differences in both human and animal disease incidence; for example, babesiosis does not occur naturally in Canada but is endemic in the US [47], anaplasmosis is extremely rare in Canada but endemic in the US [48] and Rocky Mountain spotted fever is found only in Western Canada but is distributed throughout the US [49]. Regional differences in disease trend also likely contributed to different rankings, for example, paralytic shellfish poisoning has been increasing in the US [50] but is stable in Canada. Other differences can be explained by differences in part-worth utility values in disease criteria by country (Table 5).
To test the uncertainty in the part-worth utility values derived from the statistical models, we applied the 95% lower and upper confidence intervals of the mean part-worth utility values (Table 5) to compare the ranking of diseases. There was no statistical differences in the ranking of the diseases when uncertainty estimates were applied with 58 of 62 disease ranks unchanged and 4 diseases changing by one rank in Canada (Friedman’s χ2 = 0.000, p = 1.000) and 53 of 62 disease ranks unchanged, 8 diseases changing by one rank and one disease changing by two ranks in the US (Friedman’s χ2 = 0.008, p = 0.996) (S2). There were also no statistical differences in the ranking of the diseases when uncertainty estimates were applied to cluster of similar diseases (vector-borne diseases, food-borne and enteric diseases, exotic diseases and endemic diseases) (p>0.993 for all disease groups for both Canada and the US).
Comparison between Public and Professional Groups
Results from the same survey administered to the public were published previously [33]. The Canadian and US public groups completed the survey faster with a lower pass rate compared to their respective professional groups (Table 7). There was no difference in the gender and age distribution between the Canadian public and Canadian professional groups, but there were a higher number of study participants from Alberta, Manitoba, Ontario and Saskatchewan in the professional group. There was no difference in the geographic distribution between the US public and the US professionals, but there were a higher number of middle to older age females amongst the study participants. This may reflect the target population in the professional group. A highly educated population was observed in both professional groups compared to the public groups despite the public groups already representing a highly educated population in comparison to the national populations [33].
The percent certainty for the Canadian and US public models were 79.4% each [33]. The models presented here for the Canadian and US professionals indicate the professional models were better fitted than the public models (83.7% and 84.2%, respectively), though models of percent certainty of 70% or higher are regarded as models of good fit [51].
The mean disease criteria importance scores by country and by groups are presented in Figure 2. Similarities were observed between the combined public groups and combined professional groups while differences were observed between the public and professional groups. We identified differences in mean scores between the public and the professional groups for each disease criterion (p<0.0024) with the exception of disease incidence in humans, disease incidence in animals, animal-animal transmission potential and how much is known scientifically about the disease. The public groups considered criteria relating to the individual-level of disease burden such as case-fatality (H/A), human-to-human transmission and duration of illness (H/A) more important in the decision to prioritize (p<0.0024) while the professional groups deemed criteria relating to the societal and population-level of disease burden such as socio-economic burden (H/A), disease trend (H/A) and efficacy of control measures (H/A) to be more important (p<0.0024).
Disease criteria are presented in order of the human-related criteria with the highest mean score across all four groups, followed by the corresponding animal-related criteria. (*Disease criteria with mean scores that did not differ significantly between combined public and combined professional groups (p>0.0024 - Bonferroni-corrected p-value cut-off) using t-tests. ∧Disease criteria with mean scores that did not differ significantly across all four groups (p>0.0024 - Bonferroni-corrected p-value cut-off) using ANOVA. All remaining disease criteria had mean scores that were significantly different between combined public and professionals groups (p<0.0024) and across all four groups (p<0.0024)).
A unique priority list was derived for each group in each country (Tables 6 and 8). Although differences were observed, similarities were also apparent in diseases in the top 10 and in the bottom 10. Seven of the top ten diseases were common across all groups (rabies, Nipah virus encephalitis, Ebola virus hemorrhagic fever, Marburg hemorrhagic fever, H1N1 influenza, variant Creutzfeldt-Jakob disease and listeriosis) while five of the bottom ten diseases were common across all groups (bovine tuberculosis, cyclosporiasis, Dengue fever, La Crosse encephalitis and St. Louis encephalitis). Despite more similarities in demographics, model fit, disease criteria importance scores and part-worth utility values between the two public and the two professional groups, there was more uniformity across priority lists between the Canadian public and Canadian professionals (Spearman’s rho = 0.9707, p = 0.000) and between the US public and US professionals (Spearman’s rho = 0.9774, p = 0.000) than between the Canadian and US public groups (Spearman’s rho = 0.8497, p = 0.000) and between the Canadian and US professional groups (Spearman’s rho = 0.8356, p = 0.000).
Discussion
Zoonotic diseases are a significant public health burden in North America [6]–[9]. As there are limited resources available for their control and prevention, a scientifically driven framework for the prioritization of zoonoses is essential. An increasing number of recent studies have focused on quantitative methods for disease prioritization [13], [14], [21], [24]–[26], though there is currently no agreeable standard. In this study, we present on the novel use of the quantitative method, CA, for prioritizing zoonoses in North America. Our method fits a statistical model to choice data under a robust experimental design to generate relative weighted scores for disease criteria and levels. This overcomes the primary limitation of current methods that rely on simplified linear point-scoring systems without weights [16]–[20] or an expert Delphi panel to assign subjective weights that are applied to linear point-scoring systems [13], [14], [21], [24], [26]. Further, CA allows for disease criteria to be considered jointly, acknowledging dependence between criteria. CA forces individuals to prioritize on the basis of scientific information, eliminating biases associated with disease names and as individuals are provided with all the information necessary to prioritize, the method allows for the inclusion of individuals with no prior knowledge of the diseases (public) or health professionals who may not be knowledgeable in the full range of diseases.
In comparison to other prioritization studies, this study included a larger number of disease criteria than previously used [14], [16]–[25], although many of these studies involved diseases exclusively in humans hence our animal-related criteria would not have been appropriate in these studies. We also used a broad objective for disease prioritization, asking our respondents to prioritize diseases for their control and prevention without specifying the method of control and prevention (for example, regulation, management, vaccination, laboratory diagnosis, research and surveillance). Many studies focused on a specific aspect of disease prioritization, for example, prioritizing solely for surveillance [19], [20] or prioritizing a specialized group of diseases, for example, emerging or food-borne zoonoses only [24], [25]; these studies would require fewer disease criteria to assess diseases. Our study also included the largest number of professional participants of any disease prioritization study [14], [16]–[26]; this was feasible due to the novel CA method used in which respondents were presented with all the information necessary to prioritize diseases. This allowed for wide participation including health professionals who may not be familiar with the full range of diseases being prioritized, but could add much value to the decision to prioritize due to their specific professional training and experience. While our method involves more work and a larger study population than traditional methods, the use of CA overcomes traditional limitations in prioritization methods including simplified linear scores, subjective weights, the assumption of independence between disease criteria, biases associated with disease names and limited expert participation.
Our previous study indicated individuals with no prior knowledge of infectious diseases were capable of producing meaningful results with satisfactory model fits [33]. This study suggests professionals with knowledge or experience in prioritizing zoonoses were capable of producing meaningful results with better-fitted models than the general public. Disease criteria importance scores were realistic and sensible, consistent with findings from previous prioritization studies [13], [14], [21], [24]–[26]. Part-worth utility values demonstrated face-validity with higher preferences given to higher levels and lower preferences to lower levels. The disease priority lists generated from applying CA-derived part-worth utility values to diseases produced a list of diseases that are reasonable for prioritization, particularly when further divided into subgroups such as vector-borne diseases, food-borne and enteric diseases, exotic diseases and endemic diseases. Analysis of the uncertainty estimates in the part-worth utility values showed that disease priority lists did not change substantially; indicating good model fits producing robust part-worth utility values.
Our study found that the public groups placed more importance on disease criteria relating to the individual-level of disease burden while the professional groups placed more importance on disease criteria relating to the population-level of disease burden. Our previous study on the focus groups conducted to inform this current study affirms this observation with similar findings between the public and professional focus groups [12]. Despite closer resemblances in demographics, model fit and disease criteria importance and part-worth utility values between the combined public and combined professional groups, there was more unity in the disease priority lists between the Canadian public and Canadian professionals and between the US public and US professionals than the combined respective public and professional groups. This suggests that regional differences in country-specific disease criteria (disease incidence and trend) contributed more to disease priority rankings than differences in demographics and disease criteria preference. This finding is not surprising given study participants were asked to prioritize diseases for their control and prevention in their respective countries, moreover, if disease incidence was high and disease trend was emerging in the US but the disease was not found in Canada, the part-worth utility values assigned to the disease in Canada would reflect the lowest levels despite disease incidence and disease trend having high importance to Canadians, while the part-worth utility values assigned in the US would reflect the highest levels. This would explain why diseases such as anaplasmosis, babesiosis, paralytic shellfish poisoning and Rocky Mountain spotted fever were ranked much higher by the US groups than their respective groups in Canada.
Limitations related to the study were outlined in the previous paper [33]. Additional limitations associated with the study presented in this paper relate to the multiple recruitment methods used to recruit professionals. It is unknown whether some recruitment methods resulted in better-quality responses than others; we assume study participants responded to the survey in the same manner regardless of their mode of recruitment. However, if a particular recruitment method resulted in lower-quality survey responses, inconsistent responders would have been screened out with the two fixed choice task questions. Multiple recruitment methods may have also resulted in a different group of qualified professionals within the study population; nonetheless, the demographic data collected suggest the two professional groups were representative of individuals with some prior knowledge of infectious diseases. Further, in comparison to the public groups, the two professional groups were distinct populations with higher education and a background in an eligible professional discipline.
Although the survey was offered in three languages (English, French and Spanish) across the two countries, surveys were only completed in English and French. While we found no significant difference in the pass rate between the surveys completed in English and French, it is unknown whether translation bias may have resulted in differences in responses between languages or whether bias was introduced as a result of having no surveys completed in Spanish. An additional uncertainty that cannot be account for in this study is the quality and/or the lack of data in the literature used to assign levels of criteria to diseases [33]. It is unknown to what degree this type of uncertainty may have affect the presented results, however, the best available data available at the time of analysis was used [33] and as better quality data and more scientific information about the diseases becomes available, the priority lists presented in this study can only be improved upon.
As noted in our previous paper, there are multiple objectives for prioritizing zoonoses (for example, research, regulation, control, prevention, management, vaccination, diagnosis, cost-effective and surveillance), although study participants were asked to prioritize for policy implementation for the control and prevention of zoonoses, they may have prioritized with another objective in mind. This would be even more apparent in the professional groups who are actively involved in public health and zoonoses. There is no way to measure this type of bias and we assume participants were consistent in their prioritization objectives. Additionally, our professional study population included a wide range of professional disciplines, yet not all disciplines can be treated equally. For example, medical doctors and veterinarians are more likely to have better knowledge and experience with zoonotic diseases compared to laboratory technicians, thus, treating our professional study population uniformly may weaken differences across different groups of professionals. An argument could also be made that some professional disciplines in our study may not truly reflect experts in zoonotic diseases, for example public health nurses, despite the fact that the professionals selected in our study were distinct from individuals in the general public and that as nurses usually represent the front line of defense during an outbreak, their experience would add much value to the decision to prioritize. The same argument could be made for laboratory technicians, who may not be experts in zoonotic diseases, but have a strong understanding on the identification and diagnosis of disease-causing pathogens. This raises the question of which experts should be engaged in prioritizing zoonotic diseases for public health. While our study results represent the collective opinion of a broad group of experts across multiple disciplines, we did identify important differences between our public and professional groups. Our future research goal will be to investigate how individuals from different professional disciplines may prioritize zoonotic diseases differently and explore the issue of which individuals should be responsible for making such decisions.
We present the first zoonoses prioritization exercise involving public health, veterinary and medical professionals in North America. Our novel quantitative approach is an established method in other disciplines. Our study results were validated with satisfactory model fits and reasonable disease criteria scores and part-worth utility values. These results illustrate that CA can be used as a potential tool for the prioritization of zoonoses, particularly as a method to overcome subjective weighting and scoring of disease criteria. Other limitations that can be addressed by CA include assuming independence between disease criteria; biases associated with disease names; and limited expert participation. Although this approach involves more work and a larger study population, disease priority lists can be revised on a regular basis by updating criteria levels to match the most current disease trends, thus, a large-scale CA study can be conducted to establish the baseline disease criteria scores with minor year-to-year updates of disease priority lists. This type of scientifically driven framework for disease prioritization would be of value in North America.
Supporting Information
File S1.
Professional associations, publications and conferences targeted for professional study participants recruitment.
https://doi.org/10.1371/journal.pone.0072172.s001
(DOCX)
Acknowledgments
We acknowledge the following individuals for helping us with literature searches: Karren Lemay, Mai Pham, Amber Reed and Ashley Whiteman. Dr. David Fisman is acknowledged for his early contributions to the conception of this project.
Author Contributions
Conceived and designed the experiments: VN JMS. Performed the experiments: VN. Analyzed the data: VN. Wrote the paper: VN JMS.
References
- 1.
World Health Organization (2012) World Health Organization website. Neglected Zoonotic Diseases. Available: http://www.who.int/neglected_diseases/diseases/zoonoses/en/index.html. Accessed 2012 May 24.
- 2.
Sawtooth Software (2012) CBC/HB v5.2.8. Sequim, WA, Sawtooth Software, Inc. Available: http://www.sawtoothsoftware.com/products/cbc/cbchb.shtml. Accessed 2012 Jun 25.
- 3. Woolhouse M, Gowtage-Sequeria S (2005) Host range and emerging and reemerging pathogens. Emerging Infectious Diseases 11: 1842–1847.
- 4. Taylor LH, Latham SM, Woolhouse ME (2001) Risk factors for human disease emergence. Phil Trans R Soc Lond B 356: 983–989.
- 5. Jones K, Patel N, Levy M, Storeygard A, Balk D, et al. (2008) Global trends in emerging infectious diseases. Nature 451: 990–993.
- 6. Lindsey N, Staples J, Lehman J, Fischer M (2010) CDC (2010) Surveillance for human West Nile virus disease - United States, 1999–2008. MMWR Surveillance Summary 59: 1–17.
- 7. Kermode-Scott B (2009) Canada has world’s highest rate of confirmed cases of A/H1N1, with Aboriginal people hardest hit. British Medical Journal 339: b2746.
- 8. Bacon R, Kugeler K, Mead P (2008) CDC (2008) Surveillance for Lyme Disease - United States, 1992–2006. MMWR Surveillance Summary 57: 1–9.
- 9. Borgundvaag B, Ovens H, Goldman B, Schull M, Rutledge T, et al. (2004) SARS outbreak in the Greater Toronto Area: the emergency department experience. Canadian Medical Association Journal 171: 1342–1344.
- 10.
Heymann DL (2008) Control of Communicable Diseases Manual; Heymann DL, editor. Washington D.C: American Public Health Association. 746 p.
- 11.
Pan American Health Organization (2003) Zoonoses and communicable diseases common to man and animals; Acha P, Szyfres B, editors. Washington DC: Pan American Health Organization. Three volume set, 408, 378 and 395 pages.
- 12. Ng V, Sargeant JM (2012) A stakeholder-informed approach to the identification of criteria for the prioritization of zoonoses in Canada. PLoS ONE 7: e29752
- 13. Krause G, Alpers K, Benzler J, Bremer V, Claus H, et al. (2008) How can infectious diseases be prioritized in public health. EMBO Reports. European Molecular Biology Organization 9: S22–27.
- 14. Krause G, Alpers K, Benzler J, Bremer V, Claus H, et al. (2008) Prioritisation of infectious diseases in public health - call for comments. Eurosurveillance 13: 1–6.
- 15.
Gilsdorf A, Krause G (2011) Prioritisation of infectious diseases in public health: feedback on the prioritisation methodology, 15 July 2008 to 15 January 2009. Euro Surveillance 16: pii: 19861.
- 16. Horby P, Rushdy A, Graham C, O’Mahony M (2001) PHLS overview of Communicable Diseases 1999. Communicable Disease and Public Health 4: 8–17.
- 17.
Rushdy A, O’Mahony M (1998) PHLS overview of communicable diseases 1997: results of a priority setting exercise. Communicable Disease Report Supplement 8: S1–12.
- 18.
World Health Organization (2003) The Dubrovnik pledge on surveillance and prioritization of infectious diseases. Report on a WHO meeting in Bucharest, Romania 21–23 November, 2003. CopenhagenWHO Regional Office for Europe: World Health Organisation. 27 p.
- 19. Doherty J (2000) Establishing priorities for national communicable disease surveillance. Canadian Journal of Infectious Diseases 11: 21–24.
- 20. Doherty J (2006) Final Report and Recommendations from the National Notifiable Diseases Working Group. Canadian Communicable Disease Report 32: 211–225.
- 21. Balabanova Y, Gilsdorf A, Buda S, Burger R, Eckmanns T, et al. (2011) Communicable Diseases Prioritized for Surveillance and Epidemiological Research: Results of a Standardized Prioritization Procedure in Germany, 2011. PLoS ONE 6: e25691
- 22.
Institut de Veille Sanitaire (2002) Définition des priorités dans le domaine des zoonoses non alimentaires 2000–2002. ParisFrance: Institut de Veille Sanitaire (InVS). 40 p.
- 23.
Institut de Veille Sanitaire (2010) Définition des priorités dans le domaine des zoonoses non alimentaires 2008–2009. ParisFrance: Institut de Veille Sanitaire (InVS). 31 p.
- 24. Cardoen S, van Huffel X, Berkvens D, Quoilin S, Ducoffre G, et al. (2009) Evidence-based semiquantitative methodology for prioritization of foodborne zoonoses. Foodborne pathogens and disease 6: 1083–1096.
- 25. Havelaar AH, van Rosse F, Bucura C, Toetenel MA, Haagsma JA, et al. (2010) Prioritizing Emerging Zoonoses in The Netherlands. PLoS ONE 5: e13965
- 26.
Humblet M, Vandeputte S, Albert A, Gosset C, Kirschvink N, et al.. (2012) Multidisciplinary and Evidence-based Method for Prioritizing Diseases of Food-producing Animals and Zoonoses. Emerging Infectious Diseases 18: doi: 10.3201/eid1804.111151.
- 27. Green P, Srinivasan V (1978) Conjoint analysis in consumer research: Issues and outlook. Journal of Consumer Research 5: 103–123.
- 28. Ryan M, Farrar S (2000) Using conjoint analysis to elicit preferences for health care. British Medical Journal 320: 1530–1533.
- 29. Mele N (2008) Conjoint analysis: using a market-based research model for healthcare decision making. Nursing Research 57: 220–224.
- 30. Sampietro-Colom L, Espallargues M, Rodríguez E, Comas M, Alonso J, et al. (2008) Wide social participation in prioritizing patients on waiting list for joint replacement: a conjoint analysis. Medical Decision Making 28: 554–566.
- 31. Bridges J, Hauber A, Marshall D, Lloyd A, Prosser L, et al. (2011) Conjoint Analysis Applications in Health - a Checklist: A Report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value in Health 14: 403–413.
- 32.
Orme BK (2010) Getting started with conjoint analysis: Strategies for product design and pricing research. MadisonWisconsin: Research Publishers, LLC. 210 p.
- 33.
Ng V, Sargeant JM (2012) A quantitative and novel approach to the prioritization of zoonotic diseases in North America: A public perspective. PLoS ONE 7: doi: 10.1371/journal.pone.0048519.
- 34.
Research Now (2010) Online Internet Panel Provider. Available: http://www.researchnow.com/en-CA.aspx. Accessed 2010 Aug 31.
- 35.
Sawtooth Software (2010) SSI Web v7. Sequim, WA, Sawtooth Software, Inc. Available: http://www.sawtoothsoftware.com/products/ssiweb/. Accessed 2010 Apr 23.
- 36. Patterson M, Chrzan K (2003) Partial Profile Discrete Choice: What’s the Optimal Number of Attributes? Sawtooth Software Conference Proceedings, San Antonio, Texas, April 15–17 2003: 173–185.
- 37. Chrazn K (2010) Using partial profile choice experiments to handle large numbers of attributes. International Journal of Market Research 52: 827–840.
- 38.
Sawtooth Software (2010) CBC module v7. Sequim, WA, Sawtooth Software, Inc. Available: http://www.sawtoothsoftware.com/products/cbc/. Accessed 2010 Apr 23.
- 39.
Sawtooth Software (2008) CBC v6.0 Technical Paper: The CBC System for Choice-Based Conjoint Analysis. SequimWashington: Sawtooth Software. 26 p.
- 40.
Sawtooth Software (2009) CBC/HB Version 5.0 Technical Paper: The CBC/HB System for Hierarchical Bayes Estimation. SequimWashington: Sawtooth Software. 31 p.
- 41.
Howell J (2009) Sawtooth Software Research Papers Series: CBC/HB for Beginners. SequimWashington: Sawtooth Software. 5 p.
- 42.
Sawtooth Software (2010) SMRT v4.22.0 v7. Sequim, WA, Sawtooth Software, Inc. Available: http://www.sawtoothsoftware.com/solutions/market_simulations/. Accessed 2010 Apr 23.
- 43.
Statistics Canada (2006) Statistics Canada website. Population 15 years and over by highest certificate, diploma or degree, by age groups, 2006 Census. Available: http://www.statcan.gc.ca/tables-tableaux/sum-som/l01/cst01/educ43a-eng.htm. Accessed 2012 Jun 20.
- 44.
Statistics Canada (2011) Statistics Canada website. Age and Sex Highlight Tables, 2011 Census. Available: http://www12.statcan.gc.ca/census-recensement/2011/dp-pd/hlt-fst/as-sa/?Lang=E. Accessed 2012 Jun 20.
- 45.
United States Census Bureau (2010) United States Census Bureau website. Educational Attainment in the United States: 2010 - Detailed Tables. Available: http://www.census.gov/hhes/socdemo/education/data/cps/2010/tables.html. Accessed 2012 Jun 20.
- 46.
United States Census Bureau (2010) United States Census Bureau website. 2010 Census Data. Available: http://2010.census.gov/2010census/data/. Accessed 2012 Jun 20.
- 47. Vannier E, Gewurz B, Krause P (2008) Human babesiosis. Infectious Disease Clinics of North America 22: 469–488.
- 48. Howden K, Geale D, Paré J, Golsteyn-Thomas E, Gajadhar A (2010) An update on bovine anaplasmosis (Anaplasma marginale) in Canada. The Canadian Veterinary Journal 51: 837–840.
- 49. Dantas-Torres F (2007) Rocky Mountain spotted fever. Lancet Infectious Diseases 7: 724–732.
- 50. Centers for Disease Control and Prevention (2011) Paralytic shellfish poisoning – southeast Alaska, May to June 2011. Morbidity Mortality Weekly Report 18: 1554–1556.
- 51.
Orme BK (2012) Sawtooth Software website. Model fit values. Available: http://www.sawtoothsoftware.com/forum/832/model-fit-values. Accessed 2012 Jun 20.