Accounting for Heterogeneity via Random-Effects Models and Moderator Analyses in Meta-Analysis
Abstract
Abstract. To conduct a meta-analysis, one needs to express the results from a set of related studies in terms of an outcome measure, such as a standardized mean difference, correlation coefficient, or odds ratio. The observed outcome from a single study will differ from the true value of the outcome measure because of sampling variability. The observed outcomes from a set of related studies measuring the same outcome will, therefore, not coincide. However, one often finds that the observed outcomes differ more from each other than would be expected based on sampling variability alone. A likely explanation for this phenomenon is that the true values of the outcome measure are heterogeneous. One way to account for the heterogeneity is to assume that the heterogeneity is entirely random. Another approach is to examine whether the heterogeneity in the outcomes can be accounted for, at least in part, by a set of study-level variables describing the methods, procedures, and samples used in the different studies. The purpose of the present paper is to discuss these different approaches with particular emphasis on the interpretation of the results and practical issues.
References
Babyak, M.A. (2004). What you see may not be what you get: A brief, nontechnical introduction to overfitting in regression-type models. Psychosomatic Medicine, 66, 411– 421 .Byrnes, J.P. , Miller, D.C. , Schafer, W.D. (1999). Gender differences in risk taking: A meta-analysis. Psychological Bulletin, 125, 367– 383 .Chalmers, I. , Hedges, L.V. , Cooper, H. (2002). A brief history of research synthesis. Evaluation and the Health Professions, 25, 12– 37 .Cooper, H.M. (1998). Synthesizing research: A guide for literature reviews (3rd ed.). Thousand Oaks, CA: Sage .DerSimonian, R. , Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177– 188 .Field, A.P. (2005). Is the meta-analysis of correlation coefficients accurate when population correlations vary?. Psychological Methods, 10, 444– 467 .Fleiss, J.L. (1994). Measures of effect size for categorical data. In H.M. Cooper & L.V. Hedges (Eds.), The handbook of research synthesis (pp. 245-260). New York: Russell Sage .Glass, G.V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3– 8 .Glass, G.V. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351– 379 .Glasziou, P.P. , Sanders, S.L. (2002). Investigating causes of heterogeneity in systematic reviews. Statistics in Medicine, 21, 1503– 1511 .Gleser, L.J. , Olkin, I. (1994). Stochastically dependent effect sizes. In H.M. Cooper & L.V. Hedges (Eds.), The handbook of research synthesis (pp. 339-355). New York: Russell Sage .Green, S.B. (1991). How many subjects does it take to do a regression analysis?. Multivariate Behavioral Research, 26, 499– 510 .Hall, J.A. , Rosenthal, R. (1991). Testing for moderator variables in meta-analysis: Issues and methods. Communication Monographs, 58, 438– 448 .Hedges, L.V. (1982). Estimation of effect size from a series of independent experiments. Psychological Bulletin, 92, 490– 499 .Hedges, L.V. (1983). A random effects model for effect sizes. Psychological Bulletin, 93, 388– 395 .Hedges, L.V. , Olkin, I. (1985). Statistical methods for meta-analysis . San Diego, CA: Academic Press .Hedges, L.V. , Pigott, T.D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9, 426– 445 .Hedges, L.V. , Vevea, J.L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486– 504 .Higgins, J.P.T. , Thompson, S.G. , Deeks, J.J. , Altman, D.G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557– 560 .Horst, P. (1941). The role of predictor variables which are independent of the criterion. Social Science Research Council Bulletin, 48, 431– 436 .Hunter, J.E. , Schmidt, F.L. (1990). Methods of meta-analysis: Correcting error and bias in research findings . Newbury Park, CA: Sage .Jackson, G.B. (1980). Methods for integrative reviews. Review of Educational Research, 50, 438– 460 .Knapp, G. , Hartung, J. (2003). Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine, 22, 2693– 2710 .Konstantopoulos, S. , Hedges, L.V. (2004). Meta-analysis. In D. Kaplan (Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 281-297). Thousand Oaks, CA: Sage .Lau, J. , Ioannidis, J.P.A. , Schmid, C.H. (1998). Summing up evidence: One answer is not always enough. Lancet, 351, 123– 127 .Lecrubier, Y. , Clerc, G. , Didi, R. , Kieser, M. (1994). Efficacy of St John's wort extract WS 5570 in major depression: A double-blind, placebo-controlled trial. American Journal of Psychiatry, 159, 1361– 1366 .Lee, W.-L. , Bausell, R.B. , Berman, B.M. (2001). The growth of health-related meta-analyses from 1980 to 2000. Evaluation and the Health Professions, 24, 327– 335 .Light, R.J. (1987). Accumulating evidence from independent studies: What we can win and what we can lose. Statistics in Medicine, 6, 221– 228 .Light, R.J. , Pillemer, D.B. (1984). Summing up: The science of reviewing research . Cambridge, MA: Harvard University Press .Linde, K. , Berner, M. , Egger, M. , Mulrow, C. (2005). St. John's wort for depression: Meta-analysis of randomized controlled trials. British Journal of Psychiatry, 186, 99– 107 .Lipsey, M.W. , Wilson, D.B. (2001a). Practical meta-analysis . Thousand Oaks, CA: Sage .Lipsey, M.W. , Wilson, D.B. (2001b). The way in which intervention studies have “personality” and why it is so important to meta-analysis. Evaluation and the Health Professions, 24, 236– 254 .MacCallum, R.C. , Zhang, S. , Preacher, K.J. , Rucker, D.D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19– 40 .McDaniel, M.A. , Whetzel, D.L. , Schmidt, F.L. , Maurer, S.D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79, 599– 616 .Meadows, A.J. (1998). Communication research . San Diego: Academic Press .Miller, A. (2002). Subset selection in regression (2nd ed.). London: Chapman & Hall .Montgomery, S.A. , Hübner, W.D. , Grigoleit, H.G. (2000). Efficacy and tolerability of St John's wort extract compared with placebo in patients with a mild to moderate depressive disorder. Phytomedicine, 7(suppl. 2), 107–Moher, D. , Jadad, A.R. , Nichol, G. , Penman, M. , Tugwell, P. , Walsh, S. (1995). Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists. Controlled Clinical Trials, 16, 62– 73 .Neter, J. , Kutner, M.H. , Nachtsheim, C.J. , Wasserman, W. (1996). Applied linear statistical models (4th ed.). Chicago: Irwin .Pigott, T.D. (2001). Missing predictors in models of effect size. Evaluation and the Health Professions, 24, 277– 307 .Pillemer, D.B. , Light, R.J. (1980). Benefiting from variation in study outcomes. In R. Rosenthal (Ed.), New directions for methodology of social and behavioral science: Quantitative assessment of research domains (Vol. 5, pp. 1-12). San Francisco: Jossey-Bass .Raudenbush, S.W. (1994). Random effects models. In H.M. Cooper & L.V. Hedges (Eds.), The handbook of research synthesis (pp. 301-321). New York: Russell Sage .Raudenbush, S.W. , Bryk, A.S. (1985). Empirical Bayes meta-analysis. Journal of Educational Statistics, 10, 75– 98 .Robinson, W.S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351– 357 .Rosenthal, R. (1991). Meta-analytic procedures for social research . Newbury Park, CA: Sage .Rosenthal, R. (1994). Parametric measures of effect size. In H.M. Cooper & L.V. Hedges (Eds.), The handbook of research synthesis (pp. 231-244). New York: Russell Sage .Rosenthal, R. , DiMatteo, M.R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59– 82 .Rothstein, H.R. , Sutton, A.J. , Borenstein, M. Eds. (2005). Publication bias in meta-analysis: Prevention, assessment, and adjustments . Chichester, England: Wiley .Sánchez-Meca, J. , Marín-Martínez, F. (1997). Homogeneity tests in meta-analysis: A Monte Carlo comparison of statistical power and Type I error. Quality and Quantity, 31, 385– 399 .Schafer, J.L. , Graham, W.K. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147– 177 .Schulze, R. (2004). Meta-analysis: A comparison of approaches . Cambridge, MA: Hogrefe & Huber .Shaffer, J.P. (1995). Multiple hypothesis testing. Annual Review of Psychology, 46, 561– 584 .Sharp, S. (1998). Meta-analysis regression. Stata Technical Bulletin, 42, 16– 24 .Sheu, C.-F. , Suzuki, S. (2001). Meta-analysis using linear mixed models. Behavior Research Methods, Instruments, and Computers, 33, 102– 107 .Sidik, K. , Jonkman, J.N. (2005). Simple heterogeneity variance estimation for meta-analysis. Applied Statistics, 54, 367– 384 .Slavin, R.E. (1986). Best-evidence synthesis: An alternative to meta-analytic and traditional reviews. Educational Researcher, 15, 5– 11 .Smith, M.L. (1980). Publication bias and meta-analysis. Evaluation in Education, 4, 22– 24 .Smith, M.L. , Glass, G.V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752– 760 .Steel, P.D. , Kammeyer-Mueller, J.D. (2002). Comparing meta-analytic moderator estimation techniques under realistic conditions. Journal of Applied Psychology, 87, 96– 111 .Sterne, J.A.C. , Bradburn, M.J. , Egger, M. (2001). Meta-analysis in Stata. In M. Egger, G.D. Smith, & D.G. Altman (Eds.), Systematic reviews in health care: Meta-analysis in context (pp. 347-369). London: BMJ Books .Stewart, L.A. , Tierney, J.F. (2002). To IPD or not to IPD? Advantages and disadvantages of systematic reviews using individual patient data. Evaluation and the Health Professions, 25, 76– 97 .Thompson, S.G. , Sharp, S.J. (1999). Explaining heterogeneity in meta-analysis: A comparison of methods. Statistics in Medicine, 18, 2693– 2708 .Viechtbauer, W. (2005). Bias and efficiency of meta-analytic variance estimators in the random-effects model. Journal of Educational and Behavioral Statistics, 30, 261– 293 .Viechtbauer, W. (2007). Hypothesis tests for population heterogeneity in meta-analysis. British Journal of Mathematical and Statistical Psychology, 60, 29– 60 .Viswesvaran, C. , Sanchez, J.I. (1998). Moderator search in meta-analysis: A review and cautionary note on existing approaches. Educational and Psychological Measurement, 58, 77– 87 .Wicherts, J.M. , Borsboom, D. , Kats, J. , Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726– 728 .
