Skip to main content
  • Research article
  • Open access
  • Published:

Comparison of three tests of homogeneity of odds ratios in multicenter trials with unequal sample sizes within and among centers

Abstract

Background

Mixed effects logistic models have become a popular method for analyzing multicenter clinical trials with binomial data. However, the statistical properties of these models for testing homogeneity of odds ratios under various conditions, such as within-center and among-centers inequality, are still unknown and not yet compared with those of commonly used tests of homogeneity.

Methods

We evaluated the effect of within-center and among-centers inequality on the empirical power and type I error rate of the three homogeneity tests of odds ratios including likelihood ratio (LR) test of a mixed logistic model, DerSimonian-Laird (DL) statistic and Breslow-Day (BD) test by simulation study. Moreover, the impacts of number of centers (K), number of observations in each center and amount of heterogeneity were investigated by simulation.

Results

As compared with the equal sample size design, the power of the three tests of homogeneity will decrease if the same total sample size, which can be allocated equally within one center or among centers, is allocated unequally. The average reduction in the power of these tests was up to 11% and 16% for within-center and among-centers inequality, respectively. Moreover, in this situation, the ranking of the power of the homogeneity tests was BD≥DL≥LR and the power of these tests increased with increasing K.

Conclusions

This study shows that the adverse effect of among-centers inequality on the power of the homogeneity tests was stronger than that of within-center inequality. However, the financial limitations make the use of unequal sample size designs inevitable in multicenter trials. Moreover, although the power of the BD is higher than that of the LR when K≤6, the proposed mixed logistic model is recommended when K≥8 due to its practical advantages.

Peer Review reports

Background

The results from multicenter clinical trials or meta-analysis studies with binomial data are often summarized in K 2 × 2 contingency tables, where K denotes the total number of centers or studies. Combining data in such tables and proposition a summary measure is the primary objective of such studies. However, before computing the overall odds ratio, we often need to assess whether the specific odds ratios are homogeneous across tables [14].

Nowadays, investigators have a wide range of methods available for this purpose, including model-based and test-based approaches. The excellent simulation studies conducted by pioneer researchers in this field assist us in choosing the most appropriate test for the assessment of homogeneity among K 2 × 2 tables [16]. Nevertheless, the results of these simulation studies indicate that homogeneity tests show different behaviors under combinations of parameters such as the number of centers, center sizes and amount of heterogeneity [3, 5, 7].

In recent years, a class of models called mixed logistic models has been used for analysis of multicenter clinical trials with binomial data. Although Agresti has discussed a likelihood ratio (LR) test based on a mixed logistic model for testing homogeneity of odds ratios in K 2 × 2 contingency tables [8], the statistical properties of this test and the other traditional homogeneity tests such as Breslow-Day (BD) [9] and DerSimonian-Laird (DL) [10] are still unknown. A situation which occurs frequently in multicenter trials and has not been evaluated in previous studies is the effect of unequal sample size designs on the statistical properties of these homogeneity tests. For example, in some multicenter clinical trials, when one center is larger, it may seem reasonable to select a larger sample from it, but this leads to among-centers inequality. Moreover, within-center inequality could occur when the costs of two treatment groups are very different. In this situation, due to financial restrictions, it is reasonable to allocate more patients to the cheaper treatment in each center. This simulation study hence compares the empirical power and type I error rate of the three tests of homogeneity of odds ratios, including LR, DL and BD tests when the sample size is unequal within one center or among centers.

Methods

Consider a series of K independent 2 × 2 contingency tables, with the data in the kth table denoted as shown in Table 1. In this paper, K is the number of centers in multicenter clinical trials.

Table 1 Summary of data from the kth 2 × 2 contingency table

Suppose that each y ik follows a binomial distribution with parameters n ik and π ik (i = 1,2; k = 1,2,...,K). Let n ik denote the total number of observations in the ith treatment arm and kth center, and let π ik denote the success probability at treatment level x ik in the kth center, where x ik is the treatment indicator with x ik = 1 representing treatment 1 and x ik = 0 treatment 2.

In this paper, we focus on the following logistic-normal probability model in order to assess the homogeneity of odds ratios:

(1)

In this model, the segment α + βx ik is the fixed effect part in which β is the common treatment effect. Here, u k and b ik are independent random components of the model, where u k is the center effect and and the parameter summarizes center heterogeneity. In addition, b ik is the center-by-treatment interaction random effect, and the parameter describes variability in the log-odds ratios [8]. The two advantages of working with this model are: first, a test of homogeneity of odds ratios can be performed by testing the null hypothesis: against ; and, second, the common treatment effect and also center-specific odds ratios can be obtained by estimating β and b ik [8].

It should be noted that the homogeneity test can be performed using likelihood ratio (LR) test Δ = -2(l 0 - l 1 ), where l 0 is the log-likelihood under the assumption of homogeneity of odds ratios and when there is just one random effect, u k , in the Model 1 and l 1 is the log-likelihood when both u k and b ik are in the model. Under the null hypothesis, the asymptotic distribution of the LR test is a mixture of a chi-squared distribution with zero and one degrees of freedom, respectively, both with weight of 1/2 [11, 12].

Since the properties of the LR test in order to assess homogeneity of odds ratios had not been evaluated in the previous studies, we investigated the behavior of this test under various conditions, particularly under within-center and among-centers inequality in multicenter trials and also compared it with the other two most common test statistics, including DL and BD tests. Brief calculation details of these statistics are given in the appendix [912].

Simulation study

We studied the statistical properties of the three mentioned-above homogeneity tests, in terms of empirical power and type I error rate, by simulation in SAS statistical package v 9.1. Our simulation design was based on the mixed logistic model 1; we set the fixed effect parameters for all simulation scenarios arbitrarily as: α = -1 and β = 1 . The random components u k and b ik are generated independently from normal distribution with mean zero and variance and . In order to generate the response variable, the binomial distribution with parameters n ik and was used. The influence of within-center and among-centers inequality, different values of K, , and also number of observations in each centers, n k , on the statistical properties of all the homogeneity tests were evaluated. We generated 1000 random data sets for each simulation scenario. Namely, we chose three values for K (4, 6 and 8), two values for (0.1 and 0.5) and five values for (0, 0.2, 0.4, 0.6 and 0.8). In the equal sample size design, in which equal numbers of patients were allocated between two treatment arms and also among all centers, the number of individuals per center, n k , was set at 40, 100 and 200. In addition, in this study two forms of unequal sample size designs were considered. In the first one, ie within-center inequality, the same sample size, which can be allocated equally between two treatment arms within a center, is allocated unequally in the ratio of 3:1. In this case, the sample sizes were considered equal in all centers. In the latter one, ie among-centers inequality, the same total sample size, which can be allocated equally among centers, is allocated unequally. In this situation, one center has a much larger sample size in comparison with the other ones. The exact details of different sample size configurations are described in Table 2. In addition, SAS simulation code for performing the three homogeneity tests is presented in the additional file 1.

Table 2 Description of different configurations of sample size in equal and unequal sample size designs.

Results

Equal sample size design

Tables 3 shows empirical type I error rate (when ) and power (when ) of the three homogeneity tests under the equal sample size design and various combinations of n k , K, , and . In terms of power, the tests are ordered BD≥DL≥LR in all cases and they perform similarly when the total sample size in each center is as large as 200 and . In addition, our findings reveal that substantial gain in the power of all homogeneity tests occurred with increasing number of centers, sample size per center and degree of heterogeneity of odds ratios, . However, increasing degree of heterogeneity of center, , from 0.1 to 0.5 resulted in a reduction of approximately 4.93% and 3.76% in the power of DL and BD, respectively, while no regular trend was detected for the change in LR.

Table 3 Type I error rate (when ) and the statistical power (when ) of homogeneity tests under equal sample size design.

Unequal sample size designs

Table 4 shows type I error rate and the power of the homogeneity tests under within-center inequality for different values of n k , K, , and . Indeed, when = 0.1, as compared with the equal sample size design, there were decreases of approximately 16.35%, 9.93% and 11.62% in the power of LR test for within-center inequality, when K was equal to 4, 6 and 8, respectively. In addition, the power of the BD statistic decreased approximately by 10.67%, 7.79% and 7.15% and the power of DL was reduced approximately by 14.46%, 11.24% and 9.76% for K = 4, 6 and 8, respectively. It should be noted that, for a given value of K, the reported amount of reduction was the average reduction for all values of .

Table 4 Type I error rate (when ) and the statistical power (when ) of homogeneity tests under within-center inequality.

Type I error rate and the power of the homogeneity tests under among-centers inequality are illustrated in Table 5. In fact, when = 0.1 as compared with the equal sample size design, there were decreases of approximately 17.15%, 11.78% and 11.89% in the power of LR test and 16.63%, 13.62% and 12.57% in the power of the BD statistic for among-centers inequality, when K was equal to 4, 6 and 8, respectively. In addition, this amount of reduction in the power of DL test was more critical and approximately equal to 21.56%, 18.4% and 16.61% for the same sequence of K.

Table 5 Type I error rate (when ) and the statistical power (when ) of homogeneity tests under among-centers inequality.

It should be pointed out that the amount of reduction in the power of the three homogeneity tests for = 0.5 under two forms of unequal sample size designs was approximately similar to those of = 0.1.

Type I error rate

Tables 3, 4 and 5, when = 0, represent empirical type I error rate of the three homogeneity tests at the nominal significance level of 0.05. As indicated, LR test showed conservative behavior when K = 4, otherwise, the type I error rate was close to the nominal level. In addition, BD performed adequately in terms of type I error rate in almost all cases. On the other hand, type I error rate of the DL statistic was close or below the nominal level.

Discussion and conclusions

In a simple randomized clinical trial, the use of unequal allocation ratios, particularly the allocation ratio of 3:1, will significantly reduce the power of study for detecting significance difference between two treatments [1315]. To our knowledge, few published studies investigated the impact of within-center and among-centers inequality on the statistical properties of the tests of homogeneity of odds ratios in multicenter clinical trials [1, 3, 4]. As illustrated in Tables 3, 4 and 5, the type I error rate of the three homogeneity tests is approximately close to the nominal level of 0.05 except for LR when K = 4. Since the results show that these tests have almost the same type I error rate, power comparisons are possible. As compared with the equal sample size design, the power of the LR, BD and DL tests will decrease if the same total sample size, which can be allocated equally within one center or among centers, is allocated unequally. In this case, the power ranking of the tests was BD≥DL≥LR. It is worth mentioning that, as compared with within-center inequality, among-centers inequality has stronger adverse effect on the power of the homogeneity tests. Despite the use of different tests, these findings are inconsistent with those of Paul, who reported the adverse effect of within-center inequality to be stronger [3].

Also, this paper shows how to use a mixed logistic model to test homogeneity of odds ratios in multicenter trials. In Model 1, there are two types of homogeneity: homogeneity of odds ratios among centers and homogeneity of centers. However, removing the center-by-treatment interaction from Model 1 leads to a model which can only be used to test homogeneity of centers. This model, which has been previously discussed by Gao, assumes that the odds ratios are constant over centers [16]. Therefore, it should not be used to generate data for comparing the tests of heterogeneity of odds ratios. Furthermore, the power of the three tests of homogeneity increases more when we increase the number of K and n k compared to when we increase the number of K and . This result is in agreement with previous studies which have evaluated the influence of K and n k on the power of the homogeneity tests [5, 1719]. Nevertheless, our simulation study shows that the degree of among-centers heterogeneity, , has little or no effect on the power of the three tests of homogeneity, except for DL when and sample size is small.

In addition, it is noteworthy that we used the DL statistic calculated from the one-way random effects model, which has approximately a chi-square distribution. However, Biggerstaff and Jackson [20] have calculated the exact distribution and power of the well-known Q statistic based on the same random effects model, which can be used for testing homogeneity of odds ratios and be compared with the tests used in the present study.

In conclusion, of the three tests of homogeneity, the BD seems to be the most appealing with regard to its statistical properties: its type I error rate is close to the nominal level and its power is greater than that of DL and LR. Moreover, it has the advantage of simplicity of calculation and is recommended by a number of authors [1, 46]. However, one limitation of BD test is that it has low power when the sample size within each center is small, even if the number of centers is large [1, 2]. Nevertheless, despite having low power under small number of centers and its complexity, Model 1 has its own advantages. Firstly, when the centers are a random sample themselves, the LR test from the Model 1 enables inferences to extend to the population of centers. Secondly, a further consideration is that common odds ratio can be estimated from the fixed part of the Model 1, even when the odds ratios are not homogeneous. Thirdly, in each center, Model 1 provides a predicted log-odds ratio that shrinks the sample value toward the mean. This is especially useful when the sample size in a center is small and the ordinary sample odds ratio has a large standard error [8]. In addition, the mixed logistic model described in this study will potentially be applicable to meta-analysis studies.

It is clear that, based on Model 1, the odds ratio in the kth center, as given in the appendix 4, is exp(β + b 1k - b 2k ), which can be written as C × exp(b 1k - b 2k ) where C = exp(β) .This indicates that the odds ratio in each center is absolutely independent of α and u k . Indeed, the odds ratios are affected by b 1k and b 2k , and β has the same effect on odds ratio in all centers. Hence, to generate heterogeneous odds ratios among centers, the fixed simulation parameters, ie α and β, can be chosen arbitrarily.

It should be noted that, although using unequal sample size designs in multicenter clinical trials reduces both the power of the study and the power of the homogeneity tests, a substantial reduction in the total cost of the trial will compensate for the reduction in the power of the statistical tests [14, 15]. Finally, further research is warranted to investigate the influence of the number of centers, unequal sample size design, sparseness and also deviation from normal assumption of the random effects on the robustness and accuracy of the estimates of the fixed and random parameters of the Model 1.

Appendix

1. Breslow-Day statistic

Breslow and Day [9] proposed the test statistic:

Where OR MH is the Mantel-Haenszel estimator of common odds ratio. E(y 1k |OR MH ) and V(y 1k |OR MH ) are the expected value and variance of y 1k under the null hypothesis of homogeneity of odds ratios. Under the assumption of large sample size in each 2 × 2 table, BD has approximately chi-square distribution with k - 1 degree of freedom [9].

2. DerSimonian-Laird statistic

The Dersimonian and laird statistic is based on random effects model, which obtained by:

When log-odds ratio is used as a summary measure ie, common treatment effect, y k = ln(OR k ) where ln(OR k ) is the log-odds ratio in kth center and where . DL statistic has approximately chi-square distribution with k - 1 degree of freedom [10].

3. Likelihood ratio test based on mixed logistic models

To test homogeneity of odds ratios based on mixed logistic models, we compare the model

to the simpler model not containing the random effect term of the center-by-treatment interaction, namely:

which is equivalent to test the null hypothesis of versus [8]. As the null hypothesis is on the boundary of the parameter space, the asymptotic null distribution for the likelihood ratio test is a mixture of a and a with equal probability 1/2, rather than classical single chi-square distribution [11, 12].

4. Calculation of odds ratio based on the mixed logistic model with two random effects

Based on Model * which was presented above, the odds of success in treatment 1 in kth center is derived as:

and in a similar manner the odds of success in treatment 2 is derived as: therefore, the odds ratio for treatment 1 versus 2 in kth center is exp(β+b 1k - b 2k ).

5. SAS code for performing likelihood ratio test in a 2 × 2 × 2 contingency tables

data binomial;

input center treat y n; * y successes out of n trials;

cards;

1 1 30 100

1 2 50 100

2 1 45 100

2 2 75 100

;

run;

proc nlmixed data = binomial qpoints = 15; *Mixed logistic model with one random effect, no interaction;

parms alpha = -1 beta = 1 su = 0.2; *Initial values for parameters estimates;

bounds su>=0;

z=alpha+beta*treat+u; *Logistic formula;

expz=exp(z);

pi=expz/(1+expz);

model y~binomial(n,pi);

random u~ normal(0,su*su) subject=center;

ods output FitStatistics=test1;

ods listing select test1;

run;

proc nlmixed data=binomial qpoints = 15; *Mixed logistic model with two random effects, interaction;

parms alpha=-1 beta = 1 su = 0.2 sb = 0.8; *Initial values;

bounds su>0; bounds sb>0;

z=alpha+beta*treat+a+b*treat; *Logistic formula;

expz=exp(z);

pi=expz/(1+expz);

model y~binomial(n,pi);

random a b~ normal([0,0],[su*su,0,sb*sb]) subject=center;

ods output FitStatistics=test2;

ods listing select test2;

run;

data mixed; *Calculating likelihood ratio statistic;

merge test1(rename=(value=d1)) test2(rename=(value=d2));

if descr='-2 Log Likelihood';

run;

data combmix; set mixed;

delta=d1-d2;

run;

data lr; set combmix;

x=probchi(delta,1); *Testing hypothesis of homogeneity of odds ratios, based on mixture chi-square;

run;

data rejectmixed;

set lr;

rej1 = 0.5*(1-x);

a=(rej1<0.05);

run;

References

  1. Jones MP, O'Gorman TW, Lemke JH, Woolson RF: A Monte Carlo investigation of homogeneity tests of the odds ratio under various sample size configurations. Biometrics. 1989, 45: 171-81. 10.2307/2532043.

    Article  CAS  PubMed  Google Scholar 

  2. O'Gorman TW, Woolson RF, Jones MP, Lemke JH: Statistical analysis of K 2 × 2 tables: A comparative study of estimators/test statistics for association and homogeneity. Environ Health Perspect. 1990, 87: 103-107.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Paul SR, Donner A: A comparison of tests of homogeneity of odds ratios in K 2 × 2 tables. Stat Med. 1989, 8: 1455-1468. 10.1002/sim.4780081205.

    Article  CAS  PubMed  Google Scholar 

  4. Reis IM, Hirji KF, Afifi AA: Exact and asymptotic tests for homogeneity in several 2 × 2 tables. Stat Med. 1999, 18: 893-906. 10.1002/(SICI)1097-0258(19990430)18:8<893::AID-SIM84>3.0.CO;2-5.

    Article  CAS  PubMed  Google Scholar 

  5. Gavaghana DJ, Moore RA, McQuay HJ: An evaluation of homogeneity tests in meta-analyses in pain using simulations of individual patient data. Pain. 2000, 85: 415-424. 10.1016/S0304-3959(99)00302-4.

    Article  Google Scholar 

  6. Paul SR, Donner A: Small sample performance of tests of homogeneity of odds ratios in K 2 × 2 tables. Stat Med. 1992, 11: 159-165. 10.1002/sim.4780110203.

    Article  CAS  PubMed  Google Scholar 

  7. Mittlböck M, Heinzl H: A simulation study comparing properties of heterogeneity measures in meta-analyses. Stat Med. 2006, 25: 4321-4333. 10.1002/sim.2692.

    Article  PubMed  Google Scholar 

  8. Agresti A, Hartzel J: Strategies for comparing treatments on a binary response with multi-centre data. Stat Med. 2000, 19: 1115-1139. 10.1002/(SICI)1097-0258(20000430)19:8<1115::AID-SIM408>3.0.CO;2-X.

    Article  CAS  PubMed  Google Scholar 

  9. Breslow N, Day NE: Calssical methods of analysis of grouped data. Statistical Methods in Cancer Research. The Analysis of Case-Control Studies. 1980, Lyon: International Agency for Research on Cancer, 1: 122-159.

    Google Scholar 

  10. DerSimonian R, Laird N: Meta-analysis in clinical trials. Control Clin Trials. 1986, 7: 177-188. 10.1016/0197-2456(86)90046-2.

    Article  CAS  PubMed  Google Scholar 

  11. Verbeke G, Molenberghs G: The use of score tests for inference on variance components. Biometrics. 2003, 59: 254-262. 10.1111/1541-0420.00032.

    Article  PubMed  Google Scholar 

  12. Molenberghs G, Verbeke G: Likelihood ratio, score, and Wald tests in a constrained parameter space. Am Statistician. 2007, 61: 1-6. 10.1198/000313007X168173.

    Article  Google Scholar 

  13. Jafari P, Ayatollahi SMT, Behboodian J: Sequential boundaries approach in clinical trials with unequal allocation ratios. BMC Med Res Methodol. 2006, 6: 1-10.1186/1471-2288-6-1.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Torgerson DJ, Campbell MK: Use of unequal randomisation to aid the economic efficiency of clinical trials. BMJ. 2000, 321: 759-10.1136/bmj.321.7263.759.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Torgerson DJ, Campbell MK: Unequal randomisation can improve economic efficiency of clinical trials. J Health Serv Res Policy. 1997, 2: 81-85.

    CAS  PubMed  Google Scholar 

  16. Gao S: Combining binomial data using the logistic normal model. J Stat Comput. 2004, 74: 293-306. 10.1080/0094965031000151169.

    Article  Google Scholar 

  17. Hardy RJ, Thompson SG: Detecting and describing heterogeneity in meta-analysis. Stat Med. 1998, 17: 841-856. 10.1002/(SICI)1097-0258(19980430)17:8<841::AID-SIM781>3.0.CO;2-D.

    Article  CAS  PubMed  Google Scholar 

  18. Huedo-Medina TB, Sanchez-Meca J, Marin-Martinez F: Assessing heterogeneity in meta-analysis: Q statistic or I2 index?. Psychol Methods. 2006, 11: 193-206.

    Article  PubMed  Google Scholar 

  19. Viechtbauer W: Hypothesis tests for population heterogeneity in meta-analysis. Br J Math Stat Psychol. 2007, 60: 29-60. 10.1348/000711005X64042.

    Article  PubMed  Google Scholar 

  20. Biggerstaff BJ, Jackson D: The exact distribution of Cochran's heterogeneity statistic in one-way random effects meta-analysis. Stat Med. 2008, 27: 6093-6110. 10.1002/sim.3428.

    Article  PubMed  Google Scholar 

Pre-publication history

Download references

Acknowledgements

The authors are grateful to Keivan Shalileh for his kind editing of this paper. We are also thankful to the referees for their invaluable comments.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Seyyed Mohammad Taghi Ayatollahi.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

ZB and PJ were responsible for the design, simulation, analysis and interpretation. SMTA supervised the study and interpreted the results. All authors read and approved the final manuscript.

Zahra Bagheri and Peyman Jafari contributed equally to this work.

Electronic supplementary material

12874_2011_579_MOESM1_ESM.TXT

Additional file 1: SAS simulation code for performing the three homogeneity tests of odds ratios in a certain case, in which K = 6, nik = 100, and . (TXT 16 KB)

Rights and permissions

Open Access This article is published under license to BioMed Central Ltd. This is an Open Access article is distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Bagheri, Z., Ayatollahi, S.M.T. & Jafari, P. Comparison of three tests of homogeneity of odds ratios in multicenter trials with unequal sample sizes within and among centers. BMC Med Res Methodol 11, 58 (2011). https://0-doi-org.brum.beds.ac.uk/10.1186/1471-2288-11-58

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/1471-2288-11-58

Keywords