International Journal of Yoga

LETTER TO EDITOR
Year
: 2016  |  Volume : 9  |  Issue : 1  |  Page : 87--88

Unsubstantiated conclusions from improper statistical design and analysis of a randomized controlled trial


Brandon J George1, TaShauna U Goldsby2, Andrew W Brown2, Peng Li1, David B Allison3,  
1 Office of Energetics, School of Public Health, University of Alabama at Birmingham, Alabama, USA
2 Office of Energetics, School of Public Health, University of Alabama at Birmingham; Nutrition Obesity Research Center, University of Alabama at Birmingham, Alabama, USA
3 Office of Energetics, School of Public Health; Nutrition Obesity Research Center; Department of Biostatistics, University of Alabama at Birmingham, Birmingham, Alabama, USA

Correspondence Address:
Brandon J George
1700 University Blvd., Room 408, Birmingham, Alabama 35233
USA




How to cite this article:
George BJ, Goldsby TU, Brown AW, Li P, Allison DB. Unsubstantiated conclusions from improper statistical design and analysis of a randomized controlled trial.Int J Yoga 2016;9:87-88


How to cite this URL:
George BJ, Goldsby TU, Brown AW, Li P, Allison DB. Unsubstantiated conclusions from improper statistical design and analysis of a randomized controlled trial. Int J Yoga [serial online] 2016 [cited 2020 Sep 26 ];9:87-88
Available from: http://www.ijoy.org.in/text.asp?2016/9/1/87/171726


Full Text



Dear Sir,

The proper design and analysis of randomized controlled trials (RCTs) are essential for scientific progress identifying safe and efficacious interventions. We read the RCT recently reported by Jakhotia et al. with interest, which compared the effects of three forms of exercise and a control on health and fitness-related outcomes. Unfortunately, severe problems with the design and analysis of the study make the reported results highly questionable.

Specifically, the allocation of participants is not random. The randomization process generally consists of two steps: (1) Generating an unpredictable random sequence and (2) implementing the sequence in a way that conceals the treatment until subjects have been formally assigned to their groups. The protocol is described as assigning subjects "as they were recruited with n in group 1, n+1 in group 2,… and so on."[1] It is unclear if they put the blocks of individuals into groups or individuals themselves, but either method is not random if the order of enrollment is exclusively defining treatment allocation. With sequential enrollment, knowledge of the next participant assignment increases the risk of bias to include or exclude a potential participant based on demographic characteristics.[2] By violating both the principles of random allocation, the study reported cannot be properly referred to as an RCT.[2]

There are also concerns with how the statistical analysis was conducted. [Table 2] of Jakhotia et al. reports baseline characteristics and tests for differences among the four groups using a repeated measures analysis of variance; this is an inappropriate test because there should only be data from one time point to consider. Furthermore, we attempted to replicate the tests using the provided summary statistics and calculated incompatible results [Table 1]. These errors, including an impossible P > 1 (perhaps a typographical error, as P values are probabilities and must be between 0 and 1), raise further concerns because these calculated baseline imbalances were the basis for the choice of test in later analyses. We found similar errors in [Table 3] of Jakhotia et al., where the reported within-group mean difference and standard deviation, 95% confidence interval, test statistic, and P value were incommensurable with one another in several cases.{Table 1}

The objective of this study was to find differences in health-related outcomes between three exercise regimens and a control group, but the core of their conclusions focused on whether pairwise differences within the treatment groups were significantly different from baseline. This approach, where differences in nominal significance within-groups are interpreted as significant differences between the groups, is statistically invalid, producing false positives at a rate up to 87.5% for four groups.[3] That approach should never be used; instead we suggest performing post hoc exploratory pairwise comparisons with a multiple testing correction.

The title of this paper is also misleading; equivalency of treatments can only be properly determined through an equivalency trial,[4] which this study was not. Considering the compromised randomization, the irreproducible baseline statistics, the use of subsequent analytical strategies that depended on the faulty baseline statistics, and the inferences inappropriately based on within-group comparisons, we believe retraction of Jakhotia et al. is consistent with the recommendation of the International Committee of Medical Journal Editors: "Errors serious enough to invalidate a paper's results and conclusions may require retraction".[5]

Acknowledgment

Supported in part by NIH grants R25HL124208, T32DK062710, R25DK099080, and P30DK056336. The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization.

Financial support and sponsorship

Supported in part by NIH grants R25HL124208, T32DK062710, R25DK099080, and P30DK056336.

Conflicts of interest

There are no conflicts of interest.

References

1Jakhotia KA, Shimpi AP, Rairikar SA, Mhendale P, Hatekar R, Shyam A, et al. Suryanamaskar: An equivalent approach towards management of physical fitness in obese females. Int J Yoga 2015;8:27-36.
2Dettori J. The random allocation process: Two things you need to know. Evid Based Spine Care J 2010;1:7-9.
3Bland JM, Altman DG. Comparisons against baseline within randomised groups are often used and can be highly misleading. Trials 2011;12:264.
4Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ; CONSORT Group. Reporting of noninferiority and equivalence randomized trials: An extension of the CONSORT statement. JAMA 2006;295:1152-60.
5Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals [Internet]. International Committee of Medical Journal Editors. c2014. Available from: http://www.icmje.org/icmje-recommendations.pdf. [Last cited on 2015 Sep 08].