
To investigate the association of trial registration status with presence of reporting bias including publication bias and outcome reporting bias in recently published randomized controlled trials (RCTs) of acupuncture.
A PubMed search for RCTs of acupuncture published from March 2016 to February 2017 was conducted. Primary outcomes were identified and the direction of the results was judged as positive (i.e., statistically significant) or negative. The trial registration was identified by manually screening the trial registration number in the main text of the published article and classified into 1) prospective registration; 2) retrospective registration based on the registration date or; 3) no registration.
Of the 125 included RCTs, only 40 studies (32.0%) prospectively registered the study protocols. Among 65 RCTs that adequately reported the primary outcome, unregistered trials were more likely to report positive results than the registered ones (
Unregistered trials were more likely to report positive results and the discrepancies between the registered and published primary outcomes were detected in about a half of the prospectively registered studies, 42.1% of which tended to report positive findings. Journal editors and researchers in this field should be alerted to various reporting biases.
A randomized controlled trial (RCT) has been considered as a gold standard that can provide the reliable evidence for the researchers, clinicians and policy makers1,2). However, RCTs with methodological flaws can yield the skewed – usually exaggerated - results and eventually prevent the decision makers from implementing evidence-informed policy3). Such flaws are referred to as biases, which are inherent in design, conduct, analysis, and reporting4). Among many types of biases, reporting biases arise when the dissemination of research findings is influenced by the nature and direction of results5). They usually arise with the intention to emphasize the positive results and diminish the negative results. It may occur both at the study level and at the outcome level6,7). At the study level, studies with positive results are more likely to be published than those having negative results (publication bias)6,8). At the outcome level, outcomes can be selectively reported in favor of positive results (outcome reporting bias).
One of the recommended safeguards against the selective reporting and secure the transparency of research practice is trial registration before the commencement of study9). In 2005, the International Committee of Medical Journal Editors (ICMJE) launched the requirement for the trial registration before the onset of patient enrollment10). After the announcement, the number of trial registration increased rapidly11,12) and in April 2018, ClinicalTrials.gov, the largest and most known trial registry, contains protocols of more than 270,000 studies from over 200 countries13). However, some clinical trials are registered retrospectively just before submission to a journal for publication14) and moreover, a recent study reported that unregistered or retrospectively registered trials were more likely to overestimate the treatment effect than prospectively registered trials15). Thus, the importance of adequate trial registration is increasingly recognized. Simultaneously, the inconsistency between the registered information and the published data raises concerns about the reliability of the research findings16,17). Such inconsistency has been identified in several specialties including cardiology, oncology and dermatology18-20).
Acupuncture has long been used in East-Asian countries, and is recently gaining popularity not only in East-Asian countries but also in Western countries21-24). Accordingly, the number of acupuncture publications including RCT reports has increased over the past decades25,26). However, reporting biases in acupuncture RCTs hamper the reliability of the evidence. For example, a previous systematic review of acupuncture trials revealed that East-Asian countries were more likely to produce positive findings than Western countries27). Moreover, another study raised a concern about the inconsistency of primary outcomes between registered protocol and published article in acupuncture RCTs28). Nevertheless, few studies comprehensively examined whether trial registration status was associated with the reporting bias. Therefore, this study aimed to investigate the association of trial registration status with presence of reporting bias including publication bias and outcome reporting bias in recently published acupuncture RCTs.
For this study, RCTs of acupuncture published in the previous 1 year were searched in PubMed. They were subject to analysis of association between trial registration with publication bias and outcome reporting bias.
An electronic search in PubMed for RCTs of acupuncture published from March 2016 to February 2017 was conducted. Search strategy was constructed by using key words such as ‘acupuncture’, ‘electroacupuncture’, ‘auricular acupuncture’ and ‘randomized controlled trials’.
The extent of RCT was defined as a prospective study assessing the therapeutic, preventative, adverse, pharmacokinetic or physiological effects of 1 or more healthcare interventions in human participants who were allocated to study group using a random method29). Thus, non-randomized studies such as case reports, case-control studies, observational studies, narrative or systematic reviews, commentaries or letters were excluded.
The extent of acupuncture included not only invasive acupuncture needling but also non-invasive interventions applied to acupoint(s) such as acupressure, moxibustion and laser acupuncture, as defined in WHO’s publication30). Studies in which acupuncture treatment was used as a single or adjuvant therapy were included and when acupuncture treatment was tested in combination with other treatment(s), only the studies where the identical treatment applied in both intervention and control groups were considered. There was no limitation to comparators, thus, both active control such as standard treatment and inactive control such as no treatment, waiting list or sham control were included.
One researcher (S. Kim) searched the electronic database and screened the eligibility of studies based on the title and abstract. The remaining articles were independently read in full and assessed for eligibility by the two researchers (S. Kim and J. Won). Discrepancies were resolved by discussion with the corresponding author (H. Lee).
Two authors (S. Kim and J. Won) independently extracted the following information: journal name/specialty, study origin, sample size, study design, number of study arms, type of intervention(s), details of primary outcome including its adequacy and direction of the results, methodological quality including random sequence generation, allocation concealment, blinding of participants and outcome assessors and details of trial registration including registry database, registration number and primary outcomes. All extracted data were tabulated using Microsoft Excel (Microsoft Corporation, Redmond, WA, USA).
After the above information was collected from the included studies, it was aggregated into several categories. The journal specialty was dichotomized to complementary and alternative medicine (CAM) journals and non-CAM journals according to the subject area in Scopus®. In addition, whether the trial registration was mandatory was also examined for the journals where the included articles were published. The study origin was determined by the country where the study was conducted or the country where the affiliation of the first author belonged to if the study was a multi-national one. The data on country were then aggregated into East Asia or non-East Asia to explore reporting biases may emerge in a different manner: a regional comparison was sought because a previous report showed East Asian countries were more likely to report positive findings than non-East Asian countries in acupuncture studies27). The sample size data were presented with median and interquartile range and types of intervention were classified into invasive acupuncture and non-invasive acupuncture.
The methodological quality was assessed based on the Cochrane Collaboration’s risk of bias assessment tool4), to evaluate the appropriateness of the methodological processes including random sequence generation, allocation concealment, and blinding of participants and outcome assessment. In consideration of the virtual difficulty of practitioner blinding in acupuncture studies, blinding of only participants and outcome assessment was evaluated. Each included study was rated as having a low, high or unclear risk of bias for each item.
Primary outcomes were extracted when they were explicitly designated in the article. If none was specified, the outcome used in the sample size calculation was considered as primary outcome17). If there was no description of primary outcomes or outcomes used in sample size calculation, the article was considered as “primary outcomes were not specified”. In addition, the adequacy of primary outcomes was assessed as the direction of result, i.e. positive (statistically significant) or negative, needed to be determined. According to the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement which endorsed that “having several primary outcomes is not recommended”31), studies reporting multiple primary outcomes were considered inadequate. When a single primary outcome was identified in the sample size calculation, it was considered adequately reported. For the adequately reported primary outcomes, the direction of result was judged whether it was positive or negative.
The trial registration was identified by manually screening the trial registration number in the main text of the published articles and classified into 1) prospective registration 2) retrospective registration according to the registration date or 3) no registration. The prospective registration was designated 1) when the direct indication ‘prospectively registered’ was identified in trial registry or 2) if there was no direct indication in the registry, when the trials were registered before or within a month of the start date15). If the detailed information of registration was modified after the start of the study, only the initially registered primary outcome was extracted.
The most direct and convincing evidence of publication bias is from comparisons between unpublished and published results of studies that evaluate the same interventions9). However, since there is no registry providing the results of unpublished studies, it is not always feasible to compare the findings of published and unpublished studies. The indirect evidence can be sought by examining the proportion of positive results in the published literature is disproportionately high. In the present study, the association between the trial registration and the direction of primary outcome, i.e. positive or negative, was investigated only with the studies that adequately reported the primary outcome meeting the above mentioned criteria, since the judgement of the direction of result was considered valid only when the primary outcome was adequately established and reported. The association between the trial registration and the direction of primary outcome was analyzed under two scenarios of trial status; 1) ‘registered’ (both prospectively and retrospectively registered) vs. ‘unregistered’; and 2) ‘prospectively registered’ vs. ‘unregistered or retrospectively registered’. In addition, the association of trial registration with other factors including journal type, study origin, sample size and risks of bias including random sequence generation, allocation concealment, blinding of participants and blinding of outcome assessors was also analyzed.
Outcome reporting bias is defined as a selective reporting of outcome in order to emphasize positive results or diminish negative results6,17,29,31). To determine the presence of outcome reporting bias, the primary outcomes specified in the trial registration were compared with those reported in the published articles. The identification of the primary outcomes was performed only for the prospectively registered studies since it was deemed inappropriate to evaluate whether the primary outcome was changed according to the results in retrospectively registered studies17). If the primary outcomes in the registry and the published articles were discordant, the types of discrepancy was classified according to Mathieu’s classification17) as follows: 1) registered primary outcomes were not described as such in the published articles; 2) registered primary outcomes were omitted in the published articles; 3) primary outcomes in the publication were secondary outcomes in the registry; 4) new primary outcomes were introduced in the published articles; and/or 5) the timing of assessment differed. We also assessed whether the discrepancy favors the positive results. The discrepancy was judged as ‘favoring positive results’ when 1) the outcomes described as non-primary in the publication had negative results and/or 2) the outcomes changed from secondary to primary and/or newly introduced as primary in the publication had positive results. When it was impossible to determine the results of registered primary outcomes because of omission in the published articles, it was classified as ‘impossible to conclude’.
Extracted data of study characteristics, types of discordance between trial registry and published article, and discrepancy favoring positive results were reported in a descriptive manner. All the statistical analyses were conducted by using Statistical Package for Social Sciences (SPSS) software (Version 23.0, SPSS Inc.; Chicago, IL, USA). Association between trial registration status and proportion of the studies with positive results vs. negative results, journal specialty (CAM vs. non-CAM journal), mandatory trial registration (mandatory vs. not mandatory), study origin (East Asia vs. non-East Asia), primary outcome adequately specified vs. not adequately specified or not specified, and risk of bias (adequate vs. inadequate random sequence generation, allocation concealment, and participant/ outcome assessment blinding) was analyzed by using Chi-Square and/or Fisher’s Exact tests, with the significance level of
A total of 580 articles were retrieved through electronic database search. After excluding 443 articles based on the title and abstract, 137 articles were included in the full-text review. Of these, 125 articles were finally included in the analysis (Fig. 1).
Of the 125 included articles, 40 studies (32.0%) prospectively registered the protocols, while 32 studies (25.6%) retrospectively registered and for 53 studies (42.4%), registered protocols were not identified (Table 1). Among the registered studies, 41 studies (56.9%) were registered in the Clinicaltrials. gov registry, while 16 studies (22.2%) in the Chinese Clinical Trial Registry (ChiCTR) (Table 2).
When the status of the trial registration was classified into registered vs. unregistered, unregistered trials were more likely to report positive results than registered ones (
A total of 40 studies for which the protocols were prospectively registered were included in this analysis. The discrepancy between primary outcomes in the trial registration and the published article was identified in approximately a half of the studies (19 studies, 47.5%). The most common discrepancy was not reporting the registered primary outcomes as such in the published articles (16 out of 23, 69.6%). Of the 19 studies having the discrepancy between registered and published primary outcomes, 6 studies (31.6%) were judged that the discrepancy favored positive results, which might possibly imply outcome reporting bias (Table 4). For example, in a study attempting to explore synergistic effect of acupoint combination in lowering blood pressure, cerebral functional activity and blood pressure were registered as primary outcomes. In the published article, however, there were no description of specified primary outcomes (probably) due to no statistical significance in inter-group comparison.
In this study, the presence and characteristics of reporting bias and the association with the trial registration status were analyzed in the recently published acupuncture RCTs. Of the 125 acupuncture RCTs, 72 registered the protocols (57.6%) and of the registered trials, only 40 (55.6%) adequately did so. To identify whether the trial registration status was associated with publication bias, we examined the trial registration status and the direction of study results in 65 acupuncture RCTs adequately reporting the primary outcomes. We found that unregistered trials were more likely to report positive results than studies reporting as registered. The outcome reporting bias was identified that approximately a half of the prospectively registered studies showed the discrepancies between the registered and published primary outcomes and 31.6% of those favored positive results. The most common discrepancy was not to report the registered primary outcomes as such in the published articles.
The proportion of the registered studies (57.6%) was found to be slightly lower than those observed in surgery (65.1%)32), critical care (66%)33), RCTs included in the Cochrane reviews (70%)15), 3 medical areas including cardiology, rheumatology, and gastroenterology (72.4%)17), or oncology (82%)20). Compared to other areas such as psychosomatic and behavioral health (20.6%)34) or physical therapy (29%)35), however, it was considered much higher. The proportion of the prospectively registered studies found in this study (32.0%) was largely comparable to those observed in other areas such as critical care (22.2%)33), psychiatry (33.1%)36), gastroenterology/ hepatology (37.3%), or 3 medical areas including cardiology, rheumatology, and gastroenterology (45.5%)17). A higher rate of adequately registered trials in our study than the previous report of 19.3% is of note as it can be seen as an improvement over time of clinical trial practice in acupuncture field28). To sum up, the trial registration rate is improving but there is still a room for improvement regardless of the specialties.
In the analysis of publication bias, the status of trial registration was classified in two ways: registered vs. unregistered, and prospectively registered vs. unregistered or retrospectively registered. The former focuses more on appraising the attention of authors to register protocols, whereas the latter pays more attention to appraising the possibility of switching the primary outcomes in favor of statistically significant results. We found unregistered trials were more likely to report statistically significant results than registered trials but such discrepancy was no more significant under a prospectively registered vs. unregistered or retrospectively registered comparison. Given that trials retrospectively registered or unregistered tended to show larger treatment effect estimates than those prospectively registered15), this finding is subject to further investigations.
In addition, we also found no difference in trial registration status between studies published in CAM journals and those published in non-CAM journals. It suggests that CAM journal editors and reviewers have monitored registration to some degree of their counterpart to secure the trial registration before the initiation of the trial. Moreover, the association between the sample size and the trial registration status was also identified, i.e. the higher registration rate among the larger trials, as reported in a previous study on cardiovascular trials18). Perhaps it is because researchers of larger trials tend to pay more attention from the beginning stage, including the protocol registration since larger trials usually cost a lot more time, efforts, and expenses.
While it is known that East Asian countries are more likely to produce positive findings in acupuncture studies than other countries27), this study found no significant association between the study origin and the proportion of studies with positive results. This may be partly attributed to the increased awareness of reporting biases among the acupuncture researchers for the past 20 years.
Outcome reporting bias has been investigated in various fields including oncology, traditional Chinese Medicine, and acupuncture17,20,28,37). The rate of inconsistency of primary outcomes between registered protocols and published articles ranged from 14 to 45% in previous studies17,20,28,37), which is similar to that of our study (47.5%). Therefore, the outcome reporting bias is also not uncommon across the specialties. Since the presence of the outcome reporting bias can exaggerate the results of systematic reviews, or moreover, hamper to make the appropriate decision of clinicians and policy makers, the authors and reviewers should pay more attention to this phenomenon38).
In comparison to previous studies that focused on the individual type of reporting biases27,28), this study comprehensively investigated the prevalence of reporting biases in published acupuncture RCTs, in terms of publication bias and outcome reporting bias and their association with the status of trial registration. We could confirm that reporting biases were prevalent in various forms which called for more attention from editors and researchers.
This study has several limitations. Firstly, articles written in Chinese were not included in the analyses. Considering that approximately 2,000 acupuncture RCTs were published in the past 20 years26), the reporting biases in Chinese articles should be taken into account and compared with articles written in English in future analyses. Secondly, we did not further search trial registries for the studies that did not provide the registration numbers in the published articles. While the CONSORT 2010 statement announced that “authors should provide the name of the register and the trial’s unique registration number”31,39), not all trials provide registration information in their publications28). Thus, we may have misclassified some trials as unregistered simply because they did not report registration numbers in their publications. Thirdly, the publication bias was examined by simple vote counting whether the trial registration status was associated with positive vs. negative findings in consideration of a substantial clinical and statistical heterogeneity among the included trials9). A comparison of treatment effect estimates between registered and unregistered trials by meta- epidemiologic approach would have informed more accurate treatment effect estimate.
To evaluate generalizability of the present study findings, an extended research into unidentified acupuncture RCTs is needed. It needs to be confirmed whether Chinese language articles, or trials that did not state the registration number but might have actually registered in the trial registry, may have different impact on the association of trial registration status with presence of reporting biases. Given the findings in previous studies that reported the high prevalence of reporting biases in acupuncture studies published in China27,40), it can be informative to explore the difference in the presence of reporting biases between Chinese articles and non-Chinese articles. Changes in the prevalence of reporting biases in acupuncture RCTs over time can also be investigated to encourage journal editors and researchers to meet the higher standards for registering trials and more transparent reporting of study outcomes.
The rates of trial registration and adequate registration were not very high in acupuncture RCTs. It was demonstrated that unregistered trials were more likely to report positive results than registered ones. For the prospectively registered studies, about a half of them showed the discrepancies between the registered and published primary outcomes and 40% of those favored the statistically significant results. Trials with nonsignificant outcomes tended to interpret statistically nonsignificant results as equivalence between groups. To sum up, reporting biases including publication bias and outcome reporting bias in the acupuncture RCTs are still prevalent and comparable to those in other specialties, calling for more attention from journal editors and researchers in this field.
This work was supported by the National Research Foundation (NRF) of Korea funded by the Korean government (Ministry of Science and ICT, grant Nos. 2017R1A2B4006407 and 2014R1A1A2055507). H.L. was supported by Kyung Hee University for her sabbatical year. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
![]() |
![]() |