A randomized controlled trial (RCT) has been considered as a gold standard that can provide the reliable evidence for the researchers, clinicians and policy makers^1,2). However, RCTs with methodological flaws can yield the skewed – usually exaggerated - results and eventually prevent the decision makers from implementing evidence-informed policy³⁾. Such flaws are referred to as biases, which are inherent in design, conduct, analysis, and reporting⁴⁾. Among many types of biases, reporting biases arise when the dissemination of research findings is influenced by the nature and direction of results⁵⁾. They usually arise with the intention to emphasize the positive results and diminish the negative results. It may occur both at the study level and at the outcome level^6,7). At the study level, studies with positive results are more likely to be published than those having negative results (publication bias)^6,8). At the outcome level, outcomes can be selectively reported in favor of positive results (outcome reporting bias).

One of the recommended safeguards against the selective reporting and secure the transparency of research practice is trial registration before the commencement of study⁹⁾. In 2005, the International Committee of Medical Journal Editors (ICMJE) launched the requirement for the trial registration before the onset of patient enrollment¹⁰⁾. After the announcement, the number of trial registration increased rapidly^11,12) and in April 2018, ClinicalTrials.gov, the largest and most known trial registry, contains protocols of more than 270,000 studies from over 200 countries¹³⁾. However, some clinical trials are registered retrospectively just before submission to a journal for publication¹⁴⁾ and moreover, a recent study reported that unregistered or retrospectively registered trials were more likely to overestimate the treatment effect than prospectively registered trials¹⁵⁾. Thus, the importance of adequate trial registration is increasingly recognized. Simultaneously, the inconsistency between the registered information and the published data raises concerns about the reliability of the research findings^16,17). Such inconsistency has been identified in several specialties including cardiology, oncology and dermatology^18-20).

Acupuncture has long been used in East-Asian countries, and is recently gaining popularity not only in East-Asian countries but also in Western countries^21-24). Accordingly, the number of acupuncture publications including RCT reports has increased over the past decades^25,26). However, reporting biases in acupuncture RCTs hamper the reliability of the evidence. For example, a previous systematic review of acupuncture trials revealed that East-Asian countries were more likely to produce positive findings than Western countries²⁷⁾. Moreover, another study raised a concern about the inconsistency of primary outcomes between registered protocol and published article in acupuncture RCTs²⁸⁾. Nevertheless, few studies comprehensively examined whether trial registration status was associated with the reporting bias. Therefore, this study aimed to investigate the association of trial registration status with presence of reporting bias including publication bias and outcome reporting bias in recently published acupuncture RCTs.

For this study, RCTs of acupuncture published in the previous 1 year were searched in PubMed. They were subject to analysis of association between trial registration with publication bias and outcome reporting bias.

1. Search strategy

An electronic search in PubMed for RCTs of acupuncture published from March 2016 to February 2017 was conducted. Search strategy was constructed by using key words such as ‘acupuncture’, ‘electroacupuncture’, ‘auricular acupuncture’ and ‘randomized controlled trials’.

2. Study selection

3. Data extraction

Two authors (S. Kim and J. Won) independently extracted the following information: journal name/specialty, study origin, sample size, study design, number of study arms, type of intervention(s), details of primary outcome including its adequacy and direction of the results, methodological quality including random sequence generation, allocation concealment, blinding of participants and outcome assessors and details of trial registration including registry database, registration number and primary outcomes. All extracted data were tabulated using Microsoft Excel (Microsoft Corporation, Redmond, WA, USA).

After the above information was collected from the included studies, it was aggregated into several categories. The journal specialty was dichotomized to complementary and alternative medicine (CAM) journals and non-CAM journals according to the subject area in Scopus^®. In addition, whether the trial registration was mandatory was also examined for the journals where the included articles were published. The study origin was determined by the country where the study was conducted or the country where the affiliation of the first author belonged to if the study was a multi-national one. The data on country were then aggregated into East Asia or non-East Asia to explore reporting biases may emerge in a different manner: a regional comparison was sought because a previous report showed East Asian countries were more likely to report positive findings than non-East Asian countries in acupuncture studies²⁷⁾. The sample size data were presented with median and interquartile range and types of intervention were classified into invasive acupuncture and non-invasive acupuncture.

The methodological quality was assessed based on the Cochrane Collaboration’s risk of bias assessment tool⁴⁾, to evaluate the appropriateness of the methodological processes including random sequence generation, allocation concealment, and blinding of participants and outcome assessment. In consideration of the virtual difficulty of practitioner blinding in acupuncture studies, blinding of only participants and outcome assessment was evaluated. Each included study was rated as having a low, high or unclear risk of bias for each item.

Primary outcomes were extracted when they were explicitly designated in the article. If none was specified, the outcome used in the sample size calculation was considered as primary outcome¹⁷⁾. If there was no description of primary outcomes or outcomes used in sample size calculation, the article was considered as “primary outcomes were not specified”. In addition, the adequacy of primary outcomes was assessed as the direction of result, i.e. positive (statistically significant) or negative, needed to be determined. According to the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement which endorsed that “having several primary outcomes is not recommended”³¹⁾, studies reporting multiple primary outcomes were considered inadequate. When a single primary outcome was identified in the sample size calculation, it was considered adequately reported. For the adequately reported primary outcomes, the direction of result was judged whether it was positive or negative.

The trial registration was identified by manually screening the trial registration number in the main text of the published articles and classified into 1) prospective registration 2) retrospective registration according to the registration date or 3) no registration. The prospective registration was designated 1) when the direct indication ‘prospectively registered’ was identified in trial registry or 2) if there was no direct indication in the registry, when the trials were registered before or within a month of the start date¹⁵⁾. If the detailed information of registration was modified after the start of the study, only the initially registered primary outcome was extracted.

4. Data analyses

1. Search results

A total of 580 articles were retrieved through electronic database search. After excluding 443 articles based on the title and abstract, 137 articles were included in the full-text review. Of these, 125 articles were finally included in the analysis (Fig. 1).

Fig. 1.

Flow diagram of study selection.

RCTs*, randomized controlled trials.

Of the 125 included articles, 40 studies (32.0%) prospectively registered the protocols, while 32 studies (25.6%) retrospectively registered and for 53 studies (42.4%), registered protocols were not identified (Table 1). Among the registered studies, 41 studies (56.9%) were registered in the Clinicaltrials. gov registry, while 16 studies (22.2%) in the Chinese Clinical Trial Registry (ChiCTR) (Table 2).

2. Publication bias

When the status of the trial registration was classified into registered vs. unregistered, unregistered trials were more likely to report positive results than registered ones (p=0.013). However, when the status was classified into prospectively registered vs. unregistered or retrospectively registered, the association was no longer significant (p=0.598). The association of trial registration status and other methodological factors was also analyzed. Whether studies were published in CAM journals or non-CAM journals was not associated with protocol registration status of registered vs. unregistered (p=0.381) and prospectively registered vs. unregistered or retrospectively registered (p=0.109). In addition, studies reporting as registered were more likely to have a larger sample size (registered vs. unregistered; median 80 vs. 53, p=0.007 and prospectively registered vs. unregistered or retrospectively registered; 81 vs. 60, p=0.013). On the other hand, no association was observed between the trial registration status and the study origin or the methodological factors including random sequence generation, allocation concealment and blinding of participants and outcome assessors (Table 3).

3. Outcome reporting bias

A total of 40 studies for which the protocols were prospectively registered were included in this analysis. The discrepancy between primary outcomes in the trial registration and the published article was identified in approximately a half of the studies (19 studies, 47.5%). The most common discrepancy was not reporting the registered primary outcomes as such in the published articles (16 out of 23, 69.6%). Of the 19 studies having the discrepancy between registered and published primary outcomes, 6 studies (31.6%) were judged that the discrepancy favored positive results, which might possibly imply outcome reporting bias (Table 4). For example, in a study attempting to explore synergistic effect of acupoint combination in lowering blood pressure, cerebral functional activity and blood pressure were registered as primary outcomes. In the published article, however, there were no description of specified primary outcomes (probably) due to no statistical significance in inter-group comparison.

1. Summary of main findings

In this study, the presence and characteristics of reporting bias and the association with the trial registration status were analyzed in the recently published acupuncture RCTs. Of the 125 acupuncture RCTs, 72 registered the protocols (57.6%) and of the registered trials, only 40 (55.6%) adequately did so. To identify whether the trial registration status was associated with publication bias, we examined the trial registration status and the direction of study results in 65 acupuncture RCTs adequately reporting the primary outcomes. We found that unregistered trials were more likely to report positive results than studies reporting as registered. The outcome reporting bias was identified that approximately a half of the prospectively registered studies showed the discrepancies between the registered and published primary outcomes and 31.6% of those favored positive results. The most common discrepancy was not to report the registered primary outcomes as such in the published articles.

2. Interpretation of the study findings

The proportion of the registered studies (57.6%) was found to be slightly lower than those observed in surgery (65.1%)³²⁾, critical care (66%)³³⁾, RCTs included in the Cochrane reviews (70%)¹⁵⁾, 3 medical areas including cardiology, rheumatology, and gastroenterology (72.4%)¹⁷⁾, or oncology (82%)²⁰⁾. Compared to other areas such as psychosomatic and behavioral health (20.6%)³⁴⁾ or physical therapy (29%)³⁵⁾, however, it was considered much higher. The proportion of the prospectively registered studies found in this study (32.0%) was largely comparable to those observed in other areas such as critical care (22.2%)³³⁾, psychiatry (33.1%)³⁶⁾, gastroenterology/ hepatology (37.3%), or 3 medical areas including cardiology, rheumatology, and gastroenterology (45.5%)¹⁷⁾. A higher rate of adequately registered trials in our study than the previous report of 19.3% is of note as it can be seen as an improvement over time of clinical trial practice in acupuncture field²⁸⁾. To sum up, the trial registration rate is improving but there is still a room for improvement regardless of the specialties.

In the analysis of publication bias, the status of trial registration was classified in two ways: registered vs. unregistered, and prospectively registered vs. unregistered or retrospectively registered. The former focuses more on appraising the attention of authors to register protocols, whereas the latter pays more attention to appraising the possibility of switching the primary outcomes in favor of statistically significant results. We found unregistered trials were more likely to report statistically significant results than registered trials but such discrepancy was no more significant under a prospectively registered vs. unregistered or retrospectively registered comparison. Given that trials retrospectively registered or unregistered tended to show larger treatment effect estimates than those prospectively registered¹⁵⁾, this finding is subject to further investigations.

In addition, we also found no difference in trial registration status between studies published in CAM journals and those published in non-CAM journals. It suggests that CAM journal editors and reviewers have monitored registration to some degree of their counterpart to secure the trial registration before the initiation of the trial. Moreover, the association between the sample size and the trial registration status was also identified, i.e. the higher registration rate among the larger trials, as reported in a previous study on cardiovascular trials¹⁸⁾. Perhaps it is because researchers of larger trials tend to pay more attention from the beginning stage, including the protocol registration since larger trials usually cost a lot more time, efforts, and expenses.

While it is known that East Asian countries are more likely to produce positive findings in acupuncture studies than other countries²⁷⁾, this study found no significant association between the study origin and the proportion of studies with positive results. This may be partly attributed to the increased awareness of reporting biases among the acupuncture researchers for the past 20 years.

Outcome reporting bias has been investigated in various fields including oncology, traditional Chinese Medicine, and acupuncture^17,20,28,37). The rate of inconsistency of primary outcomes between registered protocols and published articles ranged from 14 to 45% in previous studies^17,20,28,37), which is similar to that of our study (47.5%). Therefore, the outcome reporting bias is also not uncommon across the specialties. Since the presence of the outcome reporting bias can exaggerate the results of systematic reviews, or moreover, hamper to make the appropriate decision of clinicians and policy makers, the authors and reviewers should pay more attention to this phenomenon³⁸⁾.

3. Strengths and limitations of this study

In comparison to previous studies that focused on the individual type of reporting biases^27,28), this study comprehensively investigated the prevalence of reporting biases in published acupuncture RCTs, in terms of publication bias and outcome reporting bias and their association with the status of trial registration. We could confirm that reporting biases were prevalent in various forms which called for more attention from editors and researchers.

This study has several limitations. Firstly, articles written in Chinese were not included in the analyses. Considering that approximately 2,000 acupuncture RCTs were published in the past 20 years²⁶⁾, the reporting biases in Chinese articles should be taken into account and compared with articles written in English in future analyses. Secondly, we did not further search trial registries for the studies that did not provide the registration numbers in the published articles. While the CONSORT 2010 statement announced that “authors should provide the name of the register and the trial’s unique registration number”^31,39), not all trials provide registration information in their publications²⁸⁾. Thus, we may have misclassified some trials as unregistered simply because they did not report registration numbers in their publications. Thirdly, the publication bias was examined by simple vote counting whether the trial registration status was associated with positive vs. negative findings in consideration of a substantial clinical and statistical heterogeneity among the included trials⁹⁾. A comparison of treatment effect estimates between registered and unregistered trials by meta- epidemiologic approach would have informed more accurate treatment effect estimate.

4. Implications for future research

To evaluate generalizability of the present study findings, an extended research into unidentified acupuncture RCTs is needed. It needs to be confirmed whether Chinese language articles, or trials that did not state the registration number but might have actually registered in the trial registry, may have different impact on the association of trial registration status with presence of reporting biases. Given the findings in previous studies that reported the high prevalence of reporting biases in acupuncture studies published in China^27,40), it can be informative to explore the difference in the presence of reporting biases between Chinese articles and non-Chinese articles. Changes in the prevalence of reporting biases in acupuncture RCTs over time can also be investigated to encourage journal editors and researchers to meet the higher standards for registering trials and more transparent reporting of study outcomes.

The rates of trial registration and adequate registration were not very high in acupuncture RCTs. It was demonstrated that unregistered trials were more likely to report positive results than registered ones. For the prospectively registered studies, about a half of them showed the discrepancies between the registered and published primary outcomes and 40% of those favored the statistically significant results. Trials with nonsignificant outcomes tended to interpret statistically nonsignificant results as equivalence between groups. To sum up, reporting biases including publication bias and outcome reporting bias in the acupuncture RCTs are still prevalent and comparable to those in other specialties, calling for more attention from journal editors and researchers in this field.

This work was supported by the National Research Foundation (NRF) of Korea funded by the Korean government (Ministry of Science and ICT, grant Nos. 2017R1A2B4006407 and 2014R1A1A2055507). H.L. was supported by Kyung Hee University for her sabbatical year. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.