Search for


search for




 

Association of Trial Registration with Reporting Biases in Randomized Controlled Trials of Acupuncture
침 무작위 대조 임상 시험에서 보고 비뚤림과 프로토콜 등록 여부의 관련성 연구
Korean J Acupunct 2018;35:70-81
Published online June 27, 2018;  https://doi.org/10.14406/acu.2018.009
© 2018 Society for Meridian and Acupoint.

Seoyeon Kim1,2 , Jiyoon Won1,2 , Hi-Joon Park1,2 , and Hyangsook Lee1,2
김서연1,2, 원지윤1,2, 박히준1,2, 이향숙1,2

1Department of Korean Medical Science, Graduate School, Kyung Hee University,
2Acupuncture & Meridian Science Research Centre, College of Korean Medicine, Kyung Hee University
1경희대학교 대학원 기초한의과학과,
2경희대학교 침구경락융합연구센터
Correspondence to: Hyangsook Lee Acupuncture and Meridian Science Research Centre, College of Korean Medicine, Kyung Hee University, 26 Kyung Hee Dae-ro, Dongdaemun-gu, Seoul 02447, Korea Tel: +82-2-961-0703, Fax: +82-2-963-2175, E-mail: erc633@khu.ac.kr
Received April 13, 2018; Revised May 12, 2018; Accepted May 21, 2018.
This is an open access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract

Objectives:

To investigate the association of trial registration status with presence of reporting bias including publication bias and outcome reporting bias in recently published randomized controlled trials (RCTs) of acupuncture.

Methods:

A PubMed search for RCTs of acupuncture published from March 2016 to February 2017 was conducted. Primary outcomes were identified and the direction of the results was judged as positive (i.e., statistically significant) or negative. The trial registration was identified by manually screening the trial registration number in the main text of the published article and classified into 1) prospective registration; 2) retrospective registration based on the registration date or; 3) no registration.

Results:

Of the 125 included RCTs, only 40 studies (32.0%) prospectively registered the study protocols. Among 65 RCTs that adequately reported the primary outcome, unregistered trials were more likely to report positive results than the registered ones (p=0.013). Of the 40 prospectively registered studies, 19 trials (47.5%) had the discrepancies between the registered and published primary outcomes and furthermore, 40% of them reported the positive findings.

Conclusions:

Unregistered trials were more likely to report positive results and the discrepancies between the registered and published primary outcomes were detected in about a half of the prospectively registered studies, 42.1% of which tended to report positive findings. Journal editors and researchers in this field should be alerted to various reporting biases.

Keywords: reporting bias, publication bias, trial registration, outcome reporting bias, acupuncture
Introduction

A randomized controlled trial (RCT) has been considered as a gold standard that can provide the reliable evidence for the researchers, clinicians and policy makers1,2). However, RCTs with methodological flaws can yield the skewed – usually exaggerated - results and eventually prevent the decision makers from implementing evidence-informed policy3). Such flaws are referred to as biases, which are inherent in design, conduct, analysis, and reporting4). Among many types of biases, reporting biases arise when the dissemination of research findings is influenced by the nature and direction of results5). They usually arise with the intention to emphasize the positive results and diminish the negative results. It may occur both at the study level and at the outcome level6,7). At the study level, studies with positive results are more likely to be published than those having negative results (publication bias)6,8). At the outcome level, outcomes can be selectively reported in favor of positive results (outcome reporting bias).

One of the recommended safeguards against the selective reporting and secure the transparency of research practice is trial registration before the commencement of study9). In 2005, the International Committee of Medical Journal Editors (ICMJE) launched the requirement for the trial registration before the onset of patient enrollment10). After the announcement, the number of trial registration increased rapidly11,12) and in April 2018, ClinicalTrials.gov, the largest and most known trial registry, contains protocols of more than 270,000 studies from over 200 countries13). However, some clinical trials are registered retrospectively just before submission to a journal for publication14) and moreover, a recent study reported that unregistered or retrospectively registered trials were more likely to overestimate the treatment effect than prospectively registered trials15). Thus, the importance of adequate trial registration is increasingly recognized. Simultaneously, the inconsistency between the registered information and the published data raises concerns about the reliability of the research findings16,17). Such inconsistency has been identified in several specialties including cardiology, oncology and dermatology18-20).

Acupuncture has long been used in East-Asian countries, and is recently gaining popularity not only in East-Asian countries but also in Western countries21-24). Accordingly, the number of acupuncture publications including RCT reports has increased over the past decades25,26). However, reporting biases in acupuncture RCTs hamper the reliability of the evidence. For example, a previous systematic review of acupuncture trials revealed that East-Asian countries were more likely to produce positive findings than Western countries27). Moreover, another study raised a concern about the inconsistency of primary outcomes between registered protocol and published article in acupuncture RCTs28). Nevertheless, few studies comprehensively examined whether trial registration status was associated with the reporting bias. Therefore, this study aimed to investigate the association of trial registration status with presence of reporting bias including publication bias and outcome reporting bias in recently published acupuncture RCTs.

Materials and Methods

For this study, RCTs of acupuncture published in the previous 1 year were searched in PubMed. They were subject to analysis of association between trial registration with publication bias and outcome reporting bias.

1. Search strategy

An electronic search in PubMed for RCTs of acupuncture published from March 2016 to February 2017 was conducted. Search strategy was constructed by using key words such as ‘acupuncture’, ‘electroacupuncture’, ‘auricular acupuncture’ and ‘randomized controlled trials’.

2. Study selection

1) Study design

The extent of RCT was defined as a prospective study assessing the therapeutic, preventative, adverse, pharmacokinetic or physiological effects of 1 or more healthcare interventions in human participants who were allocated to study group using a random method29). Thus, non-randomized studies such as case reports, case-control studies, observational studies, narrative or systematic reviews, commentaries or letters were excluded.

2) Intervention

The extent of acupuncture included not only invasive acupuncture needling but also non-invasive interventions applied to acupoint(s) such as acupressure, moxibustion and laser acupuncture, as defined in WHO’s publication30). Studies in which acupuncture treatment was used as a single or adjuvant therapy were included and when acupuncture treatment was tested in combination with other treatment(s), only the studies where the identical treatment applied in both intervention and control groups were considered. There was no limitation to comparators, thus, both active control such as standard treatment and inactive control such as no treatment, waiting list or sham control were included.

3) Procedure

One researcher (S. Kim) searched the electronic database and screened the eligibility of studies based on the title and abstract. The remaining articles were independently read in full and assessed for eligibility by the two researchers (S. Kim and J. Won). Discrepancies were resolved by discussion with the corresponding author (H. Lee).

3. Data extraction

Two authors (S. Kim and J. Won) independently extracted the following information: journal name/specialty, study origin, sample size, study design, number of study arms, type of intervention(s), details of primary outcome including its adequacy and direction of the results, methodological quality including random sequence generation, allocation concealment, blinding of participants and outcome assessors and details of trial registration including registry database, registration number and primary outcomes. All extracted data were tabulated using Microsoft Excel (Microsoft Corporation, Redmond, WA, USA).

After the above information was collected from the included studies, it was aggregated into several categories. The journal specialty was dichotomized to complementary and alternative medicine (CAM) journals and non-CAM journals according to the subject area in Scopus®. In addition, whether the trial registration was mandatory was also examined for the journals where the included articles were published. The study origin was determined by the country where the study was conducted or the country where the affiliation of the first author belonged to if the study was a multi-national one. The data on country were then aggregated into East Asia or non-East Asia to explore reporting biases may emerge in a different manner: a regional comparison was sought because a previous report showed East Asian countries were more likely to report positive findings than non-East Asian countries in acupuncture studies27). The sample size data were presented with median and interquartile range and types of intervention were classified into invasive acupuncture and non-invasive acupuncture.

The methodological quality was assessed based on the Cochrane Collaboration’s risk of bias assessment tool4), to evaluate the appropriateness of the methodological processes including random sequence generation, allocation concealment, and blinding of participants and outcome assessment. In consideration of the virtual difficulty of practitioner blinding in acupuncture studies, blinding of only participants and outcome assessment was evaluated. Each included study was rated as having a low, high or unclear risk of bias for each item.

Primary outcomes were extracted when they were explicitly designated in the article. If none was specified, the outcome used in the sample size calculation was considered as primary outcome17). If there was no description of primary outcomes or outcomes used in sample size calculation, the article was considered as “primary outcomes were not specified”. In addition, the adequacy of primary outcomes was assessed as the direction of result, i.e. positive (statistically significant) or negative, needed to be determined. According to the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement which endorsed that “having several primary outcomes is not recommended”31), studies reporting multiple primary outcomes were considered inadequate. When a single primary outcome was identified in the sample size calculation, it was considered adequately reported. For the adequately reported primary outcomes, the direction of result was judged whether it was positive or negative.

The trial registration was identified by manually screening the trial registration number in the main text of the published articles and classified into 1) prospective registration 2) retrospective registration according to the registration date or 3) no registration. The prospective registration was designated 1) when the direct indication ‘prospectively registered’ was identified in trial registry or 2) if there was no direct indication in the registry, when the trials were registered before or within a month of the start date15). If the detailed information of registration was modified after the start of the study, only the initially registered primary outcome was extracted.

4. Data analyses

1) Publication bias

The most direct and convincing evidence of publication bias is from comparisons between unpublished and published results of studies that evaluate the same interventions9). However, since there is no registry providing the results of unpublished studies, it is not always feasible to compare the findings of published and unpublished studies. The indirect evidence can be sought by examining the proportion of positive results in the published literature is disproportionately high. In the present study, the association between the trial registration and the direction of primary outcome, i.e. positive or negative, was investigated only with the studies that adequately reported the primary outcome meeting the above mentioned criteria, since the judgement of the direction of result was considered valid only when the primary outcome was adequately established and reported. The association between the trial registration and the direction of primary outcome was analyzed under two scenarios of trial status; 1) ‘registered’ (both prospectively and retrospectively registered) vs. ‘unregistered’; and 2) ‘prospectively registered’ vs. ‘unregistered or retrospectively registered’. In addition, the association of trial registration with other factors including journal type, study origin, sample size and risks of bias including random sequence generation, allocation concealment, blinding of participants and blinding of outcome assessors was also analyzed.

2) Outcome reporting bias

Outcome reporting bias is defined as a selective reporting of outcome in order to emphasize positive results or diminish negative results6,17,29,31). To determine the presence of outcome reporting bias, the primary outcomes specified in the trial registration were compared with those reported in the published articles. The identification of the primary outcomes was performed only for the prospectively registered studies since it was deemed inappropriate to evaluate whether the primary outcome was changed according to the results in retrospectively registered studies17). If the primary outcomes in the registry and the published articles were discordant, the types of discrepancy was classified according to Mathieu’s classification17) as follows: 1) registered primary outcomes were not described as such in the published articles; 2) registered primary outcomes were omitted in the published articles; 3) primary outcomes in the publication were secondary outcomes in the registry; 4) new primary outcomes were introduced in the published articles; and/or 5) the timing of assessment differed. We also assessed whether the discrepancy favors the positive results. The discrepancy was judged as ‘favoring positive results’ when 1) the outcomes described as non-primary in the publication had negative results and/or 2) the outcomes changed from secondary to primary and/or newly introduced as primary in the publication had positive results. When it was impossible to determine the results of registered primary outcomes because of omission in the published articles, it was classified as ‘impossible to conclude’.

3) Statistical analysis

Extracted data of study characteristics, types of discordance between trial registry and published article, and discrepancy favoring positive results were reported in a descriptive manner. All the statistical analyses were conducted by using Statistical Package for Social Sciences (SPSS) software (Version 23.0, SPSS Inc.; Chicago, IL, USA). Association between trial registration status and proportion of the studies with positive results vs. negative results, journal specialty (CAM vs. non-CAM journal), mandatory trial registration (mandatory vs. not mandatory), study origin (East Asia vs. non-East Asia), primary outcome adequately specified vs. not adequately specified or not specified, and risk of bias (adequate vs. inadequate random sequence generation, allocation concealment, and participant/ outcome assessment blinding) was analyzed by using Chi-Square and/or Fisher’s Exact tests, with the significance level of p＜0.05. In addition, median sample size was compared between registered and unregistered studies, and between prospectively registered and unregistered or retrospectively registered studies using Mann-Whitney test18).

Results

1. Search results

A total of 580 articles were retrieved through electronic database search. After excluding 443 articles based on the title and abstract, 137 articles were included in the full-text review. Of these, 125 articles were finally included in the analysis (Fig. 1).

Fig. 1.

Flow diagram of study selection.

RCTs*, randomized controlled trials.



Of the 125 included articles, 40 studies (32.0%) prospectively registered the protocols, while 32 studies (25.6%) retrospectively registered and for 53 studies (42.4%), registered protocols were not identified (Table 1). Among the registered studies, 41 studies (56.9%) were registered in the Clinicaltrials. gov registry, while 16 studies (22.2%) in the Chinese Clinical Trial Registry (ChiCTR) (Table 2).

Characteristics and Risk of Bias of the Included Studies (n=125)

Number (%) of articles
Prospectively registered 40 (32.0) Retrospectively registered 32 (25.6) Unregistered 53 (42.4) Total 125 (100.0)
Journal specialty
 CAM* Journal 11 (8.8) 12 (9.6) 27 (21.6) 50 (40.0)
 Non-CAM Journal 29 (23.2) 20 (16.0) 26 (20.8) 75 (60.0)
Mandatory trial registration
 Mandatory 29 (23.2) 16 (12.8) 13 (10.4) 58 (46.4)
 Not mandatory 11 (8.8) 16 (12.8) 40 (32.0) 67 (53.6)
Study origin
 East Asia 19 (15.2) 15 (12.0) 28 (22.4) 62 (49.6)
 Non-East Asia 21 (16.8) 17 (13.6) 25 (20.0) 63 (50.4)
Study design
 Parallel design 38 (30.1) 30 (24.0) 48 (38.4) 116 (92.8)
 Cross-over design 1 (0.8) 1 (0.8) 5 (4.0) 7 (5.6)
 Factorial design 1 (0.8) 1 (0.8) 0 (0.0) 2 (1.6)
Sample size
 Median (IQR†) 88 (50∼166.5) 69 (44∼151.5) 56 (40∼67) 62 (43∼124)
Number of study arms
 Two 31 (24.8) 20 (16.0) 36 (28.8) 87 (69.6)
 Three 4 (3.2) 10 (8.0) 13 (10.4) 27 (21.6)
 ≥Four 5 (4.0) 2 (1.6) 4 (3.2) 11 (8.8)
Types of acupuncture
 Invasive acupuncture 36 (28.8) 28 (22.4) 42 (33.6) 106 (84.8)
 Non-invasive acupuncture 4 (3.2) 4 (3.2) 11 (8.8) 19 (15.2)
Primary outcome
 Adequately specified 29 (23.2) 19 (15.2) 17 (13.6) 65 (52.0)
 Positive results 19 (15.2) 10 (8.0) 16 (12.8) 50 (40.0)
 Negative results 10 (8.0) 9 (7.2) 1 (0.8) 20 (16.0)
 Inadequately specified 2 (1.6) 9 (7.2) 6 (4.8) 17 (13.6)
 Not specified 9 (7.2) 4 (3.2) 30 (24.0) 43 (34.4)
Random sequence generation
 Low risk 25 (20.0) 23 (18.4) 27 (21.6) 75 (60.0)
 Unclear risk 15 (12.0) 9 (7.2) 24 (19.2) 48 (38.4)
 High risk 0 (0.0) 0 (0.0) 2 (1.6) 2 (1.6)
Allocation concealment
 Low risk 21 (16.8) 14 (11.2) 12 (9.6) 47 (37.6)
 Unclear risk 19 (15.2) 18 (14.4) 41 (32.8) 78 (62.4)
 High risk 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0)
Blinding of participants
 Low risk 20 (16.0) 17 (13.6) 24 (19.2) 61 (48.8)
 Unclear risk 0 (0.0) 0 (0.0) 1 (0.8) 1 (0.8)
 High risk 20 (16.0) 15 (12.0) 28 (22.4) 63 (50.4)
Blinding of outcome assessors
 Low risk 17 (13.6) 13 (10.4) 17 (13.6) 47 (37.6)
 Unclear risk 17 (13.6) 15 (12.0) 34 (27.2) 66 (52.8)
 High risk 6 (4.8) 4 (3.2) 2 (1.6) 12 (9.6)

CAM : complementary and alternative medicine,

IQR: interquartile range.

Information of the Registries Where the Included Trials Were Registered

Number (%) of articles
Prospectively registered 40 (55.6) Retrospectively registered 32 (44.4) Total 72 (100.0)
Clinicaltrials.gov 26 (36.1) 15 (20.8) 41 (56.9)
Chinese Clinical Trial Registry (ChiCTR) 9 (12.5) 7 (9.7) 16 (22.2)
Clinical Research Information Service (CRiS), Republic of Korea - 1 (1.4) 1 (1.4)
International Standard Randomised Controlled Trial Number Registry (ISRCTN) 1 (1.4) 3 (4.2) 4 (5.6)
Brazilian Clinical Trials Registry (ReBec) 1 (1.4) - 1 (1.4)
Iranian Registry of Clinical Trials (IRCT) 1 (1.4) 3 (4.2) 4 (5.6)
German Clinical Trials Register (DRKS) 1 (1.4) - 1 (1.4)
Australian New Zealand Clinical Trials Registry (ANZCTR) - 3 (4.2) 3 (4.2)


2. Publication bias

When the status of the trial registration was classified into registered vs. unregistered, unregistered trials were more likely to report positive results than registered ones (p=0.013). However, when the status was classified into prospectively registered vs. unregistered or retrospectively registered, the association was no longer significant (p=0.598). The association of trial registration status and other methodological factors was also analyzed. Whether studies were published in CAM journals or non-CAM journals was not associated with protocol registration status of registered vs. unregistered (p=0.381) and prospectively registered vs. unregistered or retrospectively registered (p=0.109). In addition, studies reporting as registered were more likely to have a larger sample size (registered vs. unregistered; median 80 vs. 53, p=0.007 and prospectively registered vs. unregistered or retrospectively registered; 81 vs. 60, p=0.013). On the other hand, no association was observed between the trial registration status and the study origin or the methodological factors including random sequence generation, allocation concealment and blinding of participants and outcome assessors (Table 3).

Association between the Primary Outcome Results and Methodological Factors in the Articles with Adequately Specified Primary Outcomes (n=65)

Registered* vs. Unregistered Prospectively registered vs. Unregistered or retrospectively registered†
Number (%‡) of articles p value§ Number (%) of articles p value
Registered 48 (73.8) Unregistered 17 (26.2) Prospectively registered 29 (44.6) Unregistered or retrospectively registered 36 (55.4)
Journal specialty 0.381 0.109
 CAM Journal 14 (29.2) 7 (41.2) 6 (20.7) 15 (41.7)
 Non-CAM Journal 34 (70.8) 10 (58.8) 23 (79.3) 21 (58.3)
Study origin 1.000 0.618
 East Asia 22 (45.8) 8 (47.1) 12 (41.4) 18 (50.0)
 Non-East Asia 26 (54.2) 9 (52.9) 17 (58.6) 18 (50.0)
Sample size
 Median (IQR∥) 80 (50∼163.5) 53 (41.5∼72.5) 0.007§ 81 (56.5∼185.5) 60 (42.5∼101) 0.013§
Direction of primary outcome 0.013§ 0.598
 Positive 29 (60.4) 16 (94.1) 19 (65.5) 26 (72.2)
 Negative 19 (39.6) 1 (5.9) 10 (34.5) 10 (27.8)
Random sequence generation 1.000 0.212
 Low risk 34 (70.8) 12 (70.6) 20 (69.0) 26 (72.2)
 Unclear or high risk 14 (29.2) 5 (29.4) 9 (31.0) 10 (27.8)
Allocation concealment 0.571 0.802
 Low risk 28 (58.3) 8 (47.1) 17 (58.6) 19 (52.8)
 Unclear or high risk 20 (41.7) 9 (52.9) 12 (41.4) 17 (47.2)
Blinding of participants 0.408 0.620
 Low risk 22 (45.8) 10 (58.8) 13 (44.8) 19 (52.8)
 Unclear or high risk 26 (54.2) 7 (41.2) 16 (55.2) 17 (47.2)
Blinding of outcome assessors 0.582 0.800
 Low risk 21 (43.8) 6 (35.3) 13 (44.8) 14 (38.9)
 Unclear or high risk 27 (56.3) 11 (64.7) 16 (55.2) 22 (61.1)

‘Registered’ includes both prospectively and retrospectively registered studies.

‘Unregistered or retrospectively registered’ includes both retrospectively registered studies and studies which were not registered.

The percentage is represented as the column percentage.

Pearson Chi-Square or Fisher’s Exact tests were used for the analysis and p-value＜0.05 was considered significant.

IQR: interquartile range.

3. Outcome reporting bias

A total of 40 studies for which the protocols were prospectively registered were included in this analysis. The discrepancy between primary outcomes in the trial registration and the published article was identified in approximately a half of the studies (19 studies, 47.5%). The most common discrepancy was not reporting the registered primary outcomes as such in the published articles (16 out of 23, 69.6%). Of the 19 studies having the discrepancy between registered and published primary outcomes, 6 studies (31.6%) were judged that the discrepancy favored positive results, which might possibly imply outcome reporting bias (Table 4). For example, in a study attempting to explore synergistic effect of acupoint combination in lowering blood pressure, cerebral functional activity and blood pressure were registered as primary outcomes. In the published article, however, there were no description of specified primary outcomes (probably) due to no statistical significance in inter-group comparison.

Concordance of Primary Outcomes in Trial Registration and in the Published Articles (n=40)*

Number (%) of articles (n=40)
Concordant 21 (52.5)
Discordant 19 (47.5)
Types of discrepancies† 23
 Registered primary outcomes were not described as such in the published articles 16 (69.6)
 Registered primary outcomes were omitted in the published articles 1 (4.3)
 Published primary outcomes were registered as secondary outcomes in the registry 2 (8.7)
 New primary outcomes were introduced in the published articles 2 (8.7)
 The timing of assessment differed 2 (8.7)
Discrepancies in primary outcomes favoring statistically significant results
 Yes 6 (31.6)
 No 8 (42.1)
 Impossible to conclude‡ 5 (26.3)

The concordance of primary outcome was assessed for the 44 prospectively registered studies.

Four studies involved two types of discrepancies.

Two studies included both significant and nonsignificant results, two studies included registered primary outcomes that were omitted in the published articles, and the other one was an exploratory pilot study, thus primary outcomes were not specified in the published articles.
Discussion

1. Summary of main findings

In this study, the presence and characteristics of reporting bias and the association with the trial registration status were analyzed in the recently published acupuncture RCTs. Of the 125 acupuncture RCTs, 72 registered the protocols (57.6%) and of the registered trials, only 40 (55.6%) adequately did so. To identify whether the trial registration status was associated with publication bias, we examined the trial registration status and the direction of study results in 65 acupuncture RCTs adequately reporting the primary outcomes. We found that unregistered trials were more likely to report positive results than studies reporting as registered. The outcome reporting bias was identified that approximately a half of the prospectively registered studies showed the discrepancies between the registered and published primary outcomes and 31.6% of those favored positive results. The most common discrepancy was not to report the registered primary outcomes as such in the published articles.

2. Interpretation of the study findings

The proportion of the registered studies (57.6%) was found to be slightly lower than those observed in surgery (65.1%)32), critical care (66%)33), RCTs included in the Cochrane reviews (70%)15), 3 medical areas including cardiology, rheumatology, and gastroenterology (72.4%)17), or oncology (82%)20). Compared to other areas such as psychosomatic and behavioral health (20.6%)34) or physical therapy (29%)35), however, it was considered much higher. The proportion of the prospectively registered studies found in this study (32.0%) was largely comparable to those observed in other areas such as critical care (22.2%)33), psychiatry (33.1%)36), gastroenterology/ hepatology (37.3%), or 3 medical areas including cardiology, rheumatology, and gastroenterology (45.5%)17). A higher rate of adequately registered trials in our study than the previous report of 19.3% is of note as it can be seen as an improvement over time of clinical trial practice in acupuncture field28). To sum up, the trial registration rate is improving but there is still a room for improvement regardless of the specialties.

In the analysis of publication bias, the status of trial registration was classified in two ways: registered vs. unregistered, and prospectively registered vs. unregistered or retrospectively registered. The former focuses more on appraising the attention of authors to register protocols, whereas the latter pays more attention to appraising the possibility of switching the primary outcomes in favor of statistically significant results. We found unregistered trials were more likely to report statistically significant results than registered trials but such discrepancy was no more significant under a prospectively registered vs. unregistered or retrospectively registered comparison. Given that trials retrospectively registered or unregistered tended to show larger treatment effect estimates than those prospectively registered15), this finding is subject to further investigations.

In addition, we also found no difference in trial registration status between studies published in CAM journals and those published in non-CAM journals. It suggests that CAM journal editors and reviewers have monitored registration to some degree of their counterpart to secure the trial registration before the initiation of the trial. Moreover, the association between the sample size and the trial registration status was also identified, i.e. the higher registration rate among the larger trials, as reported in a previous study on cardiovascular trials18). Perhaps it is because researchers of larger trials tend to pay more attention from the beginning stage, including the protocol registration since larger trials usually cost a lot more time, efforts, and expenses.

While it is known that East Asian countries are more likely to produce positive findings in acupuncture studies than other countries27), this study found no significant association between the study origin and the proportion of studies with positive results. This may be partly attributed to the increased awareness of reporting biases among the acupuncture researchers for the past 20 years.

Outcome reporting bias has been investigated in various fields including oncology, traditional Chinese Medicine, and acupuncture17,20,28,37). The rate of inconsistency of primary outcomes between registered protocols and published articles ranged from 14 to 45% in previous studies17,20,28,37), which is similar to that of our study (47.5%). Therefore, the outcome reporting bias is also not uncommon across the specialties. Since the presence of the outcome reporting bias can exaggerate the results of systematic reviews, or moreover, hamper to make the appropriate decision of clinicians and policy makers, the authors and reviewers should pay more attention to this phenomenon38).

3. Strengths and limitations of this study

In comparison to previous studies that focused on the individual type of reporting biases27,28), this study comprehensively investigated the prevalence of reporting biases in published acupuncture RCTs, in terms of publication bias and outcome reporting bias and their association with the status of trial registration. We could confirm that reporting biases were prevalent in various forms which called for more attention from editors and researchers.

This study has several limitations. Firstly, articles written in Chinese were not included in the analyses. Considering that approximately 2,000 acupuncture RCTs were published in the past 20 years26), the reporting biases in Chinese articles should be taken into account and compared with articles written in English in future analyses. Secondly, we did not further search trial registries for the studies that did not provide the registration numbers in the published articles. While the CONSORT 2010 statement announced that “authors should provide the name of the register and the trial’s unique registration number”31,39), not all trials provide registration information in their publications28). Thus, we may have misclassified some trials as unregistered simply because they did not report registration numbers in their publications. Thirdly, the publication bias was examined by simple vote counting whether the trial registration status was associated with positive vs. negative findings in consideration of a substantial clinical and statistical heterogeneity among the included trials9). A comparison of treatment effect estimates between registered and unregistered trials by meta- epidemiologic approach would have informed more accurate treatment effect estimate.

4. Implications for future research

To evaluate generalizability of the present study findings, an extended research into unidentified acupuncture RCTs is needed. It needs to be confirmed whether Chinese language articles, or trials that did not state the registration number but might have actually registered in the trial registry, may have different impact on the association of trial registration status with presence of reporting biases. Given the findings in previous studies that reported the high prevalence of reporting biases in acupuncture studies published in China27,40), it can be informative to explore the difference in the presence of reporting biases between Chinese articles and non-Chinese articles. Changes in the prevalence of reporting biases in acupuncture RCTs over time can also be investigated to encourage journal editors and researchers to meet the higher standards for registering trials and more transparent reporting of study outcomes.

Conclusion

The rates of trial registration and adequate registration were not very high in acupuncture RCTs. It was demonstrated that unregistered trials were more likely to report positive results than registered ones. For the prospectively registered studies, about a half of them showed the discrepancies between the registered and published primary outcomes and 40% of those favored the statistically significant results. Trials with nonsignificant outcomes tended to interpret statistically nonsignificant results as equivalence between groups. To sum up, reporting biases including publication bias and outcome reporting bias in the acupuncture RCTs are still prevalent and comparable to those in other specialties, calling for more attention from journal editors and researchers in this field.

Acknowledgements

This work was supported by the National Research Foundation (NRF) of Korea funded by the Korean government (Ministry of Science and ICT, grant Nos. 2017R1A2B4006407 and 2014R1A1A2055507). H.L. was supported by Kyung Hee University for her sabbatical year. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

References
  1. Begg C, Cho M, Eastwood S, Horton R, Moher D, and Olkin I et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276:637-9.
    Pubmed CrossRef
  2. Devereaux PJ, and Yusuf S. The evolution of the randomized controlled trial and its role in evidence-based decision making. Journal of internal medicine. J Intern Med 2003;254:105-13.
    Pubmed CrossRef
  3. Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, and Altman DG et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes:meta-epidemiological study. BMJ 2008;336:601-5.
    Pubmed KoreaMed CrossRef
  4. Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, and Oxman AD et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928.
    Pubmed KoreaMed CrossRef
  5. Sterne JAC, Egger M, and Moher D. Addressing Reporting Biases. Cochrane Handbook for Systematic Reviews of Interventions. England: John Wiley & Sons, Ltd; 2008 p. 297-333.
    Pubmed CrossRef
  6. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, and Cronin E et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 2008;3:e3081.
    Pubmed KoreaMed CrossRef
  7. Dwan K, Gamble C, Williamson PR, and Kirkham JJ, Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One 2013;8:e66844.
    Pubmed KoreaMed CrossRef
  8. Hopewell S, Loudon K, Clarke MJ, Oxman AD, and Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009;21:MR000006.
    CrossRef
  9. Song F, Eastwood A, Gilbody S, Duley L, and Sutton AJ. Publication and related biases:a review. Health Technol Assess 2000;4:1-115.
  10. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, and Horton R et al. Clinical trial registration:a statement from the International Committee of Medical Journal Editors:International Committee of Medical Journal Editors. JAMA 2004;292:1363-4.
    Pubmed CrossRef
  11. Zarin DA, Tse T, and Ide NC. Trial registration at ClinicalTrials gov between May and October 2005. New Engl J Med 2005;2005:2779-87.
    Pubmed KoreaMed CrossRef
  12. Viergever RF, and Li K. Trends in global clinical trial registration:an analysis of numbers of registered clinical trials in different parts of the world from 2004 to 2013. BMJ Open 2015;5:e008932.
    Pubmed KoreaMed CrossRef
  13. National_Institutes_of_Health US. ClinicalTrials.gov.
    Available from: URL: https://clinicaltrials.gov/ [cited April 2018]
  14. Harriman SL, and Patel J. When are clinical trials registered? An analysis of prospective versus retrospective registration. Trials 2016;17:187.
    Pubmed KoreaMed CrossRef
  15. Dechartres A, Ravaud P, Atal I, Riveros C, and Boutron I. Association between trial registration and treatment effect estimates:a meta-epidemiological study. BMC Med 2016;14:100.
    Pubmed KoreaMed CrossRef
  16. Huić M, Marušić M, and Marušić A. Completeness and changes in registered data and reporting bias of randomized controlled trials in ICMJE journals after trial registration policy. PLoS One 2011;6:e25258.
    Pubmed KoreaMed CrossRef
  17. Mathieu S, Boutron I, Moher D, Altman DG, and Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 2009;302:977-84.
    Pubmed CrossRef
  18. Emdin C, Odutayo A, Hsiao A, Shakir M, Hopewell S, and Rahimi K et al. Association of cardiovascular trial registration with positive study findings:Epidemiological Study of Randomized Trials (ESORT). JAMA Int Med 2015;175:304-7.
    Pubmed CrossRef
  19. Nankervis H, Baibergenova A, Williams HC, and Thomas KS. Prospective registration and outcome-reporting bias in randomized controlled trials of eczema treatments:a systematic review. J Invest Dermatol 2012;132:2727-34.
    Pubmed KoreaMed CrossRef
  20. You B, Gan HK, Pond G, and Chen EX. Consistency in the analysis and reporting of primary end points in oncology randomized controlled trials from registration to publication:a systematic review. J Clin Oncol 2011;30:210-6.
    Pubmed CrossRef
  21. Burke A, Upchurch DM, Dye C, and Chyu L. Acupuncture use in the United States:Findings from the national health interview survey. J Altern Complement Med 2006;12:639-48.
    Pubmed CrossRef
  22. Frass M, Strassl RP, Friehs H, Müllner M, Kundi M, and Kaye AD. Use and acceptance of complementary and alternative medicine among the general population and medical personnel:a systematic review. Ochsner J 2012;12:45-56.
    Pubmed KoreaMed
  23. Hunt KJ, Coelho HF, Wider B, Perry R, Hung SK, and Terry R et al. Complementary and alternative medicine use in England:results from a national survey. Int J Clin Pract 2010;64:1496-502.
    Pubmed CrossRef
  24. Xue CC, Zhang AL, Lin V, Myers R, Polus B, and Story DF. Acupuncture, chiropractic and osteopathy use in Australia:a national population survey. BMC Public Health 2008;8:105.
    Pubmed KoreaMed CrossRef
  25. Liu Y, Zhang R, Huang J, Zhao X, Liu D, and Sun W et al. Reporting quality of systematic reviews/meta-analyses of acupuncture. PLoS One 2014;9:e113172.
    Pubmed KoreaMed CrossRef
  26. Ma B, Chen Z-m, Xu J-k, Wang YN, Chen KY, and Ke FY et al. Do the CONSORT and STRICTA Checklists Improve the Reporting Quality of Acupuncture and Moxibustion Randomized Controlled Trials Published in Chinese Journals?A Systematic Review and Analysis of Trends. PLoS One 2016;11:e0147244.
    Pubmed KoreaMed CrossRef
  27. Vickers A, Goyal N, Harland R, and Rees R. Do certain countries produce only positive results?A systematic review of controlled trials. Control Clin Trials 1998;19:159-66.
    CrossRef
  28. Su C-X, Han M, Ren J, Li WY, Yue SJ, and Hao YF et al. Empirical evidence for outcome reporting bias in randomized clinical trials of acupuncture:comparison of registered records and subsequent publications. Trials 2015;16:28.
    Pubmed KoreaMed CrossRef
  29. Chan A-W, Hróbjartsson A, Haahr MT, Gøtzsche PC, and Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials:comparison of protocols to published articles. JAMA 2004;291:2457-65.
    Pubmed CrossRef
  30. World Health Organization. Acupuncture:review and analysis of reports on controlled clinical trials. Geneva: World Health Organization; 2002.
  31. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, and Devereaux PJ et al. CONSORT 2010 explanation and elaboration:updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869.
    Pubmed KoreaMed CrossRef
  32. Killeen S, Sourallous P, Hunter IA, Hartley JE, and Grady HL. Registration rates, adequacy of registration, and a comparison of registered and published primary outcomes in randomized controlled trials published in surgery journals. Ann Surg 2014;259:193-6.
    Pubmed CrossRef
  33. Anand V, Scales DC, Parshuram CS, and Kavanagh BP. Registration and design alterations of clinical trials in critical care:a cross-sectional observational study. Intensive Care Med 2014;40:700-22.
    Pubmed CrossRef
  34. Milette K, Roseman M, and Thombs BD. Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals:A systematic review. J Psychosom Res 2011;70:205-17.
    Pubmed CrossRef
  35. Babu AS, Veluswamy SK, Rao PT, and Maiya AG. Clinical trial registration in physical therapy journals:a cross-sectional study. Phys Ther 2014;94:83-90.
    Pubmed CrossRef
  36. Scott A, Rucklidge JJ, and Mulder RT. Is Mandatory Prospective Trial Registration Working to Prevent Publication of Unregistered Trials and Selective Outcome Reporting?An Observational Study of Five Psychiatry Journals That Mandate Prospective Clinical Trial Registration. PLoS One 2015;10:e0133718.
    Pubmed KoreaMed CrossRef
  37. Liu J-P, Han M, Li XX, Mu YJ, Lewith G, and Wang YY et al. Prospective registration, bias risk and outcome-reporting bias in randomised clinical trials of traditional Chinese medicine:an empirical methodological study. BMJ Open 2013;3:e002968.
    Pubmed KoreaMed CrossRef
  38. Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, and Smyth R et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010;340:c365.
    Pubmed CrossRef
  39. Schulz KF, Altman DG, and Moher D. CONSORT 2010 statement:updated guidelines for reporting parallel group randomised trials. BMC Med 2010;8:18.
    Pubmed KoreaMed CrossRef
  40. Tang J-L, Zhan S-Y, and Ernst E. Review of randomised controlled trials of traditional Chinese medicine. BMJ 1999;319:160-61.
    CrossRef

September 2018, 35 (3)
Full Text(PDF) Free


Cited By Articles
  • CrossRef (0)
Social Network Service
Services