Pilot and feasibility trials in traditional Chinese medicine: a literature review of current practice.

Background The guidelines for pilot and feasibility studies were published in 2016. Little is known about the guideline adherence of TCM (traditional Chinese medicine) pilot trials or whether the guidelines can significantly enhance the quality of implementation and reporting of TCM pilot trials. We aimed to investigate the guideline adherence, assess the impact of guidelines on TCM pilot trials, and discuss potential challenges specific to TCM pilot trials, by conducting a literature review. Methods We systematically searched MEDLINE, EMBASE, and CNKI to retrieve TCM pilot trials. We randomly chose 50 pilot trials from the eligible studies for analyses. The CONSORT extension to pilot and feasibility studies was used as a framework to assess the methodology and reporting quality of the studies. Results The included studies had a guideline adherence level ranging from 4 to 96%, where the lowest adherence was found in the item 6c (prespecified criteria used to judge progression to future definitive trial). The guidance published in 2016 seemed to exert minimal effect on guideline adherence in TCM pilot trials. The unidentified issues related to TCM pilot trials from the guidelines included blinding, lack of standard formula of interventions, difficulty in comparison for effect assessment of interventions, and difficulty in bias control. Conclusions The current practice in TCM pilot trials required substantial improvement in the literature. Further endeavors are needed for training and dissemination of guideline adherence, and development of more detailed methodology in the field of TCM pilot trials.


Introduction
Pilot and feasibility trials have been published with a growing number. Pilot trials are significantly important for the design of a future main trial (or definitive trial) by providing evidence of feasibility issues and avoiding wasted recourses [1]. In 2016, Eldridge et al. published two critical publications aiming to reduce the misunderstanding and improve the reporting quality of pilot trials: the first providing a conceptual framework to define a pilot trial [2], and the second developing a CONSORT (Consolidated Standards of Reporting Trials) extension for pilot trials with a 26-item checklist included [3]. While the two publications may help with the design, implementation, reporting, and dissemination of pilot trials, it remains largely unknown about their impact on the pilot trials published in the literature. Confusions remained in the pilot trials including their definitions and terms, purpose, sample size determination, and criteria for progression or cessation, to mention a few [4][5][6].
Traditional Chinese medicine (TCM) is a hot topic in the health research community, especially given its alternative and integrated effect as a palliative treatment option [7]. Notably, some uncertainties and challenges exist in clinical trials for TCM that mainly include the difficulty in standardized procedures, potential heterogeneity in interventions and operators, control selection, and outcome assessment. Pilot trials for TCM offer a platform to identify and address these issues before a main trial. However, current evidence about the conduct and reporting of pilot trials for TCM is limited and sparse. Furthermore, little is known about whether the CONSORT extension for pilot trials can significantly enhance the quality of implementation and reporting of TCM pilot trials. Likewise, further evidence is needed to reveal the unidentified issues specific to TCM pilot trials from the guidelines [3]. Therefore, in this study, we conducted a literature review to investigate the guideline adherence of pilot trials for TCM, aiming to appraise the issues related to methodology and reporting. We also aimed to assess the impact of the CONSORT extension for pilot trials, and discuss any potential challenges specific to TCM pilot trials.

Search strategy and study selection
We systematically searched MEDLINE, EMBASE, and CNKI to retrieve TCM pilot trials. Descriptors including synonyms for traditional Chinese medicine or herbal medicine or folk medicine, and pilot trials or feasibility studies, were used in combination for the literature search (Supplemental Table 1 presents the search terms used). Studies were eligible for inclusion if they explicitly identified their TCM research as a randomized pilot or feasibility trial in the titles, abstracts, or introductions. Studies were excluded if they were not identified as a randomized pilot or feasibility trial, or they were not related to TCM, or they did not have information for methodological and reporting appraisal. Two reviewers (GL and XC) independently screened the records and determined study eligibility.

Data extraction
Data extraction was completed by two independent reviewers (GL and XC). We categorized the included TCM pilot trials into two groups: (1) pilot trials that had at least one objective or assessment of feasibility and were conducted in preparation for a future definitive trial (FDT) and (2) trials that did not have feasibility objectives or assessment, termed as non-feasibility trials (NFT). This methodology was similar to Horne's approach [8]. We assessed the guideline adherence about Title and Abstract (1a and 1b listed in the checklist), Introduction (2a and 2b), Methods (3a, 4c, 6a, 6c, 7a, and 12a), Results (13a), and Discussion (20,21, and 22a) [3], separated by the two groups (FDT and NFT).
To document the methodological issues specific to TCM pilot trials, we also extracted the relevant data throughout the text from the included studies, especially in their Discussion sections.

Statistical analyses
We expected that the proportion of FDT in our included studies would be approximately 15%. Therefore, we randomly chose 50 pilot trials from the 285 eligible studies for analyses ( Fig. 1 shows the process of identifying eligible studies). To assess the impact of CONSORT extension for pilot trials on reporting, we selected the 50 studies that were published in either before or after the year 2016; i.e., no studies published in 2016 were identified for our analyses.
Guideline adherence was presented using counts and percentages. We performed a chi-square test to compare the guideline adherence levels between the two groups (FDT and NFT). To evaluate the impact of the CON-SORT extension for pilot trials, we compared the guideline adherence of the included pilot trials published before and after 2016. When there was a cell with expected frequency < 5 in the contingency table, we used Fisher's exact test to compare the guideline adherence levels between the groups. All analyses were conducted using the STATA version 13 (Stata Corp., College Station, TX, USA).

Results
As shown in Fig. 1, we identified 285 eligible TCM pilot trials, among which 50 were randomly selected for analyses . The selected 50 trials were published between year 1998 and 2019, and had a sample size ranging from 7 to 160 ( Table 1). The TCM assessed in the trials included herbs, acupuncture, Chinese patent medicine, Qigong, massage, and others. There were 12 trials categorized as FDT (24%) and 38 as NFT (76%). Thirty-eight trials (76%) were published before year 2016, and 12 trials (24%) after 2016. Table 2 presents the detailed guideline adherence levels of the selected trials. The adherence ranged from 4 to 96%, with the lowest adherence found in 6c (prespecified criteria used to judge progression to future definitive trial) and highest in 12a (qualitative or quantitative methods used to address objectives). The checklist items 2b (specific objectives or research questions), 7a (rationale for sample size), and 21 (generalizability of methods and findings) also had low guideline adherence levels (18%, 8%, and 18% respectively). Table 2 also shows comparisons between FDT and NFT, and between studies published before and after year 2016. Compared with the NFT, the FDT had a significantly higher guideline adherence in the item 7a (rationale for sample size; 25% vs 3%) and 20 (discussion of study limitation, bias and uncertainty; 58% vs 34%). Guideline adherence level was only found significantly higher in the item 12a (qualitative or quantitative methods used to address objectives) in trials published after year 2016, when compared with studies published before 2016 (100% vs 55%).
The methodological issues specific to TCM pilot trials from the guidelines are shown in Table 3. There were 3 trials raising the issue of blinding in TCM pilot trials, mainly due to the acupuncture, administration forms, smells, and other reasons [12,27,51]. Other issues included lack of standard formula of interventions, difficulty in comparison for effect assessment of interventions, and difficulty in bias control [12,27,47,58] (Table 3). For instance, in a pilot trial conducted by Choi et al., they reported that it was extremely difficult to evaluate the intervention effect because no standard treatment for atopic dermatitis could be used for comparison based on the current evidence-based TCM [58].

Discussion
In this study, we performed a review to assess the guideline adherence of TCM pilot trials. The guideline adherence varied crossing the checklist items, where some items required significant improvement. The guidance papers published in 2016 seemed to exert minimal effect on guideline adherence in TCM pilot trials. We also identified several issues specific to TCM pilot trials in this review including blinding, standards for intervention and comparisons, effect assessment, and bias reduction.
Interestingly, there were only 24% TCM pilot trials that had an objective of feasibility and were performed in preparation for future definitive trials (FDT). This indicated the inappropriate use of the term pilot in many small trials that aimed to test the hypotheses of efficacy or safety with an insufficient sample size albeit being underpowered to do so [8,59,60]. It also corresponded to the item 2b (specific objectives or research questions), where surprisingly only 3 (25%) in the FDT group clearly stated their objectives related to feasibility. Furthermore, there were only two items (7a and 20) found with significant improved guideline adherence in FDT compared with NFT, implying that more endeavors were required even in those pilot trials with specified feasibility objective(s). Therefore, all these findings suggested further dissemination of the guideline to help clarify the definition of feasibility and pilot trials [2] and to enhance the guideline adherence [3].
Likewise, our study indicated that the impact of CON-SORT extension for pilot trials warranted more efforts in TCM pilot trials because the improvement was only found in one item (12a) after the guidelines were published   (Table 2). The minimal effect of the guidance papers may be because either the guidelines did not reach the relevant research parties, or that the guidelines were largely ignored by the research parties [8]. In any case, our review reveals the urgent need for both training and dissemination of research methodology and guideline adherence in TCM pilot trials. Besides the common practice of inappropriate hypothesis testing and insufficient power for conclusion in pilot trials [59,61], our study also identified some issues specific to TCM pilot trials including blinding, standards for intervention and comparisons, and bias reduction ( Table 3). This entails more guidance on methodology and reporting specific to TCM pilot trials because the existing guidelines including CONSORT extensions to acupuncture [62], herbal interventions [63], and pilot and feasibility studies [3] could not fully cover these issues in TCM pilot trials. The progression criteria (guideline adherence level, 4%), sample size rationale (18%), and generalizability of methods and findings (18%) were also notable issues found in the TCM pilot trials ( Table  2). This may be, at least in part, due to insufficient details on explanation and elaboration from the guideline. For example, even though the CONSORT extension recommended that authors should justify the number of participants in pilot trials [3], no sufficient details on how to exactly provide sample size rationale could be found in the guideline. Likewise, how to specify the progression criteria to determine whether the pilot trial can progress to future main trial, and whether the methods and findings can be generalizable to main trial and other pilot studies, required further detailed investigation and guidance in TCM pilot trials. The TCM field is substantially different from modern medicine, especially in their intervention, control, and outcome assessment. For example, our review found that the issues specific to TCM pilot trials including blinding, standards for intervention and comparisons, effect assessment, and bias reduction, were not discussed in the CONSORT extension (Table  3). Thus, our findings call for the need for further methodology and guidance in the research area of pilot and feasibility studies to address the methodological issues and the other notable issues specific to TCM pilot trials.
Our study was the first to explore the current practice of methodology and reporting in TCM pilot trials. We completed the data acquisition and analyses by two reviewers independently, thereby enhancing the accuracy of study findings [64]. There are also some limitations to our study. Due to the small numbers of the included FDT (n = 12) and studies published after year 2016 (n = FDT trials in preparation for a future definitive trial, NFT non-feasibility trials 12), we only performed raw comparisons without adjustments, which may yield biased findings in univariate analyses. We could not further extract potential solutions from the included TCM studies, indicating the important gap in methodological guidance in TCM pilot trials. Furthermore, only studies in Chinese and English were screened and selected, which may therefore introduce selection bias due to lack of studies in other languages such as Japanese and Korean. Moreover, the impact of time lag between the publication of a new guideline and the adoption and implementation of it could not be fully assessed, which may therefore weaken the findings of our study.
To conclude, the current practice in TCM pilot trials required substantial improvement in the literature. The guideline seemed to have only minimal effect on the
Additional file 1: Table S1. Search terms used in the EMBASE, MEDLINE and CNKI. Blinding; intervention "in this study JWSYS [Jia-Wey Shiau-Yau San] was given in powder form and Premelle in tablet form. The question arises as to whether the women receiving JWSYS were aware that they were taking an established traditional Chinese herbal remedy. Since the trial was not a blind one and the improvement in the symptoms of these women could be due to an expectancy/placebo effect, given the cultural milieu" Chen [12] Randomization and blinding; intervention "treatment with the complementary therapies of CM [Chinese medicine] had to be agreed by the patients or their families, thus randomly assigning the patients to the ST [standard treatment] or CH [Chinese herbs] by a completely blind method was difficult" "there was no fixed CM formula" Lee [27] Comparison and effect estimate "in addition, there is no standard treatment for AD [atopic dermatitis] based on evidence-based medicine that could be used for comparison. Therefore, it is very difficult to rate an intervention compared to a standard herbal medicine" Choi 2012 [58] Blinding "although the shape and color of the placebo were similar to Yueju, the smells of Yueju and placebo were not exactly identical, which may lead to the plausible incomplete blind treatment to patients.
Wu [51] Intervention and bias control "the current study cannot exclude the possible effects of HAT [herbal acupuncture therapy] on other factors, such as basic herbal regimens, proper acupuncture selection, and long-term therapeutic courses involved in the response of IDH [Intradialytic hypotension]" Tsai [47]