Pilot and feasibility trials in traditional Chinese medicine: a literature survey

Background The guidelines for pilot and feasibility studies were published in 2016. Little is known about the guideline adherence of TCM (Traditional Chinese Medicine) pilot trials or whether the guidelines can signi�cantly enhance the quality of implementation and reporting of TCM pilot trials. We aimed to investigate the guideline adherence, assess the impact of guidelines on TCM pilot trials, and discuss potentially undetected challenges for TCM pilot trials, by conducting a liteature survey. Methods We systematically searched MEDLINE, EMBASE and CNKI to retreive TCM pilot trials. We randomly chose 50 pilot trials from the eligible studies for analyses. The CONSORT extension to pilot and feasibility studies was used as a framework to assess the methodology and reporting quality of the studies. Results The included stuides had a guideline adherence level ranging from 4% to 96%, where the lowest adherence was found in the item 6c (prespeci�ed criteria used to judge progression to future de�nitive trial). The guidance published in 2016 seemed to exert minimal effect on guideline adherence in TCM pilot trials. The unidenti�ed issues related to TCM pilot trials from the guidelines included blinding, lack of standard formula of interventions, di�culty in comparison for effect assessment of interventions, and di�culty in bias control. Conclusions The current practice in TCM pilot trials requried substantial improvement in the literature. Further endeavors are needed for training and dissemination of guideline adherence, and development of more detailed methodology in the �eld of TCM pilot trials.


Introduction
Pilot and feasibility trials have been published with a growing number.Pilot trials are signi cantly important for the design of a future main trial (or de nitive trial) by providing evidence of feasibility issues and aoviding wasted recourses [1].In 2016, Eldridge et al published two critical publications aiming to reduce the misunderstanding and improve the reporting quality of pilot trials: the rst providing a conceptual framework to de ne a pilot trial [2], and the second developing a CONSORT (Consolidated Standards of Reporting Trials) extension for pilot trials with a 26-item checklist included [3].While the two publications may help with the design, implementation, reporting and dissemination of pilot trials, it remained largely unkonwn about their impact on the pilot trials published in the literature.Confusions remained in the pilot trials including their de nitions and terms, purpose, sample size determination, and criteria for progression or cessation, to mention a few [4][5][6].
Traditional Chinese medicine (TCM) is a hot topic in the health research community, especially given its alternative and integrated effect as a palliative treatment option [7].Notably, some uncertainties and challenges existed in clinical trials for TCM that mainly included the di culty in standardized procedures, potential heterogeneity in interventions and operators, control selection and outcome assessment.Pilot trials for TCM offered a platform to identify and address these issues before a main trial.However current evidence about the conduct and reporting of pilot trials for TCM is limited and sparse.Furthermore, little is known about whether the CONSORT extension for pilot trials can signi cantly enhance the quality of implementation and reporting of TCM pilot trials.Likewise, further evidence is needed to reveal the unidenti ed issues speci c to TCM pilot trials from the guidelines [3].Therefore in this study, we conducted a literature survey to investigate the guideline adherence of pilot trials for TCM, aiming to appraise the issuses related to methodology and reporting.We also aimed to assess the impact of CONSORT extension for pilot trials, and discuss any potentially undetected challenges for TCM pilot trials.

Search strategy and study selection
We systematically searched MEDLINE, EMBASE and CNKI to retreive TCM pilot trials.Descriptors including synonyms for traditional Chinese medicine or herbal medicine or folk medicine, and pilot trials or feasibility studies, were used in combination for the literature search (Supplemental Table 1 presents the search terms used).Studies were eligible for inclusion if they explicitly identi ed their TCM research as a randomized pilot or feasibility trial in the titles, abstracts or introductions.Studies were excluded if they did not specify as a randomized pilot or feasibility trial, or they were not related to TCM, or they did not have information for methodological and reporting appraisal.Two reviewers (GL and XC) independently screened the records and determined study eligilibity.

Data extraction
Data extraction was completed by two independent reviewers (GL and XC).We categorized the included TCM pilot trials into two groups: 1) pilot trials that had at least one objective or assessment of feasibility and were conducted in preparation for a future de nitive trial (FDT), and 2) trials that did not have feasibility objectives or assessment, termed as non-feasibility trials (NFT).This methodology was similar to Horne's approach [8].We assessed the guideline adherence about Title and Abstract (1a and 1b listed in the checklist), Introduction (2a and 2b), Methods (3a, 4c, 6a, 6c, 7a and 12a), Results (13a) and Discussion (20, 21, and 22a) [3], seperated by the two groups (FDT and NFT).
To document the unidenti ed issues speci c to TCM pilot trials, we also extracted the relevant data throughout the text from the included studies, especially in their Discussion sections.

Statistical analyses
We expected that the proportion of FDT in our included studies would be approximately 15%.Therefore we randomly chose 50 pilot trials from the 285 eligible studies for analyses (Supplemental Table 2 shows the reference details for the included studies and Fig. 1 shows the process of identifying eligible studies).To assess the impact of CONSORT extension for pilot trials on reporting, we selected the 50 studies that were published in either before or after the year 2016; i.e., no studies published in 2016 were identi ed for our analyses.
Guideline adherence was presented using counts and percentages.We performed a Chi-square test to compare the guideline adherence levels between the two groups (FDT and NFT).To evaluate the impact of the CONSORT extension for pilot trials, we compared the guideline adherene of the included pilot trials published before and after 2016.When there was a cell with expected frequency < 5 in the contigency table, we used Fisher's exact test to compare the guideline adherence levels between the groups.All analyses were conducted using the STATA Version 13 (Stata Corp., College Station, TX, USA).

Results
As shown in Fig. 1, we identi ed 285 eligible TCM pilot trials, among which 50 were randomly selected for analyses.The selected 50 trials were published between year 1998 and 2019, and had a sample size ranging from 7 to 160 (Table 1).The TCM assessed in the trials included herbs, acupuncture, Chinese patent medicine, Qigong, massage, and others.There were 12 trials categorized as FDT (24%) and 38 as NFT (76%).
Thirty-eight trials (76%) were published before year 2016, and 12 trials (24%) after 2016.(generalizability of methods and ndings) also had low guideline adherence leves (18%, 8% and 18% respectively).Table 2 also shows comparisons between FDT and NFT, and between studies published before and after year 2016.Compared with the NFT, the FDT had a signi cantly higher guideline adherence in the item 7a (rationale for sample size; 25% vs 3%) and 20 (discussion of study limitation, bias and uncertainty; 58% vs 34%).Guideline adherence level was only found signi cantly higher in the item 12a (qualitative or quantitative methods used to address objectives) in trials published after year 2016, when compared with studies published before 2016 (100% vs 55%).The unidenti ed issues related to TCM pilot trials from the guidelines were shown in Table 3.There were 3 trials raising the issue of blinding in TCM pilot trials, mainly due to the acupoints, administration forms, smells, and other reasons [9][10][11].Other issues included lack of standard formula of interventions, di culty in comparison for effect assessment of interventions, and di culty in bias control [9,10,12,13] (Table 3).
Blinding "although the shape and color of the placebo were similar to Yueju, the smells of Yueju and placebo were not exactly identical, which may lead to the plausible incomplete blind treatment to patients.

Intervention and bias control
Tsai, 2018 (

Discussion
In this study, we performed a survey to assess the guieline adherence of TCM pilot trials.The guideline adherence varied acrossing the checklist items, where some items required signi cant improvement.The guidance papers published in 2016 seemed to exert minimal effect on guideline adherence in TCM pilot trials.We also identi ed several issues speci c to TCM pilot trials in this sruvey.
Interestingly, there were only 24% TCM pilot trials that had an objective of feasibility and were performed in preparation for future de nitive trials (FDT).This indicated the inappropriate use of the term pilot in many small trials that aimed to test the hypotheses of effciacy or safety with an insu cient sample size albeit being underpowered to do so [8, 14,15].It also corresponded to the item 2b (speci c objectives or research questions), where surprisingly only 3 (25%) FDT clearly stated their objectives related to feasibility.
Furthermore, there were only two items (7a and 20) found with signi cant improved guideline adherence in FDT compared with NFT, implying that more endeavours were requried even in those pilots trials with speci ed feasibility objective(s).Therefore all these ndings suggested further dissemination of the guidline to help clarify the de nition of feasibility and pilot trials [2] and to enhance the guildeline adherence [3].
Likewise, our study indicated that the impact of CONSORT extension for pilot trials warranted more efforts in TCM pilot trials, because the improvement was only found in one item (12a) after the guidelines were published (Table 2).The minimal effect of the guidance papers may be because either that the guidelines did not reach the relevant research parties, or that the guidelines were largely ignored by the research parties [8].
In any case, our survey reveals the urgent need for both training and dissemination of research methodology and guideline adherence in TCM pilot trials.
Besides the common practice of the inappropriate hypothesis testing and insu cient power for conclusion in pilot trials [14,16], our study also identi ed some issues speci c to TCM pilot trials including blinding, standards for intervention and comparisons, and bias reduction (Table 3).This entails more guidance on methodology and reporting speci c to TCM pilot trials, because the existing guidelines including CONSORT extensions to acupuncture [17], herbal interventions [18], and PAFS [3] could not fully cover these issues in TCM pilot trials.The progression criteria (guideline adherence level: 4%), sample size rationale (18%) and generalizability of methods and ndings (18%) were also notable issues found in the TCM pilot trials (Table 2).This may be, at least in part, due to the insu cient details on explanation and elaboration from the guideline.For example, even though the CONSORT extension recommended that authors should justify the number of pariticipants in pilot trials [3], no su cient details on how to exactly provide sample size rationale could be found in the guideline.Likewise, how to specify the progression criteria to determine whether the pilot trial can progress to future main trial, and whether the methods and ndings can be generalizable to main trial and other pilot studies, required further detailed investigation and guidance in TCM pilot trials.The TCM eld is substantially different from modern medicine, especially in their intervention, control and outcome assessment.Thus, our ndings call for the need for further methodology and guidance in the research area of pilot and feasibility studies to address the unidenti ed issues and the other notable issues related to TCM pilot trials.
Our study was the rst to explore the current practice of methodology and reporting in TCM pilot trials.We completed the data acquisition and analyses by two reviewers independently, thereby enhancing the accuracy of study ndings [19].There are also some limitations to our study.Due to the small numbers of the included FDT (n = 12) and studies published after year 2016 (n = 12), we only performed raw comparions without adjustments, which may yield biased ndings in univariate analyses.We could not further extract potential solutions from the included TCM studies, indicating the important gap in methodological guidance in TCM pilot trials.Furthermore, only studies in Chinsese and English were screened and selected, which may therefore introduce selection bias due to lack of studies in other languages such as Japanese and Korean.
Moreover, the impact of time lag between the publication of a new guideline and the adoption and implementation of it could not be fully assessed, which may therefore weaken the ndings of our study.
To conclude, the current practice in TCM pilot trials requried substantial improvement in the literature.The guideline seemed to have only minimal effect on the methodology and reporting in TCM pilot trials, and some issues related to TCM pilot studies still warranted further methodology and guidance.Further endeavors are needed for training and dissemination of guideline adherence, and development of more detailed methodology in the eld of TCM pilot trials.

Figures
Figures

Table 1
Characteristics of the 50 included studies FDT: trials in preparation for a future de nitive trial NFT: non-feasibility trials

Table 2
presents the detailed guideline adherence levels of the selected trials.The adherece ranged from 4-96%, with the lowest adherence found in 6c (prespeci ed criteria used to judge progression to future de nitive trial) and highest in 12a (qualitative or quantitative methods used to address objectives).The checklist items 2b (speci c objectives or research questions), 7a (rationale for sample size) and 21

Table 2
Details for guideline adherence of the included studies *p-value < 0.05 for difference test

Table 3
Unidenti ed issues related to TCM pilot trials from the guidelines