Reporting quality of pilot clinical trials in chronic kidney disease patients on hemodialysis: a methodological survey
Pilot and Feasibility Studies volume 5, Article number: 53 (2019)
The conduct of high-quality pilot studies can help inform the success of larger clinical trials. Guidelines have been recently developed for the reporting of pilot trials.
This methodological survey evaluates the completeness of reporting in pilot randomized controlled trials in chronic kidney disease patients on hemodialysis (HD patients) and explores factors associated with better completion of reporting.
The authors searched Pubmed on July 1, 2018, for all pilot trials conducted in HD patients. Reporting quality was assessed against the 40-item Consolidated Standards of Reporting Trials (CONSORT) Extension for Pilot Trials. Study factors including year and country of publication, intervention, number of centers, type of funding, and journal endorsement of CONSORT were also examined.
The mean number of items reported from the CONSORT extension for pilot trials across all included articles was 18.4 (standard deviation [SD] = 4.4). In the adjusted analysis, studies reported in later years (IRR = 1.026, 95% CI [1.018, 1.034], p < 0.001) and an increase of 20 persons in sample size (adjusted IRR = 1.021, 95% CI [1.010, 1.031], p < 0.001) were associated with a significantly higher number of CONSORT pilot items reported.
Current reporting completeness of pilot trials in HD patients is suboptimal. Endorsing the CONSORT extension specific to pilot and feasibility studies and ensuring that pilot trials focus on the feasibility objectives may improve reporting completeness of these trials.
Chronic kidney disease (CKD) is a significant and growing global health problem, with a prevalence estimated to be between 11 to 13% . CKD is defined as having a decreased kidney function for at least 3 months, regardless of the etiology, and has many serious complications such as uremia, volume overload, and hematologic and metabolic disturbances [2, 3]. Risk of mortality, largely due to cardiovascular disease, is significantly increased in the CKD population and increases as kidney function declines . An estimated two million CKD patients with very little or no residual kidney function require kidney replacement therapy . Kidney replacement therapy is very costly, consuming 6.7% of the total Medicare budget to care for less than 1% of the covered population. The costliest modality is hemodialysis (HD), estimated at over US$87,000 per year per patient . Despite significant investment into research in the last 40 years, survival and quality of life of patients on HD remain low .
There are many pressing clinical questions in HD which require a definitive, well-powered randomized clinical trial (RCT) . Health care providers and patients alike have identified the need for further research across a range of priorities in HD including how to best address vascular access problems, reduce fatigue, risk of mortality, and cardiovascular disease and to improve dialysis adequacy . However, the immense quantity of information and resources required for the conduct of adequately powered RCTs across these clinical areas in HD can act as a barrier to their conduct [7, 8]. Pilot studies can facilitate designing such definitive RCTs by assessing feasibility of screening, recruitment, coordination and acceptability, safety, and fidelity of the intervention and the study protocol, as well as informing power calculations [7, 9, 10].
However, pilot trial methodology is often criticized for inadequacies, mainly critiquing the emphasis on hypothesis testing and the lack of criteria for evaluating feasibility [11, 12]. To address these issues, the Consolidated Standards of Reporting Trials (CONSORT) extension for reporting randomized pilot and feasibility trials was published in 2016, which builds on the statement published in 2010 . The 2016 CONSORT extension, which will be utilized in this study, lays the groundwork for the reporting of pilot trials, as well as informs their design and implementation. This is expected to enhance the completeness and transparency in the reporting of pilot RCTs and establish a standardized approach to this area of research .
Until recently, standards for the reporting of pilot studies have been unavailable and it is likely that there are significant gaps and inconsistencies in the reporting of pilot studies in HD—as similarly identified in other areas of clinical research . Identification of these gaps may not only help inform initiatives to improve reporting, but also potentially raise awareness among clinicians and research about pilot trial design and implementation. Using the newly published CONSORT extension for pilot trials, we undertook a methodological survey to assess reporting completeness among pilot trials investigating interventions in CKD patients on hemodialysis (HD patients) and explored factors associated with better completion of reporting. A methodologic review is a type of study designed to examine methodologic quality of a sample of articles, generally within a certain discipline or study design [14,15,16].
Inclusion criteria: (1) the words “pilot” or “feasibility” used to describe its design, (2) randomized control trial, (3) examine interventions in HD patients, and (4) published in English.
Exclusion criteria: (1) single-arm observational pilot or feasibility studies, (2) quasi-randomized trials, and (3) studies among nonclinical or acute populations.
In order to survey the medical literature for pilot trials in HD patients, we searched Medline/Pubmed and included studies published before July 1, 2018, using the following strategy: (1) feasibility studies/ or pilot projects/; (2) pilot stud*.mp.; (3) exp Randomized Controlled Trial/; (4) randomized controlled trial.pt.; (5) controlled clinical trial.pt.; (6) randomized controlled trials.sh.; (7) random allocation.sh.; (8) double-blind method.sh.; (9) single-blind method.sh.; (10) 3 or 4 or 5 or 6 or 7 or 8 or 9; (11) (animals not humans).sh.; (12) 10 not 11; (13) 1 or 2; (14) 12 and 13; (15) exp Renal Dialysis/; (16) renal dialysis.tw.; (17) 15 or 16; (18) hemodialysis.tw.; (19) haemodialysis.tw.; (20) 17 or 18 or 19; (21) exp peritoneal dialysis/; and (22) 20 not 21.
One author (SK) screened all titles and abstracts based on the inclusion and exclusion criteria and conducted a full-text review of randomly selected citations to assess eligibility. The full-text articles were then screened independently and in duplicate for eligibility by three teams of reviewers (AL and SL, DS and SV, and SS and SS).
The primary outcome of this survey was the completeness of reporting of each of the items on CONSORT statement extension for randomized pilot and feasibility trials checklist, measured as a number and proportion of studies reporting each of the 40 items. The secondary outcome was the completeness of reporting of the CONSORT statement extension for randomized pilot and feasibility trials checklist, as measured by the total number of applicable items reported.
The Excel-based data extraction form was developed based on a previous methodological survey  and collected study characteristics including year and country of publication, sample size, number of sites, type of funding (i.e., industry, non-industry), type of intervention (i.e., pharmaceutical, behavioral/educational, dialysis technology/technique, nutritional supplements, vascular access technology/technique, and other non-pharmaceutical interventions), whether the manuscript explicitly stated the pilot study to be prelude to definitive study (i.e., yes, no), journal endorsement of CONSORT statement if the article was published after 2010 (i.e., yes, no), and reporting of individual items on the CONSORT extension for pilot and feasibility studies (Additional file 1). For the reporting of individual items on the CONSORT extension for pilot and feasibility studies, each item on the checklist was scored as either “reported” or “not reported,” indicating whether or not the article reported the appropriate information as per the criteria outlined in the original publication (Table 1). Exceptions to this are items 6c, 7b, 11a, 11b, 18, and 19a, all of which had “not applicable” as an option .
All statistical analyses were performed in SPSS version 25.
The completeness of reporting was summarized using descriptive statistics percentages for the general characteristics and number of articles reporting each CONSORT statement item (for items 6c, 7b, 11a, 11b, 18, and 19a, the percentage was calculated based on the total number of studies for which the item was applicable). The mean, standard deviation, and range for the total number of CONSORT statement items reported, sample size, and number of sites were also calculated. To calculate the mean number of CONSORT items reported, “not applicable” responses to reporting items 6c, 7b, 11a, 11b, 18, and 19a were excluded.
We conducted a Poisson regression to explore factors, including year of publication, sample size, multisite study (yes or no), industry funding (yes or no), prelude to a definite trial (yes or no), and journal endorsement of CONSORT (yes or no), associated with completeness of reporting as measured by the number of reported CONSORT items (“not applicable” items were excluded). Based on previous research, we hypothesized that a later publication date , larger sample size , multisite study , industry funding [16, 18, 19], and journal endorsement of CONSORT  would be associated with better reporting. The results of the Poisson regression were reported as unadjusted and adjusted incidence rate ratios (IRR) including 95% confidence interval (CI) and p value (α = 0.05).
Journal of publication was also considered as a potential factor to include in the Poisson regression as some clustering has been observed in prior studies on reporting [16, 18]; however, no clustering within journals was noted in exploratory analyses, likely due to the wide breadth of journals in which the included studies were published, with no more than 10 studies published in the same journals (see characteristics of included studies below).
Our initial search retrieved 593 records, of which 86 were included in the synthesis (see Fig. 1 for PRISMA diagram).
Characteristics of included studies
The studies examined a wide range of interventions (see Table 2) including pharmaceutical (33.7%, e.g., magnesium carbonate plus calcium acetate, oral cholecalciferol), behavioral/educational (19.8%, e.g., cycling, resistance, or cycling and resistance, chairside meditation), dialysis technology/technique (17.4%, e.g., equilibrated kt/v goal of 1.4, fx-e membrane), nutritional supplements (14.0%, e.g., fish oil), vascular access technology/technique (12.8%, e.g., dialysis needles placed 2.5 cm and then 5 cm apart, u clip anastomosis), and other non-pharmaceutical interventions (2.3%, i.e., acupuncture, low-intensity vibration device). The majority of the studies were single center (68.6%) and had a sample size of 50 or less (74.4%). The mean sample size was 44.0 (standard deviation (SD) = 55.1, range = 4–448) and number of sites was 2.2 (SD = 2.7, range = 1–15).
The studies were conducted across the globe, including Africa (1.2%), Asia (15.1%), Australia/New Zealand (7.0%), Europe (27.9%), and North America (48.8%). The majority of studies were published since 2010 (60.5%). Among those published since 2010, 42.9% were published in journals that endorse the CONSORT statement. Only 17.4% of studies explicitly indicated in the manuscript that the study was a prelude to a definitive trial. The studies were published across a wide range of journals, with 10 studies published in J Ren Nutr, 8 in Am J Kidney Dis, 5 in Nephrol Dial Transplant, 4 in J Vasc Access, 3 in each of Clin J Am Soc Nephrol, J Nephrol, Nephrology (Carlton), and Ther Apher Dial, 2 in each of Int Urol Nephrol, Kidney Int, Pharmacotherapy, and PLoS One, with the remaining 39 studies published in different journals.
Evaluation of reporting quality based on CONSORT extension
The mean CONSORT reporting score across all included articles was 18.4 (SD = 4.4, minimum = 8, maximum = 29) out of a possible 34 items (6c, 7b, 11a, 11b, 18, and 19a were excluded as they had a not applicable option). Table 1 shows the level of reporting of individual CONSORT items. The items reported by the largest proportion of articles (top 10%) were “2b. Specific objectives or research questions for pilot trial” (97.7%); “4a. Eligibility criteria for participants” (93.0%); “12. Methods used to address each pilot trial objective whether qualitative or quantitative” (97.7%); and “22. Interpretation consistent with pilot trial objectives and findings, balancing potential benefits and harms, and considering other relevant evidence” (97.7%).
The most poorly reported items (bottom 10%) were “3b. Important changes to methods after pilot trial commencement (such as eligibility criteria), with reasons” (5.8%); “6b. Any changes to pilot trial assessments or measurements after the pilot trial commenced, with reasons” (3.5%); “6c. When applicable, explanation of any interim analyses and stopping guidelines” (7.9%); and “24. Where the pilot trial protocol can be accessed, if available” (8.1%).
Factors related to reporting of CONSORT extension items
Table 3 shows the unadjusted and adjusted IRRs for overall CONSORT reporting by study characteristics. In comparing the total number of reported CONSORT items by the prespecified study characteristics, studies reported in later years (adjusted IRR = 1.026, 95% CI [1.018, 1.034], p < 0.001) and an increase of 20 persons in sample size (adjusted IRR = 1.021, 95% CI [1.010, 1.031], p < 0.001) had a significantly higher number of CONSORT pilot items reported. After adjusting for other covariates and factors, the remaining study characteristics were not significantly associated with reporting completeness.
In this systematic survey, one of the few to examine the completeness of reporting in pilot and feasibility RCTs , we found that the mean number of CONSORT extension items for pilot and feasibility studies was 18.4 out of a possible 34 applicable items. We did find, however, that there is a 2.6% increase in the number of CONSORT pilot items reported for each additional year of publication. Larger sample sizes were also associated with higher number of CONSORT pilot items reported; however, the observed effect was relatively small (e.g., there is a 2.1% increase in reporting completeness for an increase in sample size of 20 participants). The number of reported items may continue to improve over time, particularly with the 2016 publication of the CONSORT extension for pilot and feasibility RCTs. Indeed, the completeness of reporting full RCTs in nephrology has improved over time, though reporting quality issues for certain CONSORT (2010) items such as clinical trial design, mode of randomization, and intention-to-treat analysis persist [8, 21,22,23]. Lack of available guidelines prior to the CONSORT extension and lack of its awareness (only recently published) could be contributing to this suboptimal reporting . These guidelines may help clarify the purpose and reporting of pilot studies in HD patients.
In our sample of pilot studies, we found that the reporting of some items was consistently reported. These items are very similar to those included in the 2010 CONSORT statement, and include specific objectives for the pilot trial, eligibility criteria for participants, methods used to address each pilot trial objective whether qualitative or quantitative, and the interpretation is consistent with pilot trial objectives and findings. However, certain items specific to pilot trials and clinical trial conduct transparency were reported in less than 10% of included studies which, as previously indicated, may be attributable to lack of reporting guidelines for pilot trials until recently. These include changes to methods and assessments or measurements after pilot trial commencement, progression criteria for a future definitive trial, and where the pilot trial protocol can be accessed. These items are critical to the development of a larger, definitive trial that is rigorously designed and well powered, which should be the primary purpose of a pilot study . The low levels of reporting for these items is consistent with our finding that only 17.4% of included studies indicated that they were a prelude to a larger trial, as well as the findings of a previous study on reporting completeness in pilot trials in behavioral interventions (13%) . The manuscripts may not have explicitly stated the pilot study to be a prelude to a definitive study (CONSORT 2a and 22a) as this might be assumed by authors given that it is the purpose of a pilot study; however, this should be explicitly reported for every study as recommended by the guidelines.
Many of the included studies were primarily designed to address questions of clinical efficacy of a wide range of interventions. Given the relatively small sample of the majority of the included trials, these studies were likely underpowered to do so. Though some items were well reported, going forward, it is critical that journals publishing pilot trials in HD patients ensure that these studies adhere to the pilot trial extension to the CONSORT reporting statement, as well as confirm that the primary objectives of these studies are related to feasibility—not efficacy—objectives. This may be accomplished as part of the peer review process or as a requirement of submission.
There are several limitations of this study that are important to acknowledge. We only included English-language studies due to feasibility purposes, which may limit the generalizability of these findings. As this was a methodological survey and not a systematic review, this study’s search was restricted to PubMed. Additionally, there may have been factors important to consider in completeness of reporting that we failed to capture in this study.
Pilot and feasibility trials of interventions in HD patients can help to inform the design of well-powered clinical trials to address critical challenges in HD today and prevent waste of resources on poorly designed trials [22, 24]. The pilot studies conducted in this patient population over the last two decades examine a wide range of interventions from pharmaceutical to behavioral; however, they were largely not preludes to larger trials. Improving the reporting completeness of these trials through promotion and endorsement of the CONSORT extension specific to pilot and feasibility trials and ensuring that pilot trials focus on feasibility objectives may improve the utility of these pilot trials. Such pilot studies are needed in HD patients to help inform and design interventional trials that are “…sufficiently robust to provide reliable answers and are not constrained by inappropriate complexities in design or conduct” (, p., 297).
Hill NR, Fatoba ST, Oke JL, Hirst JA, O’Callaghan CA, Lasserson DS, Hobbs FR. Global prevalence of chronic kidney disease–a systematic review and meta-analysis. PLoS One. 2016;11:e0158765.
Webster AC, Nagler EV, Morton RL, Masson P. Chronic kidney disease. Lancet. 2017;389:1238–52.
Jha V, Garcia-Garcia G, Iseki K, Li Z, Naicker S, Plattner B, Saran R, Wang AY, Yang CW. Chronic kidney disease: global dimension and perspectives. Lancet. 2013;382:260–72.
Couser WG, Remuzzi G, Mendis S, Tonelli M. The contribution of chronic kidney disease to the global burden of major noncommunicable diseases. Kidney Int. 2011;80:1258–70.
Damien P, Lanham HJ, Parthasarathy M, Shah NL. Assessing key cost drivers associated with caring for chronic kidney disease patients. BMC Health Serv Res. 2016;16:690.
Evangelidis N, Tong A, Manns B, Hemmelgarn B, Wheeler DC, Tugwell P, Crowe S, Harris T, Van Biesen W, Winkelmayer WC, Sautenet B. Developing a set of core outcomes for trials in hemodialysis: an international Delphi survey. Am J Kidney Dis. 2017;70:464–75.
Allon M, Lok CE. Dialysis fistula or graft: the role for randomized clinical trials. CJASN. 2010;5:2348–54.
Strippoli GF, Craig JC, Schena FP. The number, quality, and coverage of randomized controlled trials in nephrology. J Am Soc Nephrol. 2004;15:411–9.
Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, Giangregorio L, Goldsmith CH. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1.
Van Teijlingen E, Hundley V. The importance of pilot studies. Soc Res Update. 2010;35:49–59.
Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol. 2010;10:67.
Whitehead AL, Sully BG, Campbell MJ. Pilot and feasibility studies: is there a difference from each other and from a randomised controlled trial? Contemp Clin Trials. 2014;38:130–3.
Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, Lancaster GA. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ. 2016;355:i5239.
Bhatt M, Zielinski L, Sanger N, Shams I, Luo C, Bantoto B, Shahid H, Li G, Abbade LP, Nwosu I, Jin Y. Evaluating completeness of reporting in behavioral interventions pilot trials: a systematic survey. Res Soc Work Pract. 2018;28:577–84.
Alonso-Coello P, Carrasco-Labra A, Brignardello-Petersen R, Neumann I, Akl EA, Sun X, Johnston BC, Briel M, Busse JW, Glujovsky D, Granados CE, Iorio A, Irfan A, Martínez García L, Mustafa RA, Ramirez-Morera A, Solà I, Tikkinen KA, Ebrahim S, Vandvik PO, Zhang Y, Selva A, Sanabria AJ, Zazueta OE, Vernooij RW, Schünemann HJ, Guyatt GH. A methodological survey of the analysis, reporting and interpretation of Absolute Risk ReductiOn in systematic revieWs (ARROW): a study protocol. Syst Rev. 2013;2:113.
Kosa SD, Mbuagbaw L, Debono VB, Bhandari M, Dennis BB, Ene G, Leenus A, Shi D, Thabane M, Valvasori S, Vanniyasingam T, Ye C, Yranon E, Zhang S, Thabane L. Agreement in reporting between trial publications and current clinical trial registry in high impact journals: a methodological review. Contemp Clin Trials. 2018;65:144–50.
Mbuagbaw L, Thabane M, Vanniyasingam T, Borg Debono V, Kosa S, Zhang S, Ye C, Parpia S, Dennis BB, Thabane L. Improvement in the quality of abstracts in major clinical journals since CONSORT extension for abstracts: a systematic review. Contemp Clin Trials. 2014;38:245–50.
Rios LP, Odueyungbo A, Moitri MO, Rahman MO, Thabane L. Quality of reporting of randomized controlled trials in general endocrinology literature. J Clin Endocrinol Metab. 2008;93:3810–6.
Rios LP, Ye C, Thabane L. Association between framing of the research question using the PICOT format and reporting quality of randomized controlled trials. BMC Med Res Methodol. 2010;10:11.
Wilson B, Burnett P, Moher D, Altman DG, Al-Shahi Salman R. Completeness of reporting of randomised controlled trials including people with transient ischaemic attack or stroke: a systematic review. Eur Stroke J. 2018;20:337–46.
Mehrazmay A, Karambakhsh A, Salesi M. Reporting quality assessment of randomized controlled trials published in nephrology urology monthly journal. Nephrourol Mon. 2015;7:e28752.
Palmer SC, Sciancalepore M, Strippoli GF. Trial quality in nephrology: how are we measuring up? Am J Kidney Dis. 2011;58:335–7.
Kyriakos M, Chatzimanouil T, Wilkens L, Anders H. Quantity and reporting quality of kidney research. JASN. 2019;30:13–22.
Baigent C, Herrington WG, Coresh J, Landray MJ, Levin A, Perkovic V, Pfeffer MA, Rossing P, Walsh M, Wanner C, Wheeler DC. Challenges in conducting clinical trials in nephrology: conclusions from a Kidney Disease—Improving Global Outcomes (KDIGO) Controversies Conference. Kidney Int. 2017;92:297–305.
Several of the listed authors are members of the research arm of a volunteer organization, Kidney CARE Network International.
The authors received no financial support for the research and/or authorship of this article. Kidney CARE Network International supported the publication of this article.
Availability of data and materials
The dataset supporting the conclusions of this article is included in its additional file.
Ethics approval and consent to participate
As this a methodological review and does not involve human subjects, ethics board approval was not sought for this study.
Consent for publication
As this a methodological review and does not involve human subjects, consent for publication was not sought for this study.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Pilot clinical trials in chronic kidney disease patients on hemodialysis. The sample of pilot clinical trials in chronic kidney disease patients on hemodialysis included in the methodological survey. The data dictionary is included in the second tab. (XLS 158 kb)
About this article
Cite this article
Kosa, S.D., Monize, J., Leenus, A. et al. Reporting quality of pilot clinical trials in chronic kidney disease patients on hemodialysis: a methodological survey. Pilot Feasibility Stud 5, 53 (2019). https://doi.org/10.1186/s40814-019-0436-3