Reporting quality of pilot clinical trials in chronic kidney disease patients on hemodialysis: a methodological survey

Abstract

Background

The conduct of high-quality pilot studies can help inform the success of larger clinical trials. Guidelines have been recently developed for the reporting of pilot trials.

Objective

This methodological survey evaluates the completeness of reporting in pilot randomized controlled trials in chronic kidney disease patients on hemodialysis (HD patients) and explores factors associated with better completion of reporting.

Methods

The authors searched Pubmed on July 1, 2018, for all pilot trials conducted in HD patients. Reporting quality was assessed against the 40-item Consolidated Standards of Reporting Trials (CONSORT) Extension for Pilot Trials. Study factors including year and country of publication, intervention, number of centers, type of funding, and journal endorsement of CONSORT were also examined.

Results

The mean number of items reported from the CONSORT extension for pilot trials across all included articles was 18.4 (standard deviation [SD] = 4.4). In the adjusted analysis, studies reported in later years (IRR = 1.026, 95% CI [1.018, 1.034], p < 0.001) and an increase of 20 persons in sample size (adjusted IRR = 1.021, 95% CI [1.010, 1.031], p < 0.001) were associated with a significantly higher number of CONSORT pilot items reported.

Conclusions

Current reporting completeness of pilot trials in HD patients is suboptimal. Endorsing the CONSORT extension specific to pilot and feasibility studies and ensuring that pilot trials focus on the feasibility objectives may improve reporting completeness of these trials.

Background

Chronic kidney disease (CKD) is a significant and growing global health problem, with a prevalence estimated to be between 11 to 13% [1]. CKD is defined as having a decreased kidney function for at least 3 months, regardless of the etiology, and has many serious complications such as uremia, volume overload, and hematologic and metabolic disturbances [2, 3]. Risk of mortality, largely due to cardiovascular disease, is significantly increased in the CKD population and increases as kidney function declines [2]. An estimated two million CKD patients with very little or no residual kidney function require kidney replacement therapy [4]. Kidney replacement therapy is very costly, consuming 6.7% of the total Medicare budget to care for less than 1% of the covered population. The costliest modality is hemodialysis (HD), estimated at over US$87,000 per year per patient [5]. Despite significant investment into research in the last 40 years, survival and quality of life of patients on HD remain low [6].

There are many pressing clinical questions in HD which require a definitive, well-powered randomized clinical trial (RCT) [7]. Health care providers and patients alike have identified the need for further research across a range of priorities in HD including how to best address vascular access problems, reduce fatigue, risk of mortality, and cardiovascular disease and to improve dialysis adequacy [6]. However, the immense quantity of information and resources required for the conduct of adequately powered RCTs across these clinical areas in HD can act as a barrier to their conduct [7, 8]. Pilot studies can facilitate designing such definitive RCTs by assessing feasibility of screening, recruitment, coordination and acceptability, safety, and fidelity of the intervention and the study protocol, as well as informing power calculations [7, 9, 10].

However, pilot trial methodology is often criticized for inadequacies, mainly critiquing the emphasis on hypothesis testing and the lack of criteria for evaluating feasibility [11, 12]. To address these issues, the Consolidated Standards of Reporting Trials (CONSORT) extension for reporting randomized pilot and feasibility trials was published in 2016, which builds on the statement published in 2010 [13]. The 2016 CONSORT extension, which will be utilized in this study, lays the groundwork for the reporting of pilot trials, as well as informs their design and implementation. This is expected to enhance the completeness and transparency in the reporting of pilot RCTs and establish a standardized approach to this area of research [13].

Until recently, standards for the reporting of pilot studies have been unavailable and it is likely that there are significant gaps and inconsistencies in the reporting of pilot studies in HD—as similarly identified in other areas of clinical research [14]. Identification of these gaps may not only help inform initiatives to improve reporting, but also potentially raise awareness among clinicians and research about pilot trial design and implementation. Using the newly published CONSORT extension for pilot trials, we undertook a methodological survey to assess reporting completeness among pilot trials investigating interventions in CKD patients on hemodialysis (HD patients) and explored factors associated with better completion of reporting. A methodologic review is a type of study designed to examine methodologic quality of a sample of articles, generally within a certain discipline or study design [14,15,16].

Methods

Study eligibility

Inclusion criteria: (1) the words “pilot” or “feasibility” used to describe its design, (2) randomized control trial, (3) examine interventions in HD patients, and (4) published in English.

Exclusion criteria: (1) single-arm observational pilot or feasibility studies, (2) quasi-randomized trials, and (3) studies among nonclinical or acute populations.

Search strategy

In order to survey the medical literature for pilot trials in HD patients, we searched Medline/Pubmed and included studies published before July 1, 2018, using the following strategy: (1) feasibility studies/ or pilot projects/; (2) pilot stud*.mp.; (3) exp Randomized Controlled Trial/; (4) randomized controlled trial.pt.; (5) controlled clinical trial.pt.; (6) randomized controlled trials.sh.; (7) random allocation.sh.; (8) double-blind method.sh.; (9) single-blind method.sh.; (10) 3 or 4 or 5 or 6 or 7 or 8 or 9; (11) (animals not humans).sh.; (12) 10 not 11; (13) 1 or 2; (14) 12 and 13; (15) exp Renal Dialysis/; (16) renal dialysis.tw.; (17) 15 or 16; (18) hemodialysis.tw.; (19) haemodialysis.tw.; (20) 17 or 18 or 19; (21) exp peritoneal dialysis/; and (22) 20 not 21.

Study selection

One author (SK) screened all titles and abstracts based on the inclusion and exclusion criteria and conducted a full-text review of randomly selected citations to assess eligibility. The full-text articles were then screened independently and in duplicate for eligibility by three teams of reviewers (AL and SL, DS and SV, and SS and SS).

Outcome measures

The primary outcome of this survey was the completeness of reporting of each of the items on CONSORT statement extension for randomized pilot and feasibility trials checklist, measured as a number and proportion of studies reporting each of the 40 items. The secondary outcome was the completeness of reporting of the CONSORT statement extension for randomized pilot and feasibility trials checklist, as measured by the total number of applicable items reported.

Data extraction

The Excel-based data extraction form was developed based on a previous methodological survey [16] and collected study characteristics including year and country of publication, sample size, number of sites, type of funding (i.e., industry, non-industry), type of intervention (i.e., pharmaceutical, behavioral/educational, dialysis technology/technique, nutritional supplements, vascular access technology/technique, and other non-pharmaceutical interventions), whether the manuscript explicitly stated the pilot study to be prelude to definitive study (i.e., yes, no), journal endorsement of CONSORT statement if the article was published after 2010 (i.e., yes, no), and reporting of individual items on the CONSORT extension for pilot and feasibility studies (Additional file 1). For the reporting of individual items on the CONSORT extension for pilot and feasibility studies, each item on the checklist was scored as either “reported” or “not reported,” indicating whether or not the article reported the appropriate information as per the criteria outlined in the original publication (Table 1). Exceptions to this are items 6c, 7b, 11a, 11b, 18, and 19a, all of which had “not applicable” as an option [13].

Table 1 Reporting of items on the CONSORT Extension for Pilot and Feasibility Trials

Data analysis

All statistical analyses were performed in SPSS version 25.

Descriptive statistics

The completeness of reporting was summarized using descriptive statistics percentages for the general characteristics and number of articles reporting each CONSORT statement item (for items 6c, 7b, 11a, 11b, 18, and 19a, the percentage was calculated based on the total number of studies for which the item was applicable). The mean, standard deviation, and range for the total number of CONSORT statement items reported, sample size, and number of sites were also calculated. To calculate the mean number of CONSORT items reported, “not applicable” responses to reporting items 6c, 7b, 11a, 11b, 18, and 19a were excluded.

Inferential statistics

We conducted a Poisson regression to explore factors, including year of publication, sample size, multisite study (yes or no), industry funding (yes or no), prelude to a definite trial (yes or no), and journal endorsement of CONSORT (yes or no), associated with completeness of reporting as measured by the number of reported CONSORT items (“not applicable” items were excluded). Based on previous research, we hypothesized that a later publication date [17], larger sample size [18], multisite study [16], industry funding [16, 18, 19], and journal endorsement of CONSORT [20] would be associated with better reporting. The results of the Poisson regression were reported as unadjusted and adjusted incidence rate ratios (IRR) including 95% confidence interval (CI) and p value (α = 0.05).

Journal of publication was also considered as a potential factor to include in the Poisson regression as some clustering has been observed in prior studies on reporting [16, 18]; however, no clustering within journals was noted in exploratory analyses, likely due to the wide breadth of journals in which the included studies were published, with no more than 10 studies published in the same journals (see characteristics of included studies below).

Results

Our initial search retrieved 593 records, of which 86 were included in the synthesis (see Fig. 1 for PRISMA diagram).

Fig. 1

PRISMA 2009 Flow Diagram

Characteristics of included studies

The studies examined a wide range of interventions (see Table 2) including pharmaceutical (33.7%, e.g., magnesium carbonate plus calcium acetate, oral cholecalciferol), behavioral/educational (19.8%, e.g., cycling, resistance, or cycling and resistance, chairside meditation), dialysis technology/technique (17.4%, e.g., equilibrated kt/v goal of 1.4, fx-e membrane), nutritional supplements (14.0%, e.g., fish oil), vascular access technology/technique (12.8%, e.g., dialysis needles placed 2.5 cm and then 5 cm apart, u clip anastomosis), and other non-pharmaceutical interventions (2.3%, i.e., acupuncture, low-intensity vibration device). The majority of the studies were single center (68.6%) and had a sample size of 50 or less (74.4%). The mean sample size was 44.0 (standard deviation (SD) = 55.1, range = 4–448) and number of sites was 2.2 (SD = 2.7, range = 1–15).

Table 2 Characteristics of included articles

The studies were conducted across the globe, including Africa (1.2%), Asia (15.1%), Australia/New Zealand (7.0%), Europe (27.9%), and North America (48.8%). The majority of studies were published since 2010 (60.5%). Among those published since 2010, 42.9% were published in journals that endorse the CONSORT statement. Only 17.4% of studies explicitly indicated in the manuscript that the study was a prelude to a definitive trial. The studies were published across a wide range of journals, with 10 studies published in J Ren Nutr, 8 in Am J Kidney Dis, 5 in Nephrol Dial Transplant, 4 in J Vasc Access, 3 in each of Clin J Am Soc Nephrol, J Nephrol, Nephrology (Carlton), and Ther Apher Dial, 2 in each of Int Urol Nephrol, Kidney Int, Pharmacotherapy, and PLoS One, with the remaining 39 studies published in different journals.

Evaluation of reporting quality based on CONSORT extension

The mean CONSORT reporting score across all included articles was 18.4 (SD = 4.4, minimum = 8, maximum = 29) out of a possible 34 items (6c, 7b, 11a, 11b, 18, and 19a were excluded as they had a not applicable option). Table 1 shows the level of reporting of individual CONSORT items. The items reported by the largest proportion of articles (top 10%) were “2b. Specific objectives or research questions for pilot trial” (97.7%); “4a. Eligibility criteria for participants” (93.0%); “12. Methods used to address each pilot trial objective whether qualitative or quantitative” (97.7%); and “22. Interpretation consistent with pilot trial objectives and findings, balancing potential benefits and harms, and considering other relevant evidence” (97.7%).

The most poorly reported items (bottom 10%) were “3b. Important changes to methods after pilot trial commencement (such as eligibility criteria), with reasons” (5.8%); “6b. Any changes to pilot trial assessments or measurements after the pilot trial commenced, with reasons” (3.5%); “6c. When applicable, explanation of any interim analyses and stopping guidelines” (7.9%); and “24. Where the pilot trial protocol can be accessed, if available” (8.1%).

Factors related to reporting of CONSORT extension items

Table 3 shows the unadjusted and adjusted IRRs for overall CONSORT reporting by study characteristics. In comparing the total number of reported CONSORT items by the prespecified study characteristics, studies reported in later years (adjusted IRR = 1.026, 95% CI [1.018, 1.034], p < 0.001) and an increase of 20 persons in sample size (adjusted IRR = 1.021, 95% CI [1.010, 1.031], p < 0.001) had a significantly higher number of CONSORT pilot items reported. After adjusting for other covariates and factors, the remaining study characteristics were not significantly associated with reporting completeness.

Table 3 Incidence rate ratios for the total number of CONSORT Pilot Trial Extension Items Reported

Discussion

In this systematic survey, one of the few to examine the completeness of reporting in pilot and feasibility RCTs [14], we found that the mean number of CONSORT extension items for pilot and feasibility studies was 18.4 out of a possible 34 applicable items. We did find, however, that there is a 2.6% increase in the number of CONSORT pilot items reported for each additional year of publication. Larger sample sizes were also associated with higher number of CONSORT pilot items reported; however, the observed effect was relatively small (e.g., there is a 2.1% increase in reporting completeness for an increase in sample size of 20 participants). The number of reported items may continue to improve over time, particularly with the 2016 publication of the CONSORT extension for pilot and feasibility RCTs. Indeed, the completeness of reporting full RCTs in nephrology has improved over time, though reporting quality issues for certain CONSORT (2010) items such as clinical trial design, mode of randomization, and intention-to-treat analysis persist [8, 21,22,23]. Lack of available guidelines prior to the CONSORT extension and lack of its awareness (only recently published) could be contributing to this suboptimal reporting [13]. These guidelines may help clarify the purpose and reporting of pilot studies in HD patients.

In our sample of pilot studies, we found that the reporting of some items was consistently reported. These items are very similar to those included in the 2010 CONSORT statement, and include specific objectives for the pilot trial, eligibility criteria for participants, methods used to address each pilot trial objective whether qualitative or quantitative, and the interpretation is consistent with pilot trial objectives and findings. However, certain items specific to pilot trials and clinical trial conduct transparency were reported in less than 10% of included studies which, as previously indicated, may be attributable to lack of reporting guidelines for pilot trials until recently. These include changes to methods and assessments or measurements after pilot trial commencement, progression criteria for a future definitive trial, and where the pilot trial protocol can be accessed. These items are critical to the development of a larger, definitive trial that is rigorously designed and well powered, which should be the primary purpose of a pilot study [9]. The low levels of reporting for these items is consistent with our finding that only 17.4% of included studies indicated that they were a prelude to a larger trial, as well as the findings of a previous study on reporting completeness in pilot trials in behavioral interventions (13%) [14]. The manuscripts may not have explicitly stated the pilot study to be a prelude to a definitive study (CONSORT 2a and 22a) as this might be assumed by authors given that it is the purpose of a pilot study; however, this should be explicitly reported for every study as recommended by the guidelines.

Many of the included studies were primarily designed to address questions of clinical efficacy of a wide range of interventions. Given the relatively small sample of the majority of the included trials, these studies were likely underpowered to do so. Though some items were well reported, going forward, it is critical that journals publishing pilot trials in HD patients ensure that these studies adhere to the pilot trial extension to the CONSORT reporting statement, as well as confirm that the primary objectives of these studies are related to feasibility—not efficacy—objectives. This may be accomplished as part of the peer review process or as a requirement of submission.

There are several limitations of this study that are important to acknowledge. We only included English-language studies due to feasibility purposes, which may limit the generalizability of these findings. As this was a methodological survey and not a systematic review, this study’s search was restricted to PubMed. Additionally, there may have been factors important to consider in completeness of reporting that we failed to capture in this study.

Conclusions

Pilot and feasibility trials of interventions in HD patients can help to inform the design of well-powered clinical trials to address critical challenges in HD today and prevent waste of resources on poorly designed trials [22, 24]. The pilot studies conducted in this patient population over the last two decades examine a wide range of interventions from pharmaceutical to behavioral; however, they were largely not preludes to larger trials. Improving the reporting completeness of these trials through promotion and endorsement of the CONSORT extension specific to pilot and feasibility trials and ensuring that pilot trials focus on feasibility objectives may improve the utility of these pilot trials. Such pilot studies are needed in HD patients to help inform and design interventional trials that are “…sufficiently robust to provide reliable answers and are not constrained by inappropriate complexities in design or conduct” ([24], p., 297).