The curious case of an internal pilot in a multicentre randomised trial—time for a rethink?
Pilot and Feasibility Studies volume 2, Article number: 73 (2016)
Multicentre randomised trials are complex projects with many operational uncertainties. The embedding of a formal check upon study progress and viability at a pre-specified time point (sometimes referred to as an ‘internal pilot’) is becoming increasingly common within multicentre pragmatic randomised trials. However, it is worth considering this practice. We argue that most, if not all, multicentre trials have reassessment of the recruitment strategy and study processes whilst the study is running. Additionally, we propose discontinuation of the ‘internal/external pilot study’ terminology. Instead, we suggest for an alternative taxonomy along with greater recognition of the process of refinement which routinely occurs in trials and transparent reporting of it.
Multicentre randomised trials are complex projects with many operational uncertainties particularly related to recruitment of participants, set-up of sites, timing and practicalities of randomisation, and data collection. Whilst, very valuable, no amount of preliminary (feasibility, piloting) work can fully remove or anticipate all problems. Given this, a check upon progress and review of problems when the study is at an early stage has much to commend it. The embedding of a formal check upon study viability at a pre-specified time point (sometimes referred to as an ‘internal pilot study’) is becoming increasingly common within large multicentre pragmatic randomised trials [1–5]. Based upon a review of the website of one research programme, around a half of trials funded in 2015 had an ‘internal pilot’ . ‘Internal pilots’ can have a somewhat ethereal nature to their existence, and they may not be explicitly reported in published trial report ; though with the publication of trial protocols, they are becoming more readily identified .
Funders naturally want to ensure funds are used efficiently and may request a formalised assessment of recruitment early in a trial’s lifecycle to identify problems and potentially curtail trials that are failing [2, 5]. However, it is worth considering this practice. On a simple level, the increasingly common labelling of this as an ‘internal pilot study’ is, in our view, unhelpful. A pilot in the natural sense of the word has to be external to the thing to which it is a pilot for. In practice, the use (or not) of the pilot data in the analysis of the trial appears to be the main dividing line between an ‘internal’ and ‘external’ pilot study . This terminology appears to have originated from the desire to include a formal check of key assumptions underlying the sample size calculation [3, 4, 8]. At the more substantive level, it is important to recognise that most, if not all, multicentre trials undergo some modification, and also have regular reassessment of the recruitment strategy, during their life course. Typically modifications relate to the recruitment strategy and the data collection process though other changes, including adjusting the sample size or replacing the primary outcome, are not that uncommon [3–5]. Perhaps some industry-funded drug trials are the exception where identical study processes can be adopted from a previous study, and the only distinguishing feature is the active compound. There is though, a sense that all trials are unique in the specific combination of methodology, location, timing and personnel. All trials have their own ‘story’. This is particularly true of pragmatic or comparative effectiveness trials, which tend to be intentional ‘one-offs’. Many, if not all, of the features of an internal pilot will go on in most (if not all) multicentre trials with only the formal assessment at a specified time the distinguishing feature. Is there ever a trial conducted without consideration of the recruitment progress against expectations, or a review of the data collection process? Indeed, this is one of the responsibilities of trial management groups, and similarly, funders may undertake periodic checks on progress. We note that the timing of a one-off assessment and the specification of stop-go recruitment criteria (typically of the form of X centres having recruiting at least one participant and/or a total of Y participants recruited by month Z) are difficult to get right. The desire for explicit and applicable criteria needs to be balanced against the multifaceted nature of the decision to progress or not. Relevant considerations include factors external to the trial (e.g. availability of wider resources to support a study with serious recruitment problems). To recognise this is not to deny the value of a planned assessment, more the recognition that it is difficult to anticipate and to formalise all the eventualities and considerations which go into making decisions relating to what is a complex, and in some regards, unique project. Furthermore, regular monitoring of the study process is a fundamental part of good trial science. It is not without reason that trial co-ordination and management are increasingly being recognised for its necessity, expertise and value, and as an emerging area for research . Reports of trial results however tend to downplay this perhaps due to fears of how the study might be viewed.
Where can we go from here? First, our proposal is to stop using the prefixes ‘internal’ and ‘external’; true pilot studies are always external. A trial which incorporates a single-planned check of a key aspect of the study (such as recruitment) could instead refer to a ‘stop-go’ assessment or phase. In Table 1, we suggest an alternative taxonomy of approaches for developing and modifying randomised trial design with particular reference to the recruitment strategy. Clarification of when to use each approach and what the main output is also given. We note that the routine monitoring and stop-go assessment approaches are not mutually exclusive options, and that the latter is unlikely to be carried out without some form of the former. Second, trial protocols and reports should be more transparent about the potential and actual modification of trial processes and important design features (e.g. changes to outcomes), and whether it was a part of a pre-planned refinement process or a response (very likely a legitimate one) to an unforeseen event. Due to the difficulties of a one-off assessment, the ‘routine monitoring’ approach may be the most appropriate within-trial approach, and in our view better reflects what often occurs even when a stop-go assessment is specified. The ACCEPT guidance offers a helpful structure to think through where uncertainties lie when deciding upon which approaches to adopt . Third, there is an underdeveloped area of refinement of methodology and processes during the life course of a trial which needs exploration and development. BMC Pilot and feasibility studies is the natural home for publishing the findings of such work.
Refinement of study processes and reassessment of the recruitment strategy are key elements of most, if not all, multicentre randomised trials. There is a sense in which most have features of an ‘internal pilot study’. We recommend discontinuation of the terminology of ‘internal’ and ‘external’ pilot studies as it is in our view unhelpful. There should be greater recognition of the process of refinement which occurs routinely in trials, increased clarity about any planned formal assessment, and more transparent reporting of this. We have suggested an alternative way to think about this along with a proposed taxonomy. Further research in this area is needed.
Acceptance checklist for clinical effectiveness pilot trials
Lancaster G. Pilot and feasibility studies come of age! BMC Pilot and Feasibility studies. 2015;1:1.
NIHR HTA Programme. Progression rules for internal pilot studies for HTA trials. http://www.nets.nihr.ac.uk/__data/assets/pdf_file/0018/115623/Progression_rules_for_internal_pilot_studies.pdf [Accessed 22 June 2016]
Wittes J, Brittan E. The role of internal pilot studies in increasing the efficiency of clinical trials. Stat Med. 1990;9:65–72.
Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, Giangregorio L, Goldsmith CH. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1.
Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol. 2010;10:67.
Andrews PJD, Avenell A, Noble DW, Campbell MK, Croal BL, Simpson WG, et al. Randomized trial of glutamine and/or selenium supplemented parenteral nutrition for critically ill patients (SIGNET). BMJ. 2011;342:d1542.
Watson G, O’Hara J, Carding P, Lecouturier J, Stocken D, Fouweather T, et al. Community occupational therapy for people with dementia and family carers (COTiD-UK) versus treatment as usual (Valuing Active Life in Dementia [VALID] programme): study protocol for a randomised controlled trial. Trials. 2016;17:175.
Lu K. Distribution of the two-sample t-test statistic following blinded sample size re-estimation. Pharm Stat. 2016. doi:10.1002/pst.
Farrell B, Kenyon S, Shakur H. Managing clinical trials. BMC Trials. 2010;11:78.
Charlesworth G, Burnell K, Hoe J, Orrell M, Russell I. Acceptance checklist for clinical effectiveness pilot trials: a systematic approach. BMC Med Res Methodol. 2013;13:78.
The authors would like to acknowledge helpful discussions on this topic held with various colleagues in connection with a variety of projects.
No funding was received for this work.
JAC conceived the idea and drafted the initial version of the manuscript. DJB, JRC and GSM commented on drafts and approved the final version. All authors read and approved the final manuscript.
The authors declare that they have no competing of interests.
Consent for publication
Consent for publication was not required. All data referred to is available in the public domain.
Ethics approval and consent to participate
No ethical approval was needed for this work.
About this article
Cite this article
Cook, J.A., Beard, D.J., Cook, J.R. et al. The curious case of an internal pilot in a multicentre randomised trial—time for a rethink?. Pilot Feasibility Stud 2, 73 (2016). https://doi.org/10.1186/s40814-016-0113-8