The curious case of an internal pilot in a multicentre randomised trial—time for a rethink?
© The Author(s). 2016
Received: 22 July 2016
Accepted: 22 November 2016
Published: 13 December 2016
Multicentre randomised trials are complex projects with many operational uncertainties. The embedding of a formal check upon study progress and viability at a pre-specified time point (sometimes referred to as an ‘internal pilot’) is becoming increasingly common within multicentre pragmatic randomised trials. However, it is worth considering this practice. We argue that most, if not all, multicentre trials have reassessment of the recruitment strategy and study processes whilst the study is running. Additionally, we propose discontinuation of the ‘internal/external pilot study’ terminology. Instead, we suggest for an alternative taxonomy along with greater recognition of the process of refinement which routinely occurs in trials and transparent reporting of it.
KeywordsRandomised trial Pilot study Recruitment Monitoring Multicentre Feasibility
Multicentre randomised trials are complex projects with many operational uncertainties particularly related to recruitment of participants, set-up of sites, timing and practicalities of randomisation, and data collection. Whilst, very valuable, no amount of preliminary (feasibility, piloting) work can fully remove or anticipate all problems. Given this, a check upon progress and review of problems when the study is at an early stage has much to commend it. The embedding of a formal check upon study viability at a pre-specified time point (sometimes referred to as an ‘internal pilot study’) is becoming increasingly common within large multicentre pragmatic randomised trials [1–5]. Based upon a review of the website of one research programme, around a half of trials funded in 2015 had an ‘internal pilot’ . ‘Internal pilots’ can have a somewhat ethereal nature to their existence, and they may not be explicitly reported in published trial report ; though with the publication of trial protocols, they are becoming more readily identified .
Funders naturally want to ensure funds are used efficiently and may request a formalised assessment of recruitment early in a trial’s lifecycle to identify problems and potentially curtail trials that are failing [2, 5]. However, it is worth considering this practice. On a simple level, the increasingly common labelling of this as an ‘internal pilot study’ is, in our view, unhelpful. A pilot in the natural sense of the word has to be external to the thing to which it is a pilot for. In practice, the use (or not) of the pilot data in the analysis of the trial appears to be the main dividing line between an ‘internal’ and ‘external’ pilot study . This terminology appears to have originated from the desire to include a formal check of key assumptions underlying the sample size calculation [3, 4, 8]. At the more substantive level, it is important to recognise that most, if not all, multicentre trials undergo some modification, and also have regular reassessment of the recruitment strategy, during their life course. Typically modifications relate to the recruitment strategy and the data collection process though other changes, including adjusting the sample size or replacing the primary outcome, are not that uncommon [3–5]. Perhaps some industry-funded drug trials are the exception where identical study processes can be adopted from a previous study, and the only distinguishing feature is the active compound. There is though, a sense that all trials are unique in the specific combination of methodology, location, timing and personnel. All trials have their own ‘story’. This is particularly true of pragmatic or comparative effectiveness trials, which tend to be intentional ‘one-offs’. Many, if not all, of the features of an internal pilot will go on in most (if not all) multicentre trials with only the formal assessment at a specified time the distinguishing feature. Is there ever a trial conducted without consideration of the recruitment progress against expectations, or a review of the data collection process? Indeed, this is one of the responsibilities of trial management groups, and similarly, funders may undertake periodic checks on progress. We note that the timing of a one-off assessment and the specification of stop-go recruitment criteria (typically of the form of X centres having recruiting at least one participant and/or a total of Y participants recruited by month Z) are difficult to get right. The desire for explicit and applicable criteria needs to be balanced against the multifaceted nature of the decision to progress or not. Relevant considerations include factors external to the trial (e.g. availability of wider resources to support a study with serious recruitment problems). To recognise this is not to deny the value of a planned assessment, more the recognition that it is difficult to anticipate and to formalise all the eventualities and considerations which go into making decisions relating to what is a complex, and in some regards, unique project. Furthermore, regular monitoring of the study process is a fundamental part of good trial science. It is not without reason that trial co-ordination and management are increasingly being recognised for its necessity, expertise and value, and as an emerging area for research . Reports of trial results however tend to downplay this perhaps due to fears of how the study might be viewed.
Approaches to developing and modifying the trial design with particular reference to the recruitment strategy
When to use
Substantial uncertainty about whether it is possible or how to implement aspects of the trial design and conduct (e.g. delivery of the interventions and recruitment process)
Exploration of the feasibility of a trial
Assessment of the feasibility of a trial
Limited refinement of the trial design, processes and recruitment strategy anticipated
Refinement of trial design and processes prior to commencement of the main trial
Minor modifications to trial design and confirmation of practicality
Early in the recruitment period of the main trial
Substantial adjustment of recruitment strategy and study processes is considered likely
One-off within study assessment of recruitment and study processes to allow adjustment of the strategy (including the possibility of early cessation of recruitment)
Single reassessment of the recruitment and data collection strategy
Throughout the recruitment period of the main trial
Minor modifications to study processes and recruitment strategy anticipated
Regular monitoring to allow periodic within study modification
Periodic updating of the recruitment strategy and study processes
Refinement of study processes and reassessment of the recruitment strategy are key elements of most, if not all, multicentre randomised trials. There is a sense in which most have features of an ‘internal pilot study’. We recommend discontinuation of the terminology of ‘internal’ and ‘external’ pilot studies as it is in our view unhelpful. There should be greater recognition of the process of refinement which occurs routinely in trials, increased clarity about any planned formal assessment, and more transparent reporting of this. We have suggested an alternative way to think about this along with a proposed taxonomy. Further research in this area is needed.
Acceptance checklist for clinical effectiveness pilot trials
The authors would like to acknowledge helpful discussions on this topic held with various colleagues in connection with a variety of projects.
No funding was received for this work.
JAC conceived the idea and drafted the initial version of the manuscript. DJB, JRC and GSM commented on drafts and approved the final version. All authors read and approved the final manuscript.
The authors declare that they have no competing of interests.
Consent for publication
Consent for publication was not required. All data referred to is available in the public domain.
Ethics approval and consent to participate
No ethical approval was needed for this work.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Lancaster G. Pilot and feasibility studies come of age! BMC Pilot and Feasibility studies. 2015;1:1.View ArticleGoogle Scholar
- NIHR HTA Programme. Progression rules for internal pilot studies for HTA trials. http://www.nets.nihr.ac.uk/__data/assets/pdf_file/0018/115623/Progression_rules_for_internal_pilot_studies.pdf [Accessed 22 June 2016]
- Wittes J, Brittan E. The role of internal pilot studies in increasing the efficiency of clinical trials. Stat Med. 1990;9:65–72.View ArticlePubMedGoogle Scholar
- Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, Giangregorio L, Goldsmith CH. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1.View ArticlePubMedPubMed CentralGoogle Scholar
- Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol. 2010;10:67.View ArticlePubMedPubMed CentralGoogle Scholar
- Andrews PJD, Avenell A, Noble DW, Campbell MK, Croal BL, Simpson WG, et al. Randomized trial of glutamine and/or selenium supplemented parenteral nutrition for critically ill patients (SIGNET). BMJ. 2011;342:d1542.View ArticlePubMedGoogle Scholar
- Watson G, O’Hara J, Carding P, Lecouturier J, Stocken D, Fouweather T, et al. Community occupational therapy for people with dementia and family carers (COTiD-UK) versus treatment as usual (Valuing Active Life in Dementia [VALID] programme): study protocol for a randomised controlled trial. Trials. 2016;17:175.View ArticlePubMedPubMed CentralGoogle Scholar
- Lu K. Distribution of the two-sample t-test statistic following blinded sample size re-estimation. Pharm Stat. 2016. doi:10.1002/pst.Google Scholar
- Farrell B, Kenyon S, Shakur H. Managing clinical trials. BMC Trials. 2010;11:78.View ArticleGoogle Scholar
- Charlesworth G, Burnell K, Hoe J, Orrell M, Russell I. Acceptance checklist for clinical effectiveness pilot trials: a systematic approach. BMC Med Res Methodol. 2013;13:78.View ArticlePubMedPubMed CentralGoogle Scholar