Skip to main content
Download PDF

Recommendations for progression criteria during external randomised pilot trial design, conduct, analysis and reporting

Pilot and Feasibility Studies volume 9, Article number: 59 (2023) Cite this article

Abstract

Background

External randomised pilot trials aim to assess whether a future definitive Randomised Controlled Trial (RCT) is feasible. Prespecified progression criteria help guide the interpretation of pilot trial findings to decide whether, and how, a definitive RCT should be conducted. This commentary presents a set of proposed recommendations for progression criteria to guide researchers when (i) designing, (ii) conducting, (iii) analysing and (iv) reporting external randomised pilot trials.

Methods

Recommendations were developed following a mixed methods approach. This involved (i) a methodological review of pilot trial publications, (ii) a cross-sectional study of pilot trial research funding applications, (iii) qualitative interviews with pilot trial researchers and (iv) a survey of corresponding authors of identified pilot trial publications. Initial recommendations were refined following two consultation stakeholder workshops held in July 2022.

Recommendations for progression criteria for external randomised pilot trials:

i. Design: consider progression criteria from the earliest opportunity; map progression criteria to feasibility objectives; consider quantitative and qualitative interpretations of feasibility; provide justification; develop guidelines rather than rules; seek input from relevant stakeholders.

ii. Conduct: regularly monitor pilot trial data against progression criteria.

iii. Analysis: avoid considering each progression criterion in isolation; engage in discussion with relevant stakeholders; consider context and other factors external to the pilot trial; consider feasibility (can we?) and progression (will we?).

iv. Reporting: we propose a reporting checklist in relation to progression criteria and recommend reporting in a table format for clarity.

Conclusion

These recommendations provide a helpful resource for researchers to consider progression criteria at different stages of external randomised pilot trials. We have produced a simple infographic tool to summarise these recommendations for researchers to refer to. Further research is needed to evaluate whether these proposed recommendations should inform future development, or update, of established guidelines for the design, conduct, analysis and reporting of external randomised pilot trials.

Peer Review reports

Background

RCTs are integral to the practice of evidence-based medicine [1], however they are often expensive, complicated, and take a long time to set up, conduct, analyse and disseminate findings [2,3,4,5]. Despite researchers’ best efforts to deliver gold standard RCTs, many RCTs do not adequately answer their research question [6, 7], resulting in research waste and inefficiencies.

Feasibility studies ask: can the future definitive trial be done, should we proceed with it, and if so, how? [8] Pilot trials are a type of feasibility study. They ask the same question, but also ‘pilot’ or conduct all or part of a future trial on a smaller scale [8]. External pilot trials are small stand-alone studies, where any outcome data that is collected does not contribute to the definitive trial analysis [8, 9]. This is the key difference from an internal pilot trial which is embedded within a definitive RCT forming the first phase, therefore any data collected does contribute to the definitive trial analysis [9, 10].

Progression criteria help researchers interpret their pilot trial findings to decide whether, and how, to proceed with a future definitive trial. It is important that progression criteria are specified before the pilot trial begins (a priori) to avoid introducing bias associated with establishing progression criteria once external pilot trial findings are known. If progression criteria are not set a priori, there is a risk that pilot trials may be optimistic in reporting that a future RCT is feasible [11] and progress to a definitive RCT without modification or acknowledgement of potential limitations [12].

Although progression criteria are now required by some research funders in pilot trial funding applications [13] and are specified as a reporting item in the CONSORT extension for external Pilot and Feasibility Studies (PAFS) [14], existing recommendations are largely based on individual case studies or have not been developed specifically with external randomised pilot trials in mind.

Using a mixed methods approach, we conducted four distinct but complimentary research studies to examine how progression criteria inform assessment of external randomised pilot trial feasibility. In this commentary we present our recommendations for best practice.

Development of recommendations

The four research studies that underpin these recommendations are reported in detail, or are currently under review, elsewhere. In study one, we examined the current reporting of progression criteria in a large representative sample of external randomised pilot trial protocol and result publications (published between January 2018 and December 2019) [15]. In study two, we examined the inclusion of progression criteria in research funding applications by conducting a cross-sectional study of progression criteria proposed in external randomised pilot trial funding applications submitted to NIHR Research for Patient Benefit (RfPB) (with a funding decision between July 2017 and July 2019) [16]. In study three, we explored pilot trial team member perceptions and experiences of developing and using progression criteria in practice by conducting a qualitative research study with key stakeholders (between December 2020 and July 2021) [17]. Finally, in study four, we examined whether identified pilot trials were considered feasible, and progressed to further research, by conducting a survey study (between January 2022 and February 2022) of corresponding authors of publications included in study one [18].

These four studies were conducted over a three-year period between 2019 and 2022. The findings were iteratively triangulated to identify areas where recommendations about best practice for progression criteria might be useful. The lead author (KM) drafted initial recommendations which were then refined by the wider study team and following feedback received from two external stakeholder meetings (July 2022). We wanted to capture a range of different perspectives on our initial recommendations from stakeholders with different trial roles. Stakeholders were invited following contribution, participation or engagement in earlier research [15,16,17,18] or were suggested by members of a wider Pilot and Feasibility Studies Working Group. External stakeholders included three trial investigators, four trial statisticians, six trial methodologists/health services researchers, one research funder manager and one professor of primary care pharmacy.

The final recommendations are structured to indicate considerations for progression criteria when (i) designing, (ii) conducting, (iii) analysing and (iv) reporting external randomised pilot trials. In addition an infographic tool has been produced to provide a helpful summary (See Fig. 1).

Fig. 1
figure 1

Infographic to summarise recommendations for progression criteria

Full size image

Recommendations for progression criteria

Recommendations for progression criteria during pilot trial design

Consider progression criteria from the earliest opportunity

Researchers should consider progression criteria from the onset when designing their pilot trial, and some research funders now require that progression criteria are included in pilot trial funding applications [13]. Early consideration of progression criteria provides a valuable opportunity for researchers to think about where their uncertainties lie and what problems that they might face, potentially saving time and effort in the long run.

Map progression criteria to feasibility objectives

Researchers should consider whether progression criteria are needed for each of their feasibility objectives. Too often progression criteria only focus on recruitment and retention and do not account for other feasibility issues that might also be pertinent to the success of the definitive RCT. For example, if researchers are conducting qualitative research as part of their pilot trial, they should consider whether, and how, these findings will inform progression criteria. Not all data collected during the external pilot trial needs to inform progression criteria, some might be collected to pilot and refine trial processes without contributing to progression decision making, for example to optimise efficiency of recruitment processes, training or intervention delivery. However, mapping progression criteria to feasibility objectives as appropriate can ensure that they are developed with the definitive trial in mind. To contextualise progression criteria researchers might also find it useful to associate criteria with specific timeframes or settings e.g. monthly or site-specific targets.

Consider quantitative and qualitative interpretations of feasibility

Although it has become standard practice to use numerical targets for progression criteria, researchers should be mindful that this might not always be appropriate. Although quantifiable targets might seemingly ensure transparent progression criteria assessment, they should be avoided where they are not meaningful.

Mixed method approaches to PAFS data collection have been recommended [19, 20]. Guidance for maximising qualitative research in feasibility studies, outlining feasibility questions that might be best addressed using qualitative research methods, approaches to qualitative data collection and analysis have previously been outlined [21]. Researchers might therefore opt to have a combination of numerical and non-numerical progression criteria where they have used qualitative research methods as part of their external pilot trial. An example is presented in Fig. 2 [22].

Fig. 2
figure 2

Example progression criteria including both quantitative and qualitative data. Reproduced from Hynes et al (2022) Pilot Feasibility Stud 8:1–16, published CC-BY 4.0

Full size image

Not every pilot trial will have a formally embedded qualitative research study e.g. a process evaluation. However, this does not negate the importance of considering other ‘soft intelligence’ gained from conducting the pilot trial which might be interpretated qualitatively, for example through ‘informal conversations’ with healthcare professionals at investigative sites [23]. These informal conversations can occur anywhere at any time and are likely already widely being used by research teams to generate new ideas about intervention or trial deliverability that are investigated by changing aspects of the pilot trial design. However, new strategies are needed to ensure that any data generated through informal conversations that informs and underpins changes to trial design does not go undocumented. One example is to maintain a ‘lessons learned’ document to capture problems faced during the pilot trial, any attempts to resolve these issues, and whether they worked or not. Bugge et al suggest that solutions to problems faced during the pilot trial might be identified through literature searching; debate/ brainstorming within the research team; and, if available, through analysis of existing feasibility study data [24].

Provide justification

Researchers should provide some justification for any progression criteria including any stated numerical progression criteria targets to indicate how they were derived. Rationale need not be statistical as pilot trials are usually underpowered for hypothesis testing, but some rationale for criteria should be given (e.g. based on feasibility objectives; clinical or contextual assumptions; pragmatically derived; based on previous feasibility or observational work; developed using consensus methods involving a range of stakeholders). Investigators should be aware that small pilot trial sample sizes might mean that any estimates of rates are subject to considerable uncertainty [25].

Develop guidelines rather than rules

Progression criteria are best viewed as guidelines rather than strict rules. Researchers should therefore develop progression criteria that will help identify and explore potential challenges with their trial design to inform the development of actionable solutions.

For example, it is becoming increasingly common for investigators to use a traffic light or Red Amber Green (RAG) approach for progression criteria. This approach is also recommended and widely used in RCTs with internal pilot trial phases [26, 27]. There are no agreed hard and fast rules for the meanings attributed to each colour, but typically measures below a lower (red) threshold have indicated that the pilot trial is not feasible [stop], above a higher (green) threshold that it is feasible [go], and between the two (amber) that it might be feasible if appropriate changes can be made [amend] [14]. However, for many external pilot trials a red criterion might not necessarily mean that the definitive trial is not feasible. Instead, it might be more appropriate to think of the RAG system as a way to highlight, and draw attention to, problems that have been faced in the pilot trial, with red indicating major problems that require urgent attention (and perhaps cannot be remedied), amber indicating minor problems that require attention, and green indicating areas of no concern.

Seek input from relevant stakeholders

Researchers should try to involve a broad range of stakeholders to develop, or agree, progression criteria so that targets are more meaningful and less prone to bias. A multidisciplinary approach to progression criteria development might involve consulting a Research Design Service (RDS) or Clinical Trials Unit (CTU). Seeking input from clinical colleagues who are implementing the pilot trial can help ensure that progression criteria make sense for different clinical contexts, recognising that pilot trial sites are not always reflective of the sites used in the main trial. Researchers might also want to involve Patient and Public Involvement (PPI) representatives to agree their progression criteria, and if applicable, agree progression criteria with their Trial Steering Committee (TSC) in advance, as is recommended for internal pilot trials [26]. Young et al have described using plain English explanations and a useful everyday analogy to facilitate understanding of progression criteria among patient and clinician groups when co-producing progression criteria [28]. Finally, researchers should also consider appropriate funding sources for their future definitive trial if they are able to demonstrate feasibility and ensure that progression criteria encompass any recommendations for feasibility assessment made by the intended definitive trial funder.

Recommendations for progression criteria during pilot trial conduct

Regularly monitor pilot trial data against progression criteria

Researchers might find it useful to revisit their progression criteria targets regularly throughout their pilot trial at important trial milestones, or as a standing agenda item at multidisciplinary team meetings e.g. the Trial Management Group (TMG) and/or TSC. This might be particularly important when any changes to the pilot trial design are made, so that researchers can determine whether these changes have improved the trial design (i.e. whether indicators of feasibility are improving or trending towards green progression criteria). Criteria that fall within the ‘red’ domain will signify where urgent attention is needed to identify, outline and pilot actionable solutions in response to problems that are being faced.

There might be instances where changes to the pilot trial design might directly or indirectly affect the applicability of the progression criteria. In these instances, researchers might re-define their progression criteria, following consultation with relevant stakeholders (e.g. those who initially input into their development and any newly identified stakeholders of relevance), to ensure that they are still usable indicators of trial feasibility. Where any changes to progression criteria are made, reasons for these changes should be fully reported in pilot trial publications. Note that unlike an internal pilot trial, any outcome data collected during an external pilot trial does not contribute to the definitive trial analysis [8].

Recommendations for progression criteria during pilot trial analysis

Avoid considering each progression criterion in isolation

Most pilot trials will have multiple progression criteria, so researchers should consider their method of multi-criteria assessment. Researchers might opt to take a holistic approach to feasibility assessment and consider criteria in relation to each other, the implications for the future definitive trial, and whether any solutions to poorly performing criteria have been identified to come to an overall conclusion of feasibility. This may or may not involve weighting specific criteria that are regarded as more fundamental than others. Outlining clear, actionable solutions where progression criteria are not met (e.g. are within red or amber ranges) is an important component of feasibility assessment that should not be overlooked. This is in line with recommendations for evaluating the feasibility of internal pilot trials which suggests that definitive trial funders acknowledge the importance of considering supplementary data e.g. a ‘rescue plan’ that outlines any problems encountered and how they were addressed [26]. An alternative approach suggested for multi-criteria assessment is to determine overall progression based on the worst-performing criterion (e.g. if one criterion is not met then the trial is not feasible) [29].

Engage in discussion with relevant stakeholders

Just as it is important to engage a broad range of stakeholders in developing progression criteria, it is important to engage different stakeholders at the end of the pilot trial when determining feasibility. At this stage it might be particularly useful to speak to people outside of the immediate trial team, e.g. clinical and healthcare professionals who were implementing the intervention or trial processes, to gain a comprehensive understanding about what might have worked well and what might require improvement in the definitive trial. For example, a behavioural science approach might be used to identify specific challenges and potential solutions [30].

Consider context and other factors external to the pilot trial

Since progression criteria are developed during the early pilot trial design stage, they do not account for unforeseen events and challenges researchers might face whilst conducting the pilot trial. Researchers should account for these factors, drawing on evidence and experience generated from doing the pilot trial, and any relevant external evidence generated outside of their pilot trial, when drawing conclusions about feasibility and progression.

Consider feasibility (can we?) and progression (will we?)

Researchers should recognise that feasibility assessment (can we do it?) and progression decision making (will we do it?) are complimentary but distinct considerations. There might be instances where pilot trials are considered feasible (i.e. based on progression criteria and assessment of deliverability) but do not progress for other reasons that are external to the pilot trial design (e.g. funding might not be available, the healthcare context might have changed, the intervention might now be superseded, or the CI might not intend to pursue the definitive trial at this time). It is important to be transparent about both whether the pilot trial is considered feasible, and whether researchers intend to progress to further research. Doing so might further evidence wider challenges with pilot trial progression for the research community to address, or even present the opportunity for other research groups to advance completed pilot trials that were feasible but might not otherwise progress.

Recommendations for progression criteria reporting

Suggestions for what information to report

Clear and transparent reporting of progression criteria ensures that trial feasibility is assessed with integrity and rigour. Clear reporting also enhances the wider usability of pilot trial findings. For example researchers can adequately determine whether progression criteria for one pilot trial might be applicable to another if information about how those criteria were developed is provided. This should encourage better research practice and avoid researchers providing generic progression criteria that are not meaningful.

To improve transparency of progression criteria reporting, we propose that the information detailed in Table 1 is included in external randomised pilot trial publications. The suggested items for protocol publications might also be applicable to researchers who are developing funding applications for external randomised pilot trials.

Table 1 Suggested reporting items and rationale
Full size table

Consider reporting in a table for clarity

To promote clarity researchers should consider reporting progression criteria and their associated findings in a table format. This is particularly useful for completed trials to report the progression criteria set, the corresponding finding and the implications for the future RCT design. GRADE (Grading of Recommendations, Assessment, Development and Evaluation) is a transparent approach to grading the quality of evidence to inform healthcare recommendations. GRADE recommends using Summary of Findings (SoF) tables to provide a concise summary of key information that underpins recommendations. This format of evidence synthesis and reporting strikes a balance between simplicity of information presentation and completeness (or transparency) [31], and some pilot trial publications have effectively reported pilot trial findings in a similar format [32,33,34,35]. One example is presented in Fig. 3 [35].

Fig. 3
figure 3

Example table to report progression criteria findings and implications for definitive RCT. Reproduced from Wiangkham et al (2019) PLoS One 14:e0215803, published CC-BY 4.0

Full size image

Conclusions

Progression criteria for external randomised pilot trials are becoming increasingly required in publications and funding applications; however it is unclear how they should be developed, assessed and reported. To our knowledge, these are the first recommendations developed for progression criteria across key stages of external randomised pilot trials. Although these recommendations were developed with external randomised pilot trials in mind, it is possible that they might also apply to non-randomised or single-arm pilot or feasibility studies that pre-specify progression criteria to inform assessment of trial feasibility. We hope that these practical recommendations will support researchers, funders, editors and peer reviewers to include and encourage the inclusion of clear and useful progression criteria in pilot trials, leading to improved feasibility assessment and reduced research waste. Further research is needed to evaluate whether these proposed recommendations should inform future development, or update, of established guidelines for the design, conduct, analysis and reporting of external randomised pilot trials.

Availability of data and materials

Data will be made available upon request and included in a DPhil thesis published open access through the Oxford University Archive upon KM’s DPhil completion.

Abbreviations

ADePT:

A process for Decision-making after Pilot and feasibility Trials

CI:

Chief Investigator

CONSORT:

Consolidated Standards of Reporting Trials

CReDECI:

Criteria for Reporting the Development and Evaluation of Complex Interventions in healthcare

CTU:

Clinical Trials Unit

PAFS:

Pilot and Feasibility Studies

PPI:

Patient and Public Involvement

RAG:

Red Amber Green

RCT:

Randomised Controlled Trial

RDS:

Research Design Service

RfPB:

Research for Patient Benefit

TMG:

Trial Management Group

TSC:

Trial Steering Committee

REFERENCES

  1. Akobeng AK. Understanding randomised controlled trials. Arch Dis Child. 2005;90:840–4. https://doi.org/10.1136/ADC.2004.058222.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Theme 6: the cost of randomised trials, trends and determinants - Clinical trial metadata: defining and extracting metadata on the design, conduct, results and costs of 125 randomised clinical trials funded by the National Institute for Health Research Health Technology Assessment programme - NCBI Bookshelf. https://www.ncbi.nlm.nih.gov/books/NBK274324/ (Accessed 1 Dec 2021).

  3. Ioannidis JPA. Clinical trials: What a waste. BMJ 2014;349. https://doi.org/10.1136/BMJ.G7089.

  4. Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997;315:640–5. https://doi.org/10.1136/bmj.315.7109.640.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Manzoli L, Flacco ME, D’Addario M, et al. Non-publication and delayed publication of randomized trials on vaccines: survey. BMJ 2014;348. https://doi.org/10.1136/BMJ.G3058.

  6. Chapman SJ, Shelton B, Mahmood H, et al. Discontinuation and non-publication of surgical randomised controlled trials: observational study. BMJ 2014;349. https://doi.org/10.1136/BMJ.G6870.

  7. Kasenda B, Von Elm E, You J, et al. Prevalence, characteristics, and publication of discontinued randomized trials. JAMA. 2014;311:1045–52. https://doi.org/10.1001/JAMA.2014.1361.

    Article  CAS  PubMed  Google Scholar 

  8. Eldridge SM, Lancaster GA, Campbell MJ, et al. Defining Feasibility and Pilot Studies in Preparation for Randomised Controlled Trials: Development of a Conceptual Framework. PLoS One. 2016;11:e0150205. https://doi.org/10.1371/journal.pone.0150205.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Bond C, Lancaster GA, Campbell M, et al. Pilot and feasibility studies: extending the conceptual framework. Pilot Feasibility Stud. 2023;9:1–10. https://doi.org/10.1186/S40814-023-01233-1/FIGURES/2.

    Article  Google Scholar 

  10. Wittes J, Brittain E. The role of internal pilot studies in increasing the efficiency of clinical trials. Stat Med. 1990;9:65–72 https://doi.org/10.1002/sim.4780090113. Accessed 6 Aug 2020.

    Article  CAS  PubMed  Google Scholar 

  11. Morgan B, Hejdenberg J, Kuleszewicz K, et al. Are some feasibility studies more feasible than others? A review of the outcomes of feasibility studies on the ISRCTN registry.  Pilot Feasibility Stud. 2021;71:1–8. https://doi.org/10.1186/S40814-021-00931-Y.

    Article  Google Scholar 

  12. Mbuagbaw L, Kosa SD, Lawson DO, et al. The reporting of progression criteria in protocols of pilot trials designed to assess the feasibility of main trials is insufficient: a meta-epidemiological study. Pilot feasibility Stud. 2019;5:120. https://doi.org/10.1186/s40814-019-0500-z.

    Article  PubMed  PubMed Central  Google Scholar 

  13. NIHR. Guidance on applying for feasibility studies. v1.0. 2017. https://www.nihr.ac.uk/documents/nihr-research-for-patient-benefit-rfpb-programme-guidance-on-applying-for-feasibility-studies/20474 Accessed 13 Nov 2019.

    Google Scholar 

  14. Eldridge SM, Chan CL, Campbell MJ, et al. CONSORT 2010 statement: Extension to randomised pilot and feasibility trials. BMJ. 2016;355:i5239.  https://doi.org/10.1136/bmj.i5239.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Mellor K, Eddy S, Peckham N, et al. Progression from external pilot to definitive randomised controlled trial: a methodological review of progression criteria reporting. BMJ Open. 2021;11:e048178. https://doi.org/10.1136/bmjopen-2020-048178.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Mellor K, Harwood J, Hejdenberg J, et al. Inclusion of progression criteria in external randomised pilot trials: a cross-sectional study of funding applications submitted to the NIHR Research for Patient Benefit Programme. Trials 2022;23. https://doi.org/10.1186/s13063-022-06868-8.

  17. Mellor K, Dutton SJ, Hopewell S, et al. How are progression decisions made following external randomised pilot trials? A qualitative interview study and framework analysis. Trials. 2022;23:1–15. https://doi.org/10.1186/S13063-022-06063-9/FIGURES/2.

    Article  Google Scholar 

  18. Mellor K, Dutton SJ, Hopewell S. Determining external randomised pilot trial feasibility in preparation for a definitive trial: a web-based survey of corresponding authors of external pilot trial publications. Trials 2023;24. https://doi.org/10.1186/s13063-022-06981-8.

  19. Donald G. A brief summary of pilot and feasibility studies: exploring terminology, aims, and methods. Eur J Integr Med. 2018;24:65–70. https://doi.org/10.1016/j.eujim.2018.10.017.

    Article  Google Scholar 

  20. Yardley L, Ainsworth B, Arden-Close E, et al. The person-based approach to enhancing the acceptability and feasibility of interventions. Pilot Feasibility Stud 2015;1. https://doi.org/10.1186/s40814-015-0033-z

  21. O’Cathain A, Hoddinott P, Lewin S, et al. Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers. Pilot Feasibility Stud. 2015;1:1–13. https://doi.org/10.1186/s40814-015-0026-y.

    Article  Google Scholar 

  22. Hynes L, Murphy AW, Hart N, et al. The MultimorbiditY COllaborative Medication Review And DEcision Making (MyComrade) study: a protocol for a cross-border pilot cluster randomised controlled trial. Pilot Feasibility Stud. 2022;8:73. https://doi.org/10.1186/s40814-022-01018-y.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Swain J, King B. Using Informal Conversations in Qualitative Research. 2022;21:1–10. https://doi.org/10.1177/16094069221085056.

    Article  Google Scholar 

  24. Bugge C, Williams B, Hagen S, et al. A process for Decision-making after Pilot and feasibility Trials (ADePT): development following a feasibility study of a complex intervention for pelvic organ prolapse. Trials. 2013;14:1. https://doi.org/10.1186/1745-6215-14-353.

    Article  Google Scholar 

  25. Netz Y, Lidor R, Ziv G. Small samples and increased variability-discussing the need for restricted types of randomization in exercise interventions in old age. Eur Rev Aging Phys Act. 2019;16:1–8. https://doi.org/10.1186/S11556-019-0224-3/TABLES/4.

    Article  Google Scholar 

  26. Avery KNL, Williamson PR, Gamble C, et al. Informing efficient randomised controlled trials: Exploration of challenges in developing progression criteria for internal pilot studies. BMJ Open 2017;7. https://doi.org/10.1136/bmjopen-2016-013537.

  27. Herbert E, Julious SA, Goodacre S. Progression criteria in trials with an internal pilot: an audit of publicly funded randomised controlled trials. Trials. 2019;20:1–9. https://doi.org/10.1186/s13063-019-3578-y.

    Article  Google Scholar 

  28. Young H, Goodliffe S, Madhani M, et al. Co-producing progression criteria for feasibility studies: a partnership between patient contributors, clinicians and researchers. Int J Environ Res Public Health. 2019;16:3756. https://doi.org/10.3390/ijerph16193756.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Lewis M, Bromley K, Sutton CJ, et al. Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! Pilot Feasibility Stud. 2021;7:40. https://doi.org/10.1186/s40814-021-00770-x.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Gillies K, Brehaut J, Coffey T, et al. How can behavioural science help us design better trials? Trials. 2021;22:1–7. https://doi.org/10.1186/S13063-021-05853-X/FIGURES/1.

    Article  Google Scholar 

  31. Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction - GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64:383–94. https://doi.org/10.1016/j.jclinepi.2010.04.026.

    Article  PubMed  Google Scholar 

  32. Purcell A, Buckley T, King J, et al. Eutectic mixture of local anaesthetics (EMLA®) as a primary dressing on painful chronic leg ulcers: a pilot randomised controlled trial. Pilot Feasibility Stud. 2018;4:123. https://doi.org/10.1186/s40814-018-0312-6.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Newlands RSN, Ntessalen M, Clark J, et al. Pilot randomised controlled trial of Weight Watchers® referral with or without dietitianled group support for weight loss in women treated for breast cancer: the BRIGHT (BReast cancer weIGHT loss) trial. Pilot Feasibility Stud. 2019;5:24. https://doi.org/10.1186/s40814-019-0405-x.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Sharma S, Jensen MP, Moseley GL, et al. Results of a feasibility randomised clinical trial on pain education for low back pain in Nepal: The Pain Education in Nepal-Low Back Pain (PEN-LBP) feasibility trial. BMJ Open. 2019;9:e026874. https://doi.org/10.1136/bmjopen-2018-026874.

  35. Wiangkham T, Duda J, Sayeed Haque M, et al. A cluster randomised, double-blind pilot and feasibility trial of an active behavioural physiotherapy intervention for acute whiplash-associated disorder (WAD)II. PLoS One. 2019;14:e0215803. https://doi.org/10.1371/journal.pone.0215803.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We would like to acknowledge the Pilot and Feasibility Studies working group who provided guidance and input throughout the research that underpins these recommendations, and Christine Bond (University of Aberdeen) who provided initial feedback on draft recommendations. We would also like to acknowledge the following stakeholders who attended the consultation workshop to provide critical feedback on draft recommendations: Kerry Avery (University of Bristol); Mike Bradburn (University of Sheffield); Katie Gillies (University of Aberdeen); Jennie Hejdenberg (NIHR RfPB); Carmel Hughes (Queen's University Belfast); Tianshu Liu (University of Oxford); Patricia Logullo (University of Oxford); Ioana Mariam (University of Oxford); Philippa Nicolson (University of Oxford); Akshay Shah (University of Oxford); Nikki Totton (University of Sheffield); Ruth Tunn (University of Oxford); Esther Williamson (University of Oxford); David Wright (University of Leicester).

Funding

This work was supported by Medical Research Council Doctoral Training Partnership funding [grant number MR/N013468/1] awarded to KM. The funder had no role in the design, conduct, analysis or reporting of this study.

Author information

Authors and Affiliations

  1. Oxford Clinical Trials Research Unit / Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, England

    Katie Mellor, Susan J Dutton & Sally Hopewell

  2. Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK

    Charlotte Albury

  3. Institute of Population Health Sciences, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK

    Sandra Eldridge

Authors
  1. Katie Mellor
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charlotte Albury
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Susan J Dutton
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sandra Eldridge
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sally Hopewell
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The development of these recommendations was conceived and designed as part of KMs doctoral thesis under the supervision of SH, SD, CA and SE. KM designed all studies that underpinned these recommendations with input from SH, SD, CA and SE. KM collected and analysed the data. KM developed initial recommendations with input from SH. KM organised and facilitated stakeholder meetings, with CA and SD as co-facilitators at the virtual meeting. KM refined and updated recommendations following stakeholder meetings. KM drafted the manuscript. All authors reviewed and commented on manuscript drafts. All authors approved the final manuscript.

Authors’ information

KM is a DPhil candidate at the University of Oxford investigating how progression criteria are used to inform external randomised pilot trial feasibility assessment and progression decision making. SH is a senior trial methodologist with expertise in the design, conduct and transparent reporting of randomised trials and systematic reviews. SH also contributed to the development of CONSORT guidelines for pilot randomised trials. CA is an experienced qualitative methodologist with experience of using qualitative methods in the context of clinical trials. SD is a senior trial statistician with expertise in statistics and clinical trial methodology. SE is a Professor of Biostatistics and led the work to produce the CONSORT guidelines for pilot randomised trials. SE also sits on the Editorial Board for Pilot and Feasibility Studies journal. KM, SE and SH all contribute to a Pilot and Feasibility Studies working group.

Corresponding author

Correspondence to Katie Mellor.

Ethics declarations

Ethics approval and consent to participate

Ethical approval was provided by the University of Oxford MS IDREC for individual studies conducted that informed this research: Study of RfPB applications [R74410/RE001]; Qualitative research study [R72039/RE001]; Survey study R78375/RE001. MS IDREC confirmed that ethical approval was not required for the stakeholder workshops that helped refine the final recommendations since this was an engagement activity (not research).

Consent for publication

Not applicable.

Competing interests

KM, SE and SH all contribute to a Pilot and Feasibility Studies working group. SE is on the Pilot and Feasibility Studies journal editorial board. SE and SH were involved in producing the CONSORT guidelines for pilot randomised trials.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mellor, K., Albury, C., Dutton, S.J. et al. Recommendations for progression criteria during external randomised pilot trial design, conduct, analysis and reporting. Pilot Feasibility Stud 9, 59 (2023). https://doi.org/10.1186/s40814-023-01291-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40814-023-01291-5

Keywords

Pilot and Feasibility Studies

ISSN: 2055-5784