- Open Access
Raising the profile of pilot and feasibility studies in relation to the development, evaluation and implementation of patient-reported outcome measures
Pilot and Feasibility Studiesvolume 4, Article number: 3 (2017)
The Erratum to this article has been published in Pilot and Feasibility Studies 2017 3:48
This editorial introduces a new special series on the pilot and feasibility testing of patient-reported outcome measures (PROMs) in the on-line open access journal Pilot and Feasibility Studies. Pilot and feasibility studies are typically implemented to address issues of uncertainty before undertaking a larger definitive study such as a randomised controlled trial or large scale survey. This editorial considers the role that such pilot and feasibility testing plays in relation to the development, evaluation and implementation of PROMs. This is often an essential element of PROM research but is typically overlooked—especially within current methodological guidance, reporting space and also debate. This editorial aims to open up a dialogue about the role of pilot and feasibility testing in relation to PROMs. It highlights some of the areas in PROMs research where these types of studies have been carried out and discusses the ways in which the PROM community may be better supported and encouraged to integrate this element of the research process into their PROM-based work.
The application of social science methods in the evaluation of medical care has continued to grow in importance. In particular, there is an increasing demand for the design, development and implementation of questionnaires that can assess patients’ experiences of health and illness. These questionnaires are typically referred to as patient-reported outcome measures (PROMs) or patient-reported outcomes (PROs). They are questionnaires designed to provide a means of measuring the impact of illness, and its associated treatments, or other types of intervention, from the patient’s perspective.
Traditionally, medical care was evaluated using clinical measures of outcome, i.e. measures of mortality and other clinical diagnostic criteria which concentrated upon the physical components of health and ignored the dimensions of well-being and functioning, which could have an impact upon the health status of the patient . However, in the latter half of the twentieth century, there was increasing awareness that health and illness are not purely dependent upon physical well-being. In 1954, the World Health Organization (WHO) emphasised this point in their definition of health as “a state of complete physical mental and social well-being and not merely the absence of disease or infirmity” . Also, in the past, the evaluation of patients’ experiences of health and illness was primarily based upon the objective judgements of clinicians. It has been suggested that these judgements were often based upon intuition and personal experience . However, research has shown that clinical and other such proxy reports made on behalf of a patient (e.g. from parents or carers) are far from objective and show variations and low levels of agreement to those of the patient [4,5,6]. As a result, there has been a growing demand to assess and evaluate the other dimensions of well-being which can impact upon the health of patients and to develop measurement tools in the form of questionnaires which can evaluate systematically this subjective impact on well-being beyond the traditional measures of outcome such as mortality or morbidity .
Types and uses of PROMs
There are a large number of PROMs available, and they may differ in their measurement properties, content length and intended purpose. However, typically, they can be categorised as being generic, disease or condition specific. These may also have utility (preference) values estimated for the responses and therefore become preference-based measures. These are used for calculating quality-adjusted life years (QALYs), allowing for the economic value of interventions to be assessed . More recently, with the increasing drive to capture data as part of routine care and find cost-effective, time efficient ways of routinely capturing the impact of treatments and illnesses from the patient’s perspective, there has been a rapid rise in ePROMs and a move away from the traditional mode of paper completions to electronic/web-based solutions. This is evident across a large and diverse range of medical disciplines  including pre-operative assessment , surgery  and cancer  to name a few.
PROMs have an important role in evaluative research as a measure of outcome. This particularly applies to the interpretation of RCT data where their use can provide additional information on the benefits of medical therapies or interventions, as an aid to clinical decision-making . A second related factor is their use in quality assurance and audit . Thirdly, they can assess the health care needs of populations by being used in surveys to capture information on the health needs of populations beyond the traditional mortality and socio-demographic data which are not specific enough to inform decision makers about the allocation of resources .
Most importantly, though, as PROMs are all concerned with providing information on the things which matter most to the patient, they can also provide valuable information to the clinician or other health care professional about patients’ progress. This can aid in the clinical management of the patient by enabling physicians to monitor patients’ progress and consequently influence decisions about treatment. Finally, an important use which is frequently overlooked is that completing a PROM also provides the opportunity for the patient to express the impact of the illness upon his/her well-being. For example, when PROMs have been administered in routine clinical practice, it has been found that patients appreciated the opportunity to report how they were feeling and to be involved with their care .
The use of PROMs in pilot and feasibility studies
Thus, within medicine and the related disciplines, PROMs clearly have a number of important and useful roles and their use is becoming more widespread—particularly with the drive to incorporate these more routinely into clinical practice and medical decision-making [17,18,19]. For example, since 2009, the NHS has made it a requirement to collect PROM data from patients before and after surgery in four surgical conditions: hip replacement, knee replacement, varicose veins and groin hernia, with it recently reported that there were plans to extend the PROMs programme over a wider range of long-term conditions and treatments in the NHS, including mental health, chronic obstructive pulmonary disease (COPD), diabetes, epilepsy, heart failure and stroke .
Guidelines for conducting pilot and feasibility studies have been published by both the Medical Research Council (MRC) and National Institute for Health Research (NIHR) , with the MRC reporting that pilot and feasibility testing are interchangeable concepts covering all aspects of preparatory work in their guidelines on complex interventions (www.mrc.ac.uk/complexinterventionsguidance). The pilot and feasibility testing of PROMs often plays an essential role. For example, in terms of the evaluation and implementation of the DH PROM programme the measures were only selected by the Department of Health following testing in numerous pilot studies  based upon a process which involved “piloting their use and reviewing their potential to be rolled out across the NHS” . Similar projects have also been or are currently underway to evaluate the feasibility of implementing PROMs into current practice. For example, one initiative is “The cardiac revascularisation PROMs pilot” . This was originally commissioned by the Department of Health in 2011 but later passed to NHS England in 2013. Patients across 11 English NHS hospital trusts who were on a waiting list for cardiac revascularisation to treat their heart disease were invited to participate in the PROM pilot. The aim of this was to evaluate if it was possible for the NHS to collect good enough data pre- and post treatment. A few other similar feasibility studies have been published which have particularly focused on determining the feasibility of implementing PROM/s and estimating response rates—although other important aspects such as gaining stakeholder and service user feedback, exploring responsiveness and estimating costs associated with the PROM completions were also some key aims of the pilot stage [23,24,25].
However, less guidance and debate exists on the ways that pilot and feasibility testing can be integrated into all aspects of PROM research. For example, although there is no single correct way to develop a PROM measure, the Food and Drug Administration (FDA) provided a figure to summarise and describe the iterative process that can be involved during PROM development . Within this process, the development of PROMs involves five overarching stages where pilot and feasibility testing could play an important methodological role ((1) Hypothesize Conceptual Framework, (2) Adjust Conceptual Framework and Draft Instrument, (3) Confirm Conceptual framework and Assess other measurement properties, (4) Collect, Analyse and Interpret Data, and (5) Modify instrument). However, it is only in relation to stage 2 and establishing content validity and confirming the conceptual framework of a PROM that the FDA specifically recommends the importance of undertaking pilot studies. They recommend there should be an examination of:
all items and procedures in a pilot test of whether patients understand the items and instructions included in the PRO instrument .
The FDA recommends cognitive interviewing as one such way of carrying out this assessment and undertaking other small pilot studies to test the face validity of the measure (e.g. that response options and recall periods are appropriately comprehended and that the instrument’s readability is adequate for the intended population). However, more guidance on what the nature of these pilot and feasibility studies may entail is not provided. Similar gaps are evident in other key user manuals and checklists within the PROM field, e.g. COSMIN [27, 28]. It is unclear in these international guidelines how pilot and feasibility studies may be used to support the methodological quality of studies that are focused on PROM development, evaluation and implementation, or if the same recommendations as they currently stand in these documents translate to the pilot and feasibility testing of PROMs also.
Despite this, it appears as though researchers are using their own initiatives to incorporate pilot and feasibility testing. By no means an exhaustive list, some examples of these initiatives include (i) developing questionnaires to determine the relevance and acceptability of PROMs during aspects of face validity ; (ii) testing search strategies in systematic reviews of PROM measures and literature ; (iii) carrying out a pilot study to identify the domain structure of a measure and establishing the psychometric properties of the instrument (e.g. as part of demonstrating aspects of reliability, validity and responsiveness) , establishing other key aspects such as costs and generating stakeholder feedback [23,24,25]; and (iv) piloting the PROM as part of a feasibility study to inform the design of a larger definitive randomised control trial . With the rise in ePROMs, there is also more demand to undertake feasibility testing to test the equivalence of administering an e-version of a PROM compared to its paper version (although the need for such testing has been questioned [32, 33]) and/or establish its acceptability and usability as part of routine clinical care [34, 35].
Despite the essential role that pilot and feasibility testing plays in relation to PROMs, little attention to date has been given to this important stage. It would seem that there are missed opportunities to provide more guidance and ideas regarding what types of pilot or feasibility tests could be carried out. In particular, this is in relation to the ways in which pilot and feasibility tests can be integrated during PROM development, evaluation and implementation and also in terms of what the “standards” are for assessing the methodological quality of these types of studies.
This series seeks to provide a forum for the research community to share and disseminate the work they have been doing which concerns the pilot and feasibility testing of PROMs. This is often reported as one small step during the PROM reporting process and may be the result of peer review journals not considering this stage of high enough importance to dedicate publication space. Therefore, this series aims to give researchers the ability to dedicate the reporting space needed to fully report the processes undertaken, raise the profile of pilot and feasibility testing in relation to PROM research and to provide a platform on which innovative methods can be shared. By dedicating more publication space to the reporting of pilot and feasibility studies in relation to PROMs, it may also help to open up a dialogue amongst the PROM research community about some of the academic and practical issues raised above.
In the future, hopefully, the pilot and feasibility testing of PROMs can align more fully with the MRC guidance so that (i) this crucial stage of the research process can be integrated across all aspects of preparatory work related to PROM development, evaluation and implementation and (ii) clearer guidance and benchmarks for conducting such pilot and feasibility studies can become available to support the research community whilst carrying out these important stages during their PROM-based research.
Albrecht GL. Subjective health assessment. In: Jenkinson C, editor. Measuring health and medical outcomes. London: UCL Press; 1994. p. 7–26.
World Health Organisation. The first ten years of the world health. London: World Health Organisation; 1958.
Jenkinson C. Assessment and evaluation of health and medical care: an introduction and overview. In: Jenkinson C, editor. Assessment and evaluation of health and medical care: a methods text. Buckingham: Open University Press; 1997. p. 1–5.
Wigton RS. Medical applications. In: Brehmer B, Joyce CRB, editors. s. Amsterdam: Elsevier; 1988. p. 227–45.
White-Koning M, Grandjean H, Colver A, Arnaud C. Parent and professional reports of the quality of life of children with cerebral palsy and associated intellectual impairment. Dev Med Child Neurol. 2008;50(8):618–24.
Fitzpatrick R, Davey C, Buxton M, Jones DR. Patient-assessed outcome measures. In: Black N, Brazier J, Fitzpatrick R, Reeves B, editors. Health services research methods. London: BMJ Publications; 1998. p. 13–22.
Fitzpatrick R. A pragmatic defence of health status measures. Health Care Anal. 1996;4:265–72.
Brazier J, Deverill M. A checklist for judging preference-based measures of health related quality of life: learning from psychometrics. Health Econ. 1999;8(1):41–51.
Snyder CF, Jensen RE, Segal JB, Wu AW. Patient-reported outcomes (PROs): putting the patient perspective in patient-centered outcomes research. Med Care. 2013;51(8 Suppl 3):S73–9. doi:10.1097/MLR.0b013e31829b1d84.
Goodhart IM, Andrzejowski JC, Jones GL, Berthoud M, Dennis A, Mills GH, Radley SC. Patient-completed, preoperative web-based anaesthetic assessment questionnaire (electronic Personal Assessment Questionnaire PreOperative): development and validation. Eur J Anaesthesiol. 2017;34(4):221–8. Xx.
Malhotra K, Buraimoh O, Thornton J, Cullen N, Singh D, Goldberg AJ. Electronic capture of patient-reported and clinician-reported outcome measures in an elective orthopaedic setting: a retrospective cohort analysis. BMJ Open. 2016;6(6):e011975.
van de Poll-Franse LV, Horevoorts N, van Eenbergen M, Denollet J, Roukema JA, Aaronson NK, Vingerhoets A, Coebergh JW, de Vries J, Essink-Bot ML, Mols F, Profiles Registry Group. The Patient Reported Outcomes Following Initial treatment and Long term Evaluation of Survivorship registry: scope, rationale and design of an infrastructure for the study of physical and psychosocial outcomes in cancer survivorship cohorts. Eur J Cancer. 2011;47(14):2188–94.
Lane A, Metcalfe C, Young GJ, Peters TJ, Blazeby J, Avery KN, Dedman D, Down L, Mason MD, Neal DE, Hamdy FC, Donovan JL, ProtecT Study group. Patient-reported outcomes in the ProtecT randomized trial of clinically localized prostate cancer treatments: study design, and baseline urinary, bowel and sexual function and quality of life. BJU Int. 2016;118(6):869–79.
Costal Tirado A, McDermott AM, Thomas C, Ferrick D, Harris J, Edwards A, McAllister M. Using patient-reported outcome measures for quality improvement in clinical genetics: an exploratory study. J Genet Couns. 2017. doi:10.1007/s10897-017-0079-6 [Epub ahead of print].
Mamiya H, Lix LM, Gardner W, Bartlett SJ, Ahmed S, Buckeridge DL. Montreal Accord on patient-reported outcomes use series—paper 5: patient reported outcomes can be linked to epidemiologic measures to monitor populations and inform public health decisions. J Clin Epidemiol. 2017. doi: 10.1016/j.jclinepi.2017.04.018.
Wolfe F, Pincus T. Standard self-report questionnaires in routine clinical and research practice—an opportunity for patients and rheumatologists. J Rheumatol. 1991;18(5):643–6.
Ahmed S, Ware P, Gardner W, Witter J, Bingham CO 3rd, Khairy D, Bartlett SJ. Montreal Accord on Patient-Reported Outcomes Use Series—Paper 8: patient reported outcomes (PRO) in electronic health records can inform clinical and policy decisions. J Clin Epidemiol. 2017. doi: 10.1016/j.jclinepi.2017.04.011.
Dawson J, Doll H, Fitzpatrick R, Jenkinson C, Carr AJ. The routine use of patient reported outcome measures in healthcare settings. BMJ. 2010;340:c186.
Devlin N, Appleby J. Getting the most out of PROMs: putting health outcomes at the heart of NHS decision making. London: The King’s Fund; 2010.
Craig P, Dieppe P, Macintyre S, Mitchie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655.
Browne J, Jamieson L, Lawsey J, van der Meulen J, Black N, Cairns J, Lamping D, Smith S, Copley L, Horrockes J. Patient reported outcome measures (PROMs) in elective surgery. 2007. Report to the Department of Health; 2007.
Schroter S, Miles R, Green S, Jackson M. Psychometric validation of the Coronary Revascularisation Outcome Questionnaire (CROQv2) in the context of the NHS Coronary Revascularisation PROMs Pilot. BMJ Open. 2017;7(2):e015915.
Weenink JW, Braspenning J, Wensing M. Patient reported outcome measures (PROMs) in primary care: an observational pilot study of seven generic instruments. BMC Fam Pract. 2014;15:88.
Mason JD, Blencowe NS, McNair AG, et al. Investigating the collection and assessment of patient-reported outcome data amongst unplanned surgical hospital admissions: a feasibility study. Pilot Feasib Stud. 2015;1:16. doi:10.1186/s40814-015-0011-5.
Peters M, Crocker H, Dummett S, Jenkinson C, Doll H, Gibbons E, Luengo-Fernandez R, Fitzpatrick R. Pilot study of patient reported outcome measures (PROMs) in primary care. Oxford: 2013. https://www.qoru.ac.uk/about/research/application/pilot-study-of-patient%E2%80%90reported-outcomemeasures-proms-in-long-term-conditions-in-primary-care/.
US Department of Health and Human Services Guidance for Industry. Patient-reported outcome measures. Use in medical product development to support labeling claims. Washington, DC: US Food and Drug Administration; 2009.
Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, Bouter LM, de Vet HC. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19(4):539–49.
Mokkink LB, Terwee CB, Gibbons E, Stratford PW, Alonso J, Patrick DL, Knol DL, Bouter LM, de Vet HC. Inter-rater agreement and reliability of the COSMIN(COnsensus-based Standards for the selection of health status Measurement Instruments) checklist. BMC Med Res Methodol. 2010;10:82.
Moores KL, Jones GL, Radley SC. Development of an instrument to measure face validity, feasibility and utility of patient questionnaire use during health care: the QQ-10. Int J Qual Health Care. 2012;24(5):517–24.
Poku E, Duncan R, Keetharuth A, Essat M, Phillips P, Woods HB, Palfreyman S, Jones G, Kaltenthaler E, Michaels J. Patient-reported outcome measures in patients with peripheral arterial disease: a systematic review of psychometric properties. Health Qual Life Outcomes. 2016;14(1):161. Review.
Vernon-Wilson L. The Warwick-Edinburgh Mental Well-being Scale (WEMWBS) a pilot study of utility and feasibility in older aged people’s mental health services. New Forest East Community Mental Health Treatment Team: 2013.
Muehlhausen W, Doll H, Quadri N, Fordham B, O’Donohoe P, Dogar N, Wild DJ. Equivalence of electronic and paper administration of patient-reported outcome measures: a systematic review and meta-analysis of studies conducted between 2007 and 2013. Health Qual Life Outcomes. 2015;13:167.
Bjorner JB, Rose M, Gandek B, Stone AA, Junghaenel DU, Ware Jr JE. Difference in method of administration did not significantly impact item response: an IRT-based analysis from the Patient-Reported Outcomes Measurement Information System (PROMIS) initiative. Qual Life Res. 2014;23(1):217–27.
Ashley, L., Jones, H., Forman, D., Newsham, A., Brown, J., Downing, A., … Wright, P. Feasibility test of a UK-scalable electronic system for regular collection of patient-reported outcome measures and linkage with clinical cancer registry data: the electronic Patient-reported Outcomes from Cancer Survivors (ePOCS) system. BMC Med Inform Decis Mak. 2011;11:66
Andikyan V, Rezk Y, Einstein MH, Gualtiere G, Leitao Jr MM, Sonoda Y, Abu-Rustum NR, Barakat RR, Basch EM, Chi DS. A prospective study of the feasibility and acceptability of a web-based, electronic patient-reported outcome system in assessing patient recovery after major gynecologic cancer surgery. Gynecol Oncol. 2012;127(2):273–277.
The author would like to acknowledge the support of Professor Gillian Lancaster and Ella Flemyng in the preparation of this editorial.
The author was commissioned to write this article by BMC Pilot and Feasibility Studies and would like to thank Ella Flemyng and Gillian Lancaster for their support. No funding was received.
Availability of data and materials
GJ wrote this Editorial.
The views expressed are those of the author alone.
The author declares that she has no competing interests.
Consent for publication
Ethics approval and consent to participate
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
A correction to this article is available online at https://doi.org/10.1186/s40814-017-0183-2.