Skip to main content

Assessment of the face validity, feasibility and utility of a patient-completed questionnaire for polymyalgia rheumatica: a postal survey using the QQ-10 questionnaire

An Erratum to this article was published on 24 October 2017

This article has been updated



The development of a patient-reported outcome measure (PROM) for polymyalgia rheumatica (PMR), a condition that causes pain, stiffness and disability, is necessary as there is no current validated disease-specific measure. Initial literature synthesis and qualitative research established a conceptual framework for the condition along with a list of symptoms and effects of PMR that patients felt were important to them. These findings were used to derive the candidate items for a patient-completed questionnaire. We aim to establish the face validity of this initial “long form” of a PROM.


People with a current or previous diagnosis of PMR were recruited both from the community and from rheumatology clinics. They were asked to complete the PMR questionnaire along with the QQ-10 questionnaire, which is a measure used to assess the face validity, feasibility and utility of patient healthcare questionnaires.


A total of 28 participants with an age range of 59–85 years and a length of time since diagnosis from 4 months to 18 years completed the QQ-10. The overall mean “value” score was 79% (SD 12), and the mean “burden” score was 21% (SD 18). The free-text comments were analysed thematically and were found to focus on layout, content, where in the clinical pathway the questionnaire would be most beneficial, specific items missing and other areas for consideration.


The high mean value score and low burden score indicate that the questionnaire has good face validity and is acceptable to patients. The questionnaire now needs to undergo further psychometric evaluation and refinement to develop the final tool for use in clinical practice and research.

Peer Review reports


Polymyalgia rheumatica (PMR) causes significant pain, stiffness and disability in older adults and is treated with systemic glucocorticoids which can themselves cause significant additional morbidity [1, 2]. A recent systematic review of outcome measures used in PMR research studies [3] identified a wide variety of instruments in use, none of which were specifically developed to measure symptom burden in patients with PMR. Less than 10% of studies measured physical function, quality of life or fatigue, despite these being important to patients [4]. Further work, including a Delphi survey conducted by the OMERACT PMR Working Group, highlighted the need for a disease-specific outcome measure that would cover domains of life impact relevant to patients [5]. Applying international guidelines for patient-reported outcome measure (PROM) development [6], we carried out initial qualitative work with patients with PMR to better understand patient experience of the condition and establish a conceptual framework for a future PMR PROM [7]. Here, we report the next steps taken to derive a “long-list” of candidate items for a PMR-specific PROM and the assessment of face validity, feasibility and utility both in the participants of the original study and in a separate group of patients, using a validated method, the QQ-10 questionnaire [8]. The QQ-10 is a measure developed to collect standardised information on important aspects of a questionnaire’s qualities from the patient’s perspective and is used to assess the face validity, feasibility and utility of patient healthcare questionnaires. This part of the iterative PROM development process is particularly important to ensure that the instrument is acceptable and contains the content that is relevant to patients with PMR, and completion burden of the final questionnaire is minimised thus improving follow-up and questionnaire completion rates in clinical trials [9]. These data are vital for development of a PROM that is valid for either research studies or clinical practice.


A long-list of candidate items for the PROM was developed from the data obtained from a previous qualitative study carried out by our group [7]. These items were categorised into main symptoms/duration (4 items), function (24 items), emotional and psychological well-being (11 items), steroid side effects (10 items) and overall well-being (1 item). Respondent validation of the themes and items was carried out with the original interview participants. All 20 interviewees from the qualitative study were sent a copy of the long-list and asked to return it with any comments that they had on the content. Ten of these individuals agreed to a structured telephone interview to discuss the long list in more detail and the information gathered from this was used to make changes to it to form the draft questionnaire (Table 1).

Table 1 Long-list of PMR PROM questionnaire items derived from the qualitative study

To assess face validity and utility in an independent group of patients, patients with a diagnosis of PMR were recruited through two routes: (1) community-based: through a patient-led patient support group, PMR&GCAUK North East Support and (2) hospital-based: through rheumatology clinics at Leeds Teaching Hospitals NHS Trust. This recruitment strategy was designed to sample from the full spectrum of patients with PMR as it is a condition managed in both primary and secondary care. Ethical approval was received from the National Institute for Social Care and Health Research Research Ethics Service, Wales REC 7 (Ref 12/WA/0344), and all participants provided informed consent. Patients received the study materials by post (community-based recruitment) or in a sealed envelope from their treating rheumatologist (hospital-based recruitment), completed the study materials at home in their own time, and returned the completed forms to the lead researcher (HT), who was based at a different institution and in a different city to their rheumatology clinic.

The data from the QQ-10 questionnaire were analysed both quantitatively and qualitatively. For quantitative analysis we used the QQ-10 scoring method [8]. Likert ratings from strongly disagree to strongly agree (coded as 0–4) were summed separately for the first six questions comprising the value score (helped me communicate about my condition, relevant to my condition, easy to complete, included all the aspects of my condition I am concerned about, was enjoyable, would be happy to complete as part of routine care), and from the last four questions comprising the burden score (too long, embarrassing, complicated, upset me).

Qualitative thematic analysis [10] was performed on comments received in response to the three free-text questions at the end of the QQ-10:

  1. 1.

    Do you have any comments or suggestions on how the questionnaire you used could be improved (e.g., its structure, appearance or design)?

  2. 2.

    Were any of your important symptoms, problems or concerns missed out by the questionnaire you used?

  3. 3.

    Do you feel that any areas or problems in the questionnaire you used were over-represented?


Twenty-eight patients took part (20 female and 8 male; age range 59–85 years; duration since PMR diagnosis 4 months to 18 years; apart from a single participant who was 18 years post diagnosis, no patient was more than 5 years post diagnosis). Eighteen of the participants were still on steroid treatment for their PMR.

The overall mean value score was 79% (SD 12), and the mean burden score was 21% (SD 18). The median of each domain making up the value score was >2 (range 0–4), and the median of each domain making up the burden score was <1.5 (range 0–4). Figures 1 and 2 show the median scores for each question, and Fig. 3 shows the spread of responses for each question.

Fig. 1
figure 1

Mean value of domains contributing to the value score

Fig. 2
figure 2

Mean value of domains contributing to the burden score

Fig. 3
figure 3

Frequency distribution of scores for each question in the QQ-10

The five emergent themes from the free-text comments (Table 2) were (1) layout, (2) content, (3) where in the clinical pathway the questionnaire would be most beneficial, (4) specific items not covered and (5) other areas for consideration for inclusion. The content theme encompassed sub-themes of depth and detail, specificity to PMR and heterogeneity of the condition.

Table 2 Thematic analysis of the QQ10 free-text answers


This study represents a distinct, patient-orientated phase within a stepped standardised methodological approach to developing a PROM for PMR for research and clinical practice. The use of the validated QQ-10 measure is an example of how to embed patient perspectives at all stages within the PROM development process, moving beyond the paradigm of clinician-orientated outcome measures.

The high value and low burden scores are encouraging as regards face validity and feasibility of the questionnaire. They are similar to scores obtained when the QQ-10 has been used in other studies, including evaluation of the King’s Health Questionnaire [11] and evaluation of use of a bladder diary [12], and both of these tools were judged as having been proven to be useful based on these results. The free-text comments provided added richness to the response data, and their analysis highlights some important required amendments to the structure and layout of the questionnaire as well as suggesting some additional points for inclusion. The comments related to content echo some of the findings from our earlier qualitative work [7] and some of the known challenges of outcome measurement in PMR, chiefly the heterogeneity of the condition, difficulty in assessment in the presence of co-morbidities and overlap with other conditions [1, 4, 13].

The strengths of this study include the use of a validated instrument (QQ-10) to assess face validity, feasibility and utility of instruments designed to assess the life impact of particular disease states. The QQ-10 itself was rigorously developed using standard psychometric methods and has been demonstrated to have high internal consistency and item correlation and to be acceptable and understandable to patients [8]. It also has the advantages of being quick and cheap to administer and allowing comparison of different versions of a measure at several stages during development. At this stage of questionnaire development, we made the decision not to employ interview-based qualitative methods, such as cognitive interviewing, as the postal method of responding did not exclude patients too frail to participate in cognitive interviews, and allowed patients to respond honestly, in private and without any concern that the responses might be seen by their treating clinicians. A further strength of this study was the use of a combination of recruitment methods which identified patients with a range of disease durations similar to that described in previous literature [14, 15].

The main limitation of this study was the small number of participants and opportunistic (and thus not statistically representative) sampling method. However, this sample size was appropriate for this stage of PROM development [16, 17]. A further limitation of this study was that the community-based method of recruitment required patients to self-identify as being diagnosed with PMR. We did not attempt to validate diagnosis by means of classification or other criteria designed to select patients with PMR for research studies, because we wanted to assess this questionnaire in a real-life setting for clinical practice, not just for use in research studies.


The long form of our PMR questionnaire was shown to have face validity for patients and be acceptable for use in their care. The work reported here represents an essential step in the PROM development process, paving the way for further work using a larger sample size that will allow formal psychometric validation, item reduction and ultimately generation of a fully validated patient-reported outcome measure for PMR.

Change history

  • 24 October 2017

    A correction to this article has been published.



Outcome measures in rheumatology ( An independent initiative of international health professionals interested in outcome measures in rheumatology


Polymyalgia rheumatica


Patient-reported outcome measure


  1. Mackie SL, Mallen CDM. Polymyalgia rheumatica. BMJ. 2013;347:f6937.

    Article  PubMed  Google Scholar 

  2. Dejaco C, Singh YP, Perel P, et al. 2015 Recommendations for the management of polymyalgia rheumatica. A European League Against Rheumatisim/American College of Rheumatology Collaborative Initiative. Ann Rheum Dis. 2015;74:1799–807.

    Article  CAS  PubMed  Google Scholar 

  3. Duarte C, Ferreira JO, Mackie SL, et al. Outcome measures in polymyalgia rheumatica. A systematic review. J Rheumatol. 2015;42(12):2503–11.

    Article  CAS  PubMed  Google Scholar 

  4. Mackie SL, Arat S, da Silva J, et al. Polymyalgia rheumatica (PMR) special interest group at OMERACT 11: outcomes of importance for patients with PMR. J Rheumatol. 2014;41(4):819–23.

    Article  PubMed  Google Scholar 

  5. Helliwell T, Brouwer E, Pease CT, et al. Development of a provisional core domain set for polymyalgia rheumatica: report from the OMERACT 12 polymyalgia rheumatica working group. J Rheumatol. 2016;43(1):182–6.

    Article  PubMed  Google Scholar 

  6. U.S. Dept of Health and Human Services Food and Drug Administration. Guidance for industry: patient-reported outcomes measures: use in medicinal product development to support labelling claims. 2009. Accessed 1 July 2016.

  7. Twohig HJ, Mitchell CM, Mallen CDM, et al. “I suddenly felt I’d aged”: a qualitative study of patient experiences of polymyalgia rheumatica (PMR). Patient Educ Couns. 2015;98:645–50.

    Article  PubMed  Google Scholar 

  8. Moores KL, Jones GL, Radley SC. Development of an instrument to measure face validity, feasibility and utility of patient questionnaire use during health care: the QQ-10. Int J Qual Health Care. 2012;24:517–24.

    Article  CAS  PubMed  Google Scholar 

  9. Prescott RJ, Counsell CE, Gillespie WJ, et al. Factors that limit the quality, number and progress of randomised controlled trials. Health Technol Assess. 1993;3(20):1–143.

    Google Scholar 

  10. Boyatzis RE. Transforming qualitative information: thematic analysis and code development. Thousand Oaks, London, & New Delhi: SAGE Publications; 1998.

    Google Scholar 

  11. Vij M, Srikrishna S, Robinson D, et al. Quality assurance in quality of life assessment—measuring the validity of the King’s Health Questionnaire. Int Urogynaecol J. 2014;25(8):1133–5.

    Article  CAS  Google Scholar 

  12. Vella M, Robinson D, Cardozo L, et al. The bladder diary: do women perceive it as a useful investigation? Eur J Obstet Tynecol Reprod Biol. 2012;162(2):221–3.

    Article  CAS  Google Scholar 

  13. Dasgupta B, Cimmino MA, Maradit-Kremers H, et al. 2012 provisional classification criteria for polymyalgia rheumatica: a European League Against Rheumatism/American College of Rheumatology collaborative initiative. Ann Rheum Dis. 2012;71:484–92.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Bahlas S, Ramos-Remus C, Davis P. Clinical outcome of 149 patients with polymyalgia rheumatica and giant cell arteritis. J Rheumatol. 1998;25(1):99–104.

    CAS  PubMed  Google Scholar 

  15. Mackie SL, Hensor EM, Haugeberg G, et al. Can the prognosis of polymyalgia rheumatic be predicted at disease onset? Results from a 5-year prospective study. Rheumatology. 2009;49(4):716–22.

    Article  Google Scholar 

  16. Jones G, Kennedy S, Barnard A, et al. Development of an endometriosis quality-of-life instrument: The Endometriosis Health Profile-30. Obstet Gynecol. 2001;98(2):258–64.

    CAS  PubMed  Google Scholar 

  17. Mirzaee Rabor F, Taghipour A, Mirzaee M et al. Developing a questionnaire for Iranian women’s attitude on medical ethics in vaginal childbirth. Nurs Midwifery Stud 2015 Dec;4(4):e29004. Accessed 1 Jul 2016.

Download references


We would like to thank all of the study participants and the trustees and members of PMR&GCA UK North East Support for their continued support and cooperation.


This work was funded by a National Institute for Health Research (NIHR) In-Practice Fellowship and a Royal College of General Practitioners Scientific Foundation Board grant awarded to Helen Twohig.

CDM is funded by the National Institute for Health Research (NIHR) Collaborations for Leadership in Applied Health Research and Care West Midlands, the NIHR School for Primary Care Research and a NIHR Research Professorship in General Practice (NIHR-RP-2014-04-026). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Availability of data and materials

The datasets generated and used during the current study are available from the corresponding author on reasonable request.

Authors’ contributions

All authors contributed to the design of the study and contributed critically to the content of the final manuscript. HT organised the data collection, carried out the qualitative and quantitative analysis and drafted the initial manuscript. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Ethical approval was received from the National Institute for Social Care and Health Research Research Ethics Service, Wales REC 7 (Ref 12/WA/0344), and all participants provided informed consent.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Helen Twohig.

Additional information

A correction to this article is available online at

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Twohig, H., Jones, G., Mackie, S. et al. Assessment of the face validity, feasibility and utility of a patient-completed questionnaire for polymyalgia rheumatica: a postal survey using the QQ-10 questionnaire. Pilot Feasibility Stud 4, 7 (2018).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: