Skip to main content


We're creating a new version of this page. See preview

  • Research
  • Open Access
  • Open Peer Review

Assessment of the face validity, feasibility and utility of a patient-completed questionnaire for polymyalgia rheumatica: a postal survey using the QQ-10 questionnaire

Pilot and Feasibility Studies20174:7

  • Received: 28 February 2017
  • Accepted: 31 May 2017
  • Published:

The Erratum to this article has been published in Pilot and Feasibility Studies 2017 3:48

Open Peer Review reports



The development of a patient-reported outcome measure (PROM) for polymyalgia rheumatica (PMR), a condition that causes pain, stiffness and disability, is necessary as there is no current validated disease-specific measure. Initial literature synthesis and qualitative research established a conceptual framework for the condition along with a list of symptoms and effects of PMR that patients felt were important to them. These findings were used to derive the candidate items for a patient-completed questionnaire. We aim to establish the face validity of this initial “long form” of a PROM.


People with a current or previous diagnosis of PMR were recruited both from the community and from rheumatology clinics. They were asked to complete the PMR questionnaire along with the QQ-10 questionnaire, which is a measure used to assess the face validity, feasibility and utility of patient healthcare questionnaires.


A total of 28 participants with an age range of 59–85 years and a length of time since diagnosis from 4 months to 18 years completed the QQ-10. The overall mean “value” score was 79% (SD 12), and the mean “burden” score was 21% (SD 18). The free-text comments were analysed thematically and were found to focus on layout, content, where in the clinical pathway the questionnaire would be most beneficial, specific items missing and other areas for consideration.


The high mean value score and low burden score indicate that the questionnaire has good face validity and is acceptable to patients. The questionnaire now needs to undergo further psychometric evaluation and refinement to develop the final tool for use in clinical practice and research.


  • Polymyalgia rheumatica
  • Patient perspective
  • Outcomes research
  • Patient-reported outcome measures
  • Questionnaire validity and utility assessment


Polymyalgia rheumatica (PMR) causes significant pain, stiffness and disability in older adults and is treated with systemic glucocorticoids which can themselves cause significant additional morbidity [1, 2]. A recent systematic review of outcome measures used in PMR research studies [3] identified a wide variety of instruments in use, none of which were specifically developed to measure symptom burden in patients with PMR. Less than 10% of studies measured physical function, quality of life or fatigue, despite these being important to patients [4]. Further work, including a Delphi survey conducted by the OMERACT PMR Working Group, highlighted the need for a disease-specific outcome measure that would cover domains of life impact relevant to patients [5]. Applying international guidelines for patient-reported outcome measure (PROM) development [6], we carried out initial qualitative work with patients with PMR to better understand patient experience of the condition and establish a conceptual framework for a future PMR PROM [7]. Here, we report the next steps taken to derive a “long-list” of candidate items for a PMR-specific PROM and the assessment of face validity, feasibility and utility both in the participants of the original study and in a separate group of patients, using a validated method, the QQ-10 questionnaire [8]. The QQ-10 is a measure developed to collect standardised information on important aspects of a questionnaire’s qualities from the patient’s perspective and is used to assess the face validity, feasibility and utility of patient healthcare questionnaires. This part of the iterative PROM development process is particularly important to ensure that the instrument is acceptable and contains the content that is relevant to patients with PMR, and completion burden of the final questionnaire is minimised thus improving follow-up and questionnaire completion rates in clinical trials [9]. These data are vital for development of a PROM that is valid for either research studies or clinical practice.


A long-list of candidate items for the PROM was developed from the data obtained from a previous qualitative study carried out by our group [7]. These items were categorised into main symptoms/duration (4 items), function (24 items), emotional and psychological well-being (11 items), steroid side effects (10 items) and overall well-being (1 item). Respondent validation of the themes and items was carried out with the original interview participants. All 20 interviewees from the qualitative study were sent a copy of the long-list and asked to return it with any comments that they had on the content. Ten of these individuals agreed to a structured telephone interview to discuss the long list in more detail and the information gathered from this was used to make changes to it to form the draft questionnaire (Table 1).
Table 1

Long-list of PMR PROM questionnaire items derived from the qualitative study




How severe has the pain from your PMR been in the last 2 weeks? (0-10 visual analogue scale (VAS) with 0 = no pain and 10 = the worst pain you’ve ever had)


How severe has the stiffness from your PMR been in the last 2 weeks? (0-10 VAS with 0 = no stiffness and 10 = the worst stiffness you’ve ever felt)


How severe has the weakness from your PMR been in the last 2 weeks? (0-10 VAS with 0 = no weakness and 10 = complete weakness)


On average, for how much of each day has the pain/stiffness/weakness from your PMR been present for during the last 2 weeks?

All day/About half the day/Around 1-3 hours/< 1 hour


FUNCTION: Over the last 2 weeks, compared to what you can normally do, has PMR limited your ability to do the following activities?

Graded as 1) no, not limited at all, 2) yes, limited a little, 3) yes, limited a lot, 4) not relevant

Bend down

Get up after bending down

Get in and out of a car

Drive a car

Get in and out of bed

Get in or out of a chair

Get in or out of a bath

Wash yourself fully

Dry yourself fully after a shower/bath

Take your coat on or off

Put on or take off your socks and shoes

Comb or blow dry your hair

Get on or off the toilet

Wipe yourself after going to the toilet

Engage in intimate/sexual activity

Walk up stairs

Walk up hills

Walk on the flat

Carry or lift things

Reach above your head for things

Grip objects

Do housework

Do gardening

Sit for more than 30 minutes at a time

Participate in sports


EMOTIONAL AND PSYCHOLOGICAL WELL-BEING: In the last 2 weeks have your PMR symptoms…

Graded as 1) none of the time, 2) a little of the time, 3) some of the time, 4) most of the time, 5) all of the time

Caused you to feel low in mood

Caused you to feel anxious

Caused you to feel vulnerable

Lowered your self-confidence

Made you worried that you might fall over

Caused you to need more help with looking after yourself

Made you less inclined to go out

Stopped you doing hobbies that you used to do

Made you worry about the future

Affected your sleep

Made you feel more tired than usual


TREATMENT SIDE EFFECTS: How much have you been affected by side effects from your medication in the last 2 weeks? (VAS with 0 = unaffected, 10 = severely affected)


In the last 2 weeks, have you been bothered by any of the following side effects of your steroid medication? (Yes/No)

Weight gain

Change in appearance (fatter face, saggy skin)


Low mood



Easy bruising



Hair loss


Do you feel back to the level of health you were at before you first experienced PMR symptoms? (Yes/No)

To assess face validity and utility in an independent group of patients, patients with a diagnosis of PMR were recruited through two routes: (1) community-based: through a patient-led patient support group, PMR&GCAUK North East Support and (2) hospital-based: through rheumatology clinics at Leeds Teaching Hospitals NHS Trust. This recruitment strategy was designed to sample from the full spectrum of patients with PMR as it is a condition managed in both primary and secondary care. Ethical approval was received from the National Institute for Social Care and Health Research Research Ethics Service, Wales REC 7 (Ref 12/WA/0344), and all participants provided informed consent. Patients received the study materials by post (community-based recruitment) or in a sealed envelope from their treating rheumatologist (hospital-based recruitment), completed the study materials at home in their own time, and returned the completed forms to the lead researcher (HT), who was based at a different institution and in a different city to their rheumatology clinic.

The data from the QQ-10 questionnaire were analysed both quantitatively and qualitatively. For quantitative analysis we used the QQ-10 scoring method [8]. Likert ratings from strongly disagree to strongly agree (coded as 0–4) were summed separately for the first six questions comprising the value score (helped me communicate about my condition, relevant to my condition, easy to complete, included all the aspects of my condition I am concerned about, was enjoyable, would be happy to complete as part of routine care), and from the last four questions comprising the burden score (too long, embarrassing, complicated, upset me).

Qualitative thematic analysis [10] was performed on comments received in response to the three free-text questions at the end of the QQ-10:
  1. 1.

    Do you have any comments or suggestions on how the questionnaire you used could be improved (e.g., its structure, appearance or design)?

  2. 2.

    Were any of your important symptoms, problems or concerns missed out by the questionnaire you used?

  3. 3.

    Do you feel that any areas or problems in the questionnaire you used were over-represented?



Twenty-eight patients took part (20 female and 8 male; age range 59–85 years; duration since PMR diagnosis 4 months to 18 years; apart from a single participant who was 18 years post diagnosis, no patient was more than 5 years post diagnosis). Eighteen of the participants were still on steroid treatment for their PMR.

The overall mean value score was 79% (SD 12), and the mean burden score was 21% (SD 18). The median of each domain making up the value score was >2 (range 0–4), and the median of each domain making up the burden score was <1.5 (range 0–4). Figures 1 and 2 show the median scores for each question, and Fig. 3 shows the spread of responses for each question.
Fig. 1
Fig. 1

Mean value of domains contributing to the value score

Fig. 2
Fig. 2

Mean value of domains contributing to the burden score

Fig. 3
Fig. 3

Frequency distribution of scores for each question in the QQ-10

The five emergent themes from the free-text comments (Table 2) were (1) layout, (2) content, (3) where in the clinical pathway the questionnaire would be most beneficial, (4) specific items not covered and (5) other areas for consideration for inclusion. The content theme encompassed sub-themes of depth and detail, specificity to PMR and heterogeneity of the condition.
Table 2

Thematic analysis of the QQ10 free-text answers






“I think it’s well set out in a fairly simple and effective format.”

“Where the table goes over the page, it would be helpful to repeat the headers. Page numbering would be helpful.”

“Sheets should be numbered and column headings should be repeated where they go onto 2 pages.”

“Questions about age etc. should be at the start.”

“Figure diagrams are a more direct indication of the type and location of pain than the written word.”


Depth and detail

“Too much detail, cut down on the number of boxes, they overlap too much. Q5 has too many choices.”

“Seems quite straightforward.”

“Q9 (Do you feel back to the level of health you were at before you had PMR?) - if answer is no, ask why?”

Specificity to PMR

“PMR is not that specific. For me I sometimes ache and sometimes feel a bit down. Muscle power has diminished but that may be age.”

“Many things you ask could be for other reasons such as depression, arthritis, cancer etc. it’s not all PMR.”

“Some info on other conditions should be included e.g., I had had a stroke previously and PMR imposed symptoms on top of those resulting from that.”

“Some symptoms (like weakness and difficulty reaching things in cupboards) I already had from a shoulder injury it’s difficult to tell whether the PMR made it worse or not.”

“As everyone is different there should be room for an individual’s particular symptoms and concerns.”

“Q9 is irrelevant, no-one gets back to feeling as well as before.”


“Should pain/ache be quantified? Different people will have different meanings.”

“Some questions didn’t seem to fit my symptoms but I don’t think I’m very severely affected - time will tell.”

“PMR affects all of us in different ways and the questionnaire covers all aspects and does no harm even if some questions overlap.”

Where in the clinical pathway it would be most beneficial


“I think it’s an excellent questionnaire from the outset of a PMR diagnosis to a record of the PMR journey.”

“I think the questionnaire will be very helpful, especially to people at the beginning of their treatment when they probably have all of the problems listed. I would have been reassured to think the doctors knew how I was feeling.”

Specific items missing


“The side effects of prednisolone”

“Diabetes, fluid retention leading to lymphoedema”

“The area of pain”

“More questions about fatigue could be included. Skin/hair condition missed out.”

“Swelling of the joints is not mentioned (hands, wrists, feet and ankles).”

“No real questions about where the pain was or what I couldn’t do.”

“One of my main concerns at the start was inability to fasten my own bra - had to ask for help.”

Other concerns that could be included


“Concerns about recurrence are important.”

“You might like to know how the steroid treatment has helped/been successful.”

“It may be helpful to ask patients to put down aspects of their health that may not be PMR related but which they wish to discuss.”


This study represents a distinct, patient-orientated phase within a stepped standardised methodological approach to developing a PROM for PMR for research and clinical practice. The use of the validated QQ-10 measure is an example of how to embed patient perspectives at all stages within the PROM development process, moving beyond the paradigm of clinician-orientated outcome measures.

The high value and low burden scores are encouraging as regards face validity and feasibility of the questionnaire. They are similar to scores obtained when the QQ-10 has been used in other studies, including evaluation of the King’s Health Questionnaire [11] and evaluation of use of a bladder diary [12], and both of these tools were judged as having been proven to be useful based on these results. The free-text comments provided added richness to the response data, and their analysis highlights some important required amendments to the structure and layout of the questionnaire as well as suggesting some additional points for inclusion. The comments related to content echo some of the findings from our earlier qualitative work [7] and some of the known challenges of outcome measurement in PMR, chiefly the heterogeneity of the condition, difficulty in assessment in the presence of co-morbidities and overlap with other conditions [1, 4, 13].

The strengths of this study include the use of a validated instrument (QQ-10) to assess face validity, feasibility and utility of instruments designed to assess the life impact of particular disease states. The QQ-10 itself was rigorously developed using standard psychometric methods and has been demonstrated to have high internal consistency and item correlation and to be acceptable and understandable to patients [8]. It also has the advantages of being quick and cheap to administer and allowing comparison of different versions of a measure at several stages during development. At this stage of questionnaire development, we made the decision not to employ interview-based qualitative methods, such as cognitive interviewing, as the postal method of responding did not exclude patients too frail to participate in cognitive interviews, and allowed patients to respond honestly, in private and without any concern that the responses might be seen by their treating clinicians. A further strength of this study was the use of a combination of recruitment methods which identified patients with a range of disease durations similar to that described in previous literature [14, 15].

The main limitation of this study was the small number of participants and opportunistic (and thus not statistically representative) sampling method. However, this sample size was appropriate for this stage of PROM development [16, 17]. A further limitation of this study was that the community-based method of recruitment required patients to self-identify as being diagnosed with PMR. We did not attempt to validate diagnosis by means of classification or other criteria designed to select patients with PMR for research studies, because we wanted to assess this questionnaire in a real-life setting for clinical practice, not just for use in research studies.


The long form of our PMR questionnaire was shown to have face validity for patients and be acceptable for use in their care. The work reported here represents an essential step in the PROM development process, paving the way for further work using a larger sample size that will allow formal psychometric validation, item reduction and ultimately generation of a fully validated patient-reported outcome measure for PMR.




Outcome measures in rheumatology ( An independent initiative of international health professionals interested in outcome measures in rheumatology


Polymyalgia rheumatica


Patient-reported outcome measure



We would like to thank all of the study participants and the trustees and members of PMR&GCA UK North East Support for their continued support and cooperation.


This work was funded by a National Institute for Health Research (NIHR) In-Practice Fellowship and a Royal College of General Practitioners Scientific Foundation Board grant awarded to Helen Twohig.

CDM is funded by the National Institute for Health Research (NIHR) Collaborations for Leadership in Applied Health Research and Care West Midlands, the NIHR School for Primary Care Research and a NIHR Research Professorship in General Practice (NIHR-RP-2014-04-026). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Availability of data and materials

The datasets generated and used during the current study are available from the corresponding author on reasonable request.

Authors’ contributions

All authors contributed to the design of the study and contributed critically to the content of the final manuscript. HT organised the data collection, carried out the qualitative and quantitative analysis and drafted the initial manuscript. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Ethical approval was received from the National Institute for Social Care and Health Research Research Ethics Service, Wales REC 7 (Ref 12/WA/0344), and all participants provided informed consent.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

Academic Unit of Primary Medical Care, University of Sheffield, Samuel Fox House, Northern General Hospital, Herries Road, Sheffield, S5 7AU, UK
Faculty of Health and Social Sciences, Leeds Beckett University, Leeds, UK
Leeds Institute of Rheumatic and Musculoskeletal Medicine, University of Leeds, Leeds, UK
Arthritis Research UK Primary Care Centre, Institute for Primary Care and Health Sciences, Keele University, Keele, UK
Academic Unit of Primary Medical Care, University of Sheffield, Sheffield, UK


  1. Mackie SL, Mallen CDM. Polymyalgia rheumatica. BMJ. 2013;347:f6937.View ArticlePubMedGoogle Scholar
  2. Dejaco C, Singh YP, Perel P, et al. 2015 Recommendations for the management of polymyalgia rheumatica. A European League Against Rheumatisim/American College of Rheumatology Collaborative Initiative. Ann Rheum Dis. 2015;74:1799–807.View ArticlePubMedGoogle Scholar
  3. Duarte C, Ferreira JO, Mackie SL, et al. Outcome measures in polymyalgia rheumatica. A systematic review. J Rheumatol. 2015;42(12):2503–11.View ArticlePubMedGoogle Scholar
  4. Mackie SL, Arat S, da Silva J, et al. Polymyalgia rheumatica (PMR) special interest group at OMERACT 11: outcomes of importance for patients with PMR. J Rheumatol. 2014;41(4):819–23.View ArticlePubMedGoogle Scholar
  5. Helliwell T, Brouwer E, Pease CT, et al. Development of a provisional core domain set for polymyalgia rheumatica: report from the OMERACT 12 polymyalgia rheumatica working group. J Rheumatol. 2016;43(1):182–6.View ArticlePubMedGoogle Scholar
  6. U.S. Dept of Health and Human Services Food and Drug Administration. Guidance for industry: patient-reported outcomes measures: use in medicinal product development to support labelling claims. 2009. Accessed 1 July 2016.
  7. Twohig HJ, Mitchell CM, Mallen CDM, et al. “I suddenly felt I’d aged”: a qualitative study of patient experiences of polymyalgia rheumatica (PMR). Patient Educ Couns. 2015;98:645–50.View ArticlePubMedGoogle Scholar
  8. Moores KL, Jones GL, Radley SC. Development of an instrument to measure face validity, feasibility and utility of patient questionnaire use during health care: the QQ-10. Int J Qual Health Care. 2012;24:517–24.View ArticlePubMedGoogle Scholar
  9. Prescott RJ, Counsell CE, Gillespie WJ, et al. Factors that limit the quality, number and progress of randomised controlled trials. Health Technol Assess. 1993;3(20):1–143.Google Scholar
  10. Boyatzis RE. Transforming qualitative information: thematic analysis and code development. Thousand Oaks, London, & New Delhi: SAGE Publications; 1998.Google Scholar
  11. Vij M, Srikrishna S, Robinson D, et al. Quality assurance in quality of life assessment—measuring the validity of the King’s Health Questionnaire. Int Urogynaecol J. 2014;25(8):1133–5.View ArticleGoogle Scholar
  12. Vella M, Robinson D, Cardozo L, et al. The bladder diary: do women perceive it as a useful investigation? Eur J Obstet Tynecol Reprod Biol. 2012;162(2):221–3.View ArticleGoogle Scholar
  13. Dasgupta B, Cimmino MA, Maradit-Kremers H, et al. 2012 provisional classification criteria for polymyalgia rheumatica: a European League Against Rheumatism/American College of Rheumatology collaborative initiative. Ann Rheum Dis. 2012;71:484–92.View ArticlePubMedPubMed CentralGoogle Scholar
  14. Bahlas S, Ramos-Remus C, Davis P. Clinical outcome of 149 patients with polymyalgia rheumatica and giant cell arteritis. J Rheumatol. 1998;25(1):99–104.PubMedGoogle Scholar
  15. Mackie SL, Hensor EM, Haugeberg G, et al. Can the prognosis of polymyalgia rheumatic be predicted at disease onset? Results from a 5-year prospective study. Rheumatology. 2009;49(4):716–22.View ArticleGoogle Scholar
  16. Jones G, Kennedy S, Barnard A, et al. Development of an endometriosis quality-of-life instrument: The Endometriosis Health Profile-30. Obstet Gynecol. 2001;98(2):258–64.PubMedGoogle Scholar
  17. Mirzaee Rabor F, Taghipour A, Mirzaee M et al. Developing a questionnaire for Iranian women’s attitude on medical ethics in vaginal childbirth. Nurs Midwifery Stud 2015 Dec;4(4):e29004. Accessed 1 Jul 2016.


© The Author(s). 2017


By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate. Please note that comments may be removed without notice if they are flagged by another user or do not comply with our community guidelines.