Skip to main content

More pilot trials could plan to use qualitative data: a meta-epidemiological study

Abstract

Background

Pilot trials often use quantitative data such as recruitment rate and retention rate to inform the design and feasibility of a larger trial. However, qualitative data such as patient, healthcare provider, and research staff perceptions of an intervention may also provide insights for a larger trial.

Methods

As part of a larger study investigating the reporting of progression criteria in pilot studies, we sought to determine how often pilot studies planned to use qualitative data to inform the design and feasibility of a larger trial and the factors associated with plans to use qualitative data. We searched for protocols of pilot studies of randomized trials in PubMed between 2013 and 2017.

Results

We included 227 articles. Only 92 (40.5%; 95% confidence interval [CI] 34.1–47.2) reported plans to collect qualitative data. The factors associated with collecting qualitative data were large studies (defined as sample size ≥ 60; adjusted odds ratio [aOR] 2.77; 95% CI 1.47–5.23; p = 0.002) and studies from Europe (aOR 3.86; 95% CI 1.68–8.88; p = 0.001) compared to North America and the rest of the world. Pilot trials with pharmacological interventions were less likely to plan to collect qualitative data (aOR 0.20; 95% CI 0.07–0.58; p = 0.003).

Conclusions

Qualitative data is not used enough in pilot trials. Large pilot trials, pilot trials from Europe, and pilot trials of non-pharmacological interventions are more likely to plan for qualitative data.

Peer Review reports

Background

Feasibility studies are a category of study which aim to determine if a study should be done, can be done, and, if so, how it might be done [1]. Pilot studies are a type of feasibility study that maintains an element of study design (e.g. randomization), albeit on a smaller scale than what is intended for the full trial [1,2,3]. Data from pilot studies is of great value to researchers for identifying and correcting problems which might otherwise compromise acceptability and delivery of interventions in the full study. By correcting these issues before carrying out a larger study, researchers may reduce misuse of resources and inappropriate evaluation techniques in the larger trial [4,5,6].

Since pilot studies are based on the methods intended for a full-scale investigation, quantitative and qualitative methods can be used. Both methods have the potential to yield useful data in the context of a pilot study, and it is recommended by the United Kingdom Medical Research Council (MRC) that qualitative and quantitative methods be used concurrently with a pilot study or full trial [5]. For example, in 2018, Irish researchers evaluated the feasibility and acceptability of a sexual counselling intervention for patients undergoing cardiac rehabilitation [7]. An evaluation was performed by documenting recruitment and attrition rates, as well as recording feedback from interviews with staff and patients.

Qualitative methods are increasingly being used in pilot trials [8]. The reason qualitative methods were not used as much in prior years is not entirely clear but may have been because of a lack of guidance on the topic at the time or limited understanding of the value of qualitative methods among certain researchers. In the past, some researchers have expressed concern regarding the practicality, reliability, and generalizability of qualitative methods [9]. However, in recent years, there has been an increase in the number of resources available to researchers seeking to perform and publish pilot studies. In 2015, the journal Pilot and Feasibility Studies was created, which has acted as a platform to promote pilot and feasibility research [10]. There has also been a considerable amount of guidance published on the use of qualitative methods in pilot studies [11] and reporting of pilot studies [12,13,14].

Given the recent developments in the field, practices regarding the use of qualitative methods in pilot studies may have changed. More information on the collection and use of qualitative data in pilot studies may help researchers develop further guidance and improve the design of pilot studies as well as the larger investigations they inform. With the publication of pilot study protocols becoming more common in Pilot and Feasibility Studies, we have greater insight into the researchers’ intentions to use qualitative methods [13]. Therefore, the purpose of this study was to determine how often pilot trials are designed to collect qualitative data and the study characteristics associated with planning to collect qualitative data. Accordingly, we conducted a methodological review of published protocols for pilot randomized trials. This work was conducted as part of another methodological study investigating the use of progression criteria in pilot studies [15].

Methods

Data collection

We conducted a methodological review of protocols for pilot randomized trials. We applied the following search strategy in the PubMed database, including terms for three journals known to publish protocols and pilot studies: (BMJ Open [Journal] OR Pilot Feasibility Stud [Journal] OR Trials [Journal]) AND (Pilot [Title] OR Feasibility [Title] AND Protocol [Title]). We restricted our search to studies published between 01 January 2013 and 31 December 2017—a 5-year period—to keep the data manageable.

Two separate reviewers (SDK, DOL, RS, ORU, AA, or LM) screened each of the full texts of identified citations for eligibility. In order to meet eligibility, the studies had to be (1) published in Pilot and Feasibility Studies (PAFS), British Medical Journal (BMJ) Open, or Trials; (2) a protocol for a pilot randomized trial; and (3) published between 2013 and 2017. Data were extracted from the eligible studies by one reviewer and verified by a second independent senior reviewer. Verification involved comparing the extracted data to the manuscript to identify errors or discrepancies. New data were extracted by the second reviewer only when there were discrepancies or errors.

Reviewers determined whether each included pilot trial had planned to collect qualitative data (including but not limited to in-depth focus groups, one-on-one interviews, qualitative surveys/questionnaires) and the population from which the data were to be retrieved (i.e. participants, staff, investigators, others). Data on the study characteristics (i.e. bibliographic information, country of origin, source of funding, study objectives and outcomes, intervention type (pharmacological vs. non-pharmacological), whether the study considered feasibility outcomes, sample size estimation, and adequacy of the justification for sample size) were extracted. These characteristics were selected because they have been found to be associated with reporting in other studies [16]. When the planned sample size was reported as a range of values, the median was taken. When different sample sizes were reported for the different participants (e.g. health workers, patients, caretakers), we used the sample size for those who would be randomized. To facilitate analysis, we categorized the studies as small (< 60) or large (≥ 60), based on the median sample size (60) across studies. The current guidance suggests that the sample size of a pilot trial should be based on a feasibility outcome and not clinical outcomes (intervention effect size) [17]. As such, using the intervention effect size to determine the sample size, stating that it was selected because other studies reported the same way, or providing no justification was deemed inadequate. The country in which the pilot was planned was also collected and categorized into three world regions (North America, Europe, and rest of the world) to facilitate analyses (other regions contributed very few studies).

All data were collected and managed using the Research Electronic Data Capture (REDCap) tool hosted at St Joseph’s Healthcare Hamilton (Ontario, Canada) [18].

Data analysis

First, counts of the plans to collect qualitative data, as well as the study characteristics, were summarized descriptively in cross tabulations. Second, we used logistic regression to determine the relationships between the study characteristics and the odds of planning to collect qualitative data (categorized as yes/no). The covariates were entered as a block: journal (PAFS, BMJ Open, and Trials), year of publication (continuous), region (North America, Europe, or rest of the world), source of funding (industry or government/private), intervention type (pharmacological or non-pharmacological), and sample size (small [0–60], large [> 60]). These variables have been shown to be associated with reporting standards [16]. The goodness of fit of the model was assessed using the Hosmer-Lemeshow test (α = 0.05). Crude odds ratios (OR) and adjusted odds ratios (aOR), corresponding 95% confidence intervals (CI), and p values are reported. Data was analysed using IBM SPSS Statistics version 25 and WinPepi (PEPI-for-Windows) [19].

Results

Our search retrieved 276 articles, of which 49 were not eligible (21 were protocols for non-randomized studies; 19 were full reports and not a protocol; 7 were errata or corrigenda; 1 was a methodological paper; 1 was a trial update). Of the 227 included pilot trial protocols, only 92 (40.5%; 95% CI 34.1–47.2) reported plans to collect qualitative data in their pilot trials. Approximately half (50.2 %) of the included articles were published in Trials and conducted in Europe (52.9%). Many of the pilot trials (75.5%) were supported by government or private funding. More pilot trial protocols were published in 2017 than in any other year (31.7%), and the most common intervention type was non-pharmacological (84.6%). The flow of studies is reported in Fig. 1. The complete description of the study characteristics and frequency of qualitative data collection is reported in Table 1.

Fig. 1
figure1

Flow of studies

Table 1 Characteristics of included studies

After adjusting for covariates, we found that pilot trials that were large (sample size > 60; aOR 2.77; 95% CI 1.47–5.23; p = 0.002) and conducted in Europe (aOR 3.86; 95% CI 1.68–8.88; p = 0.001) were more likely to plan to collect qualitative data. Pilot trials using pharmacological interventions (aOR 0.20; CI 0.07–0.58; p = 0.003) were less likely to plan to collect qualitative data. Our complete univariate and multivariate analyses of factors associated with the collection of qualitative data are presented in Table 2.

Table 2 Factors associated with collecting qualitative data+

Discussion

We found that qualitative data collection is planned for in less than half of the protocols of pilot trials. However, we found that qualitative data are more likely to be collected in pilot trials conducted in Europe, in studies assessing non-pharmacological interventions, and in larger pilot trials. When qualitative data collection was planned for, it was most often collected from study participants.

Our estimate of the frequency with which researchers plan to collect qualitative data may not be an accurate representation of the frequency of use of qualitative methods used in pilot trials because the methods outlined in protocols may not be implemented. Conversely, unplanned methods may be implemented. Despite that, our results show a discrepancy between researchers’ intent to perform qualitative methods and MRC recommendations to use qualitative and quantitative methods concurrently for all pilot studies [5]. It is not surprising that less than half of pilot trial protocols report plans to use qualitative data given the misconceptions about the value of qualitative methods among some researchers, lack of guidance on how to incorporate qualitative research in pilot trials, or concerns with the practicality, reliability, and generalizability of qualitative data [9]. Further research could access researchers’ perspectives on the use of qualitative data in pilot trials. Also, if researchers unduly place emphasis on effectiveness (rather than feasibility) in pilot studies, they may be less likely to use qualitative data. However, the percentage of pilot trials that did plan to use qualitative data in this study was higher than a 2013 study of the use of qualitative methods before a full trial [8]. This suggests that the frequency of qualitative data collection may be on the rise, and current guidance on pilot studies is correcting researchers’ misconceptions about qualitative methods. We strongly encourage researchers to refer to relevant guidance and consider qualitative methods more often for their own pilot studies [11].

It is unclear why studies from Europe were more likely to use qualitative data. However, other studies have found that there are regional differences in reporting quality [20,21,22], with at least one suggesting that reporting is better in studies from Europe [23]. It is possible that launch of the journal Pilot and Feasibility Studies (based in the UK) in 2015 and subsequent publication of guidance for reporting randomized and feasibility trials in the journal (e.g. Consolidated Standards of Reporting Trials statement for reporting feasibility studies) have had the greatest impact in Europe [12,13,14]. Otherwise, we are not aware of any stringent practices in Europe for pilot trial protocol reporting by agencies or institutions that support studies.

Pilot studies of non-pharmacological interventions were more likely to use qualitative data. This may be because non-pharmacological interventions are more amenable to modifications based on user feedback. For example, a 2016 mixed methods feasibility and pilot cluster randomized controlled trial examined the use of a nurse-led goal-setting intervention for adults with asthma using semi-structured interviews [24]. They identified several issues with the intervention, including the need to make it more team-based and simplistic. They also identified a variety of procedural issues (e.g. reducing the impact of the intervention on consultation time). In comparison, it may not be possible to change as many aspects of pill-based, pharmaceutical interventions. Even though qualitative data would still have value (e.g. patient feedback may help to inform the design of pills—two smaller tablets vs. one—or help in adapting language during recruitment if patients report symptoms or issues that impact their swallowing of oral pills), its applicability may be limited. However, even if the intervention may not be amenable to change, there is value in using qualitative methods to assess acceptability [25, 26].

Additionally, research indicates that larger studies tend to be better reported [16], although this may not be applicable to pilot studies given their generally smaller sample sizes. This is likely due to the availability of resources for larger studies, which may include larger multidisciplinary teams and funds for better design and more in-depth data collection.

Not surprisingly, investigators most often planned to collect qualitative data from study participants. Participant interviews may provide rich data on satisfaction with the intervention, acceptability of the procedures, and issues that may prevent participants from enrolling in the main trial. Family and caregivers may provide such information as well. Likewise, staff working in the hospitals in which the study will be conducted can provide information on how the study affects their workflow as well as any logistic challenges that they might face. Careful consideration of who to interview can optimize the chances of success in the future trial. For example, in a pilot trial of a structured physical activity intervention for colorectal cancer patients, Hubbard et al. identified recruitment biases, imprecision, and issues with adherence by interviewing clinicians and patients [27, 28].

This study is not without some limitations. Our search for pilot trial protocols was focused on only 3 leading journals from 2013 to 2017. There may be other journals that publish protocols for pilot studies of which are unaware. Therefore, it is possible that our data may not be representative of patterns of plans to use qualitative data in other journals. However, the strengths of this study are robust methods used to identify, screen, and analyse the data, and the novelty of the findings.

Conclusion

Qualitative data is not incorporated in the designs of pilot studies often enough, despite its potential value in informing intervention development and the definitive study design. Investigators conducting pilot studies are encouraged to consider the role of qualitative data in their feasibility assessments, particularly when planning the pilot trial.

Availability of data and materials

The full dataset is available from the corresponding author upon request.

Abbreviations

aOR:

Adjusted odds ratio

BMJ:

British Medical Journal

CI:

Confidence interval

OR:

Odds ratio

PAFS:

Pilot and Feasibility Studies

References

  1. 1.

    Eldridge SM, Lancaster GA, Campbell MJ, Thabane L, Hopewell S, Coleman CL, et al. Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. Plos One. 2016;11(3):e0150205.

    Article  PubMed  PubMed Central  Google Scholar 

  2. 2.

    Kate BDZ. Pilot and Feasibility Studies. In: Kate B DZ, editor. Health research methods: a Canadian perspective. Don Mills: Oxford University Press; 2014. p. 111.

    Google Scholar 

  3. 3.

    Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1.

    Article  PubMed  PubMed Central  Google Scholar 

  4. 4.

    Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.

    Article  PubMed  PubMed Central  Google Scholar 

  5. 5.

    Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res. 2011;45(5):626–9.

    Article  PubMed  Google Scholar 

  6. 6.

    Kistin C, Silverstein M. Pilot studies: a critical but potentially misused component of interventional research. JAMA. 2015;314(15):1561–2.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  7. 7.

    Murphy PJ, Noone C, D’Eath M, Casey D, Doherty S, Jaarsma T, et al. The CHARMS pilot study: a multi-method assessment of the feasibility of a sexual counselling implementation intervention in cardiac rehabilitation in Ireland. Pilot Feasibility Stud. 2018;4:88.

    Article  PubMed  PubMed Central  Google Scholar 

  8. 8.

    O’Cathain ATK, Drabble SJ, Rudolph A, Hewison J. What can qualitative research do for randomised controlled trials? A systematic mapping review. BMJ Open. 2013;3:e002889.

  9. 9.

    Khankeh H, Ranjbar M, Khorasani-Zavareh D, Zargham-Boroujeni A, Johansson E. Challenges in conducting qualitative research in health: a conceptual paper. Iran J Nurs Midwifery Res. 2015;20(6):635–41.

    Article  PubMed  PubMed Central  Google Scholar 

  10. 10.

    Lancaster GA. Pilot and feasibility studies come of age! Pilot Feasibility Stud. 2015;1(1):1.

    Article  PubMed  PubMed Central  Google Scholar 

  11. 11.

    O’Cathain A, Hoddinott P, Lewin S, Thomas KJ, Young B, Adamson J, et al. Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers. Pilot Feasibility Stud. 2015;1:32.

    Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. Pilot Feasibility Stud. 2016;2:64.

    Article  PubMed  PubMed Central  Google Scholar 

  13. 13.

    Thabane L, Lancaster G. A guide to the reporting of protocols of pilot and feasibility trials. Pilot Feasibility Stud. 2019;5:37.

    Article  PubMed  PubMed Central  Google Scholar 

  14. 14.

    Lancaster GA, Thabane L. Guidelines for reporting non-randomised pilot and feasibility studies. Pilot Feasibility Stud. 2019;5:114.

    Article  PubMed  PubMed Central  Google Scholar 

  15. 15.

    Mbuagbaw L, Kosa SD, Lawson DO, Stalteri R, Olaiya OR, Alotaibi A, et al. The reporting of progression criteria in protocols of pilot trials designed to assess the feasibility of main trials is insufficient: a meta-epidemiological study. Pilot and Feasibility Studies. 2019;5(1):120.

    Article  PubMed  PubMed Central  Google Scholar 

  16. 16.

    Samaan Z, Mbuagbaw L, Kosa D, Borg Debono V, Dillenburg R, Zhang S, et al. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169–88.

    PubMed  PubMed Central  Google Scholar 

  17. 17.

    L. Thabane JM, R. Chu, J. Cheng, A. Ismaila, L. Rios, R. Robson, M. Thabane, L. Giangregorio, C. Goldsmith. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10(1). https://doi.org/10.1186/1471-2288-10-1.

  18. 18.

    Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81.

    Article  Google Scholar 

  19. 19.

    Abramson JH. WINPEPI updated: computer programs for epidemiologists, and their teaching potential. Epidemiol Perspect Innov. 2011;8(1):1.

    Article  PubMed  PubMed Central  Google Scholar 

  20. 20.

    Farrokhyar F, Chu R, Whitlock R, Thabane L. A systematic review of the quality of publications reporting coronary artery bypass grafting trials. Canadian Journal of Surgery Journal canadien de chirurgie. 2007;50(4):266–77.

    PubMed  PubMed Central  Google Scholar 

  21. 21.

    Ntala C, Birmpili P, Worth A, Anderson NH, Sheikh A. The quality of reporting of randomised controlled trials in asthma: a systematic review. Prim Care Respir J. 2013;22(4):417–24.

    Article  PubMed  PubMed Central  Google Scholar 

  22. 22.

    Zhong Y, Zhou W, Jiang H, Fan T, Diao X, Yang H, et al. Quality of reporting of two-group parallel randomized controlled clinical trials of multi-herb formulae: a survey of reports indexed in the Science Citation Index Expanded. Eur J Integr Med. 2011;3(4):e309–e16.

    Article  Google Scholar 

  23. 23.

    Peron J, Pond GR, Gan HK, Chen EX, Almufti R, Maillet D, et al. Quality of reporting of modern randomized controlled trials in medical oncology: a systematic review. J Natl Cancer Inst. 2012;104(13):982–9.

    Article  PubMed  Google Scholar 

  24. 24.

    Hoskins G, Williams B, Abhyankar P, Donnan P, Duncan E, Pinnock H, et al. Achieving Good Outcomes for Asthma Living (GOAL): mixed methods feasibility and pilot cluster randomised controlled trial of a practical intervention for eliciting, setting and achieving goals for adults with asthma. Trials. 2016;17(1):584.

    Article  PubMed  PubMed Central  Google Scholar 

  25. 25.

    Bluebelle Study G, Reeves BC, Andronis L, Blazeby JM, Blencowe NS, Calvert M, et al. A mixed-methods feasibility and external pilot study to inform a large pragmatic randomised controlled trial of the effects of surgical wound dressing strategies on surgical site infections (Bluebelle Phase B): study protocol for a randomised controlled trial. Trials. 2017;18(1):401.

    Article  Google Scholar 

  26. 26.

    Chesser T, Fox R, Harding K, Greenwood R, Javaid K, Barnfield S, et al. The administration of intermittent parathyroid hormone affects functional recovery from pertrochanteric fractured neck of femur: a protocol for a prospective mixed method pilot study with randomisation of treatment allocation and blinded assessment (FRACTT). BMJ Open. 2014;4(1):e004389.

    Article  PubMed  PubMed Central  Google Scholar 

  27. 27.

    Hubbard G, O’Carroll R, Munro J, Mutrie N, Haw S, Mason H, et al. The feasibility and acceptability of trial procedures for a pragmatic randomised controlled trial of a structured physical activity intervention for people diagnosed with colorectal cancer: findings from a pilot trial of cardiac rehabilitation versus usual care (no rehabilitation) with an embedded qualitative study. Pilot Feasibility Stud. 2016;2:51.

    Article  PubMed  PubMed Central  Google Scholar 

  28. 28.

    Winder R, Richards SH, Campbell JL, Richards DA, Dickens C, Gandhi M, et al. Development and refinement of a complex intervention within cardiac rehabilitation services: experiences from the CADENCE feasibility study. Pilot Feasibility Stud. 2017;3:9.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

Not applicable

Funding

None

Author information

Affiliations

Authors

Contributions

LM, TM, and LT conceptualized this study. TB, LM, SDK, RS, ORO, AA, and TM performed the searches, extracted the data, and performed the analysis. TB and TM wrote the first draft of the manuscript. LM, SDK, DOL, RS, ORO, AA, and LT provided input to revise the draft. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Tejan Baldeh.

Ethics declarations

Ethics approval and consent to participate

Ethics approval was not required because in this study, we used already published data.

Consent for publication

Not applicable

Competing interests

The authors declare that there were no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Baldeh, T., MacDonald, T., Kosa, S.D. et al. More pilot trials could plan to use qualitative data: a meta-epidemiological study. Pilot Feasibility Stud 6, 164 (2020). https://doi.org/10.1186/s40814-020-00712-z

Download citation

Keywords

  • Pilot
  • Feasibility
  • Trials
  • Qualitative data
  • Protocols