Development and pilot testing of the 2019 Canadian Abortion Provider Survey

Abstract

Background

Substantial changes in abortion care regulations, available medications and national clinical practice guidelines have occurred since a 2012 national Canadian Abortion Provider Survey (CAPS). We developed and piloted the CAPS 2019 survey instrument to explore changes of the abortion provider workforce, their clinical care as well as experiences with stigma and harassment.

Methods

We undertook development and piloting in three phases: (1) development of the preliminary survey sections and questions based on the 2012 survey instrument, (2) content validation and feasibility of including certain content aspects via a modified Delphi Method with panels of clinical and research experts, and (3) pilot testing of the draft survey for face validity and clarity of language; assessing usability of the web-based Research Electronic Data Capture platform including the feasibility of complex skip pattern functionality. We performed content analysis of phase 2 results and used a general inductive approach to identify necessary survey modifications.

Results

In phase 1, we generated a survey draft that reflected the changes in Canadian abortion care regulations and guidelines and included questions for clinicians and administrators providing first and second trimester surgical and medical abortion. In phase 2, we held 6 expert panel meetings of 5–8 participants each representing clinicians, administrators and researchers to provide feedback on the initial survey draft. Due to the complexity of certain identified aspects, such as interdisciplinary collaboration and interprovincial care delivery differences, we revised the survey sections through an iterative process of meetings and revisions until we reached consensus on constructs and questions to include versus exclude for not being feasible. In phase 3, we made minor revisions based on pilot testing of the bilingual, web-based survey among additional experts chosen to be widely representative of the study population. Demonstrating its feasibility, we included complex branching and skip pattern logic so each respondent only viewed applicable questions based on their prior responses.

Conclusions

We developed and piloted the CAPS 2019 survey instrument suitable to explore characteristics of a complex multidisciplinary workforce, their care and experience with stigma on a national level, and that can be adapted to other countries.

Background

In Canada, individuals have the right to safe, legal, and voluntary abortion care. Each year, over 80,000 surgical and medical abortions (MA) are provided in Canada [1]. In 2012, the first national Canadian Abortion Provider Survey (CAPS) captured responses from 78 abortion facility administrators and 178 physicians. Important findings included that there were relatively few abortion providers in Canada, most responding facilities and clinicians were non-hospital-based and focused the majority of their clinical care on abortion provision, rural urban abortion access disparities existed, and that MA contributed to less than 4% of all abortions [2, 3]. However, since 2012, substantial changes in the abortion care landscape have occurred; Health Canada approved mifepristone, the international gold standard drug for MA, and it became available in 2017 [4, 5]. In the same year, restrictive regulations around prescribing and dispensing mifepristone were removed [6] and subsequently the cost of the medication was covered in most provinces [7]. Cost of abortion care otherwise was already covered under Canada’s universal health care. Regulations now allow nurse practitioners (NPs) to independently provide first trimester MA care [8]. The Society of Obstetricians and Gynaecologists of Canada (SOGC) published updated evidence-based clinical practice guidelines for both medical and surgical abortions [9, 10]. To fill a knowledge gap, we aimed to develop the CAPS 2019 to assess the impact of these changes.

Based on our team’s clinical experiences and supported by emerging evidence [7], we hypothesized that the workforce has increased and that care has shifted away from primarily occurring in urban locations and abortion specific clinics to enhanced rural access and in primary care settings [7]. The aims of the CAPS 2019 study were to survey the multidisciplinary workforce and document the change in their characteristics and distribution compared to CAPS 2012; to assess the care they provide, i.e., characteristics of actual abortion practices as compared to the revised Canadian clinical practice guidelines, in both MA and surgical abortion practices; and to determine the extent abortion providers experienced harassment and stigma in their work, and explore their related resilience and retention. Our objective for this study was to develop the CAPS 2019 survey instrument to meet our aims (phase 1), validate the instrument using expert panels (phase 2), and pilot test the usability of the survey in a web-based format (phase 3).

Methods

The CAPS 2019 survey was developed and conducted by members of the Contraception and Abortion Research Team–Groupe de recherche sur l’avortement et la contraception (CART-GRAC), a national, interdisciplinary, cross-sectoral research group that supports the provision of health services and implementation of policies to promote equitable access to high-quality family planning knowledge, methods, and health care services throughout Canada [11]. In this study, we aimed to develop a national cross-sectional survey, to explore changes in the abortion workforce, their clinical care and experience with stigma and harassment.

Chronologically, the survey development involved (1) development of the preliminary survey sections and questions, (2) content validation and feasibility of including certain content aspects via a modified Delphi Method with panels of clinical and research experts, and (3) pilot testing of the draft survey for face validity and clarity of language; assessing usability of the web-based Research Electronic Data Capture platform including the feasibility of complex skip pattern functionality [12]. Ethics approval for this study was obtained from the University of British Columbia’s Children’s and Women’s Hospital Research Ethics Board (H18-03,313).

Phase 1: development of the preliminary survey sections and questions

We used the CAPS 2012 instrument as the basis for the 2019 survey. We leveraged the expertise of our multisectoral CART-GRAC members (including clinicians, workforce specialists, and researchers) in virtual working-group meetings to decide which survey changes would be required to capture changes in workforce and clinical practice. We refined the study population of interest to include non-physician clinicians and all types and aspects of clinical abortion care (first and second trimester surgical and first, second, and third trimester MA). In collaboration with our Institutional Research Board (IRB), the decision was made to anonymize the survey in order to maximize protection of participants’ personal identity. This was important due to the sensitive nature of abortion care.

We identified relevant publications to inform the required changes such as professional regulations for NPs, Health Canada regulations on mifepristone prescribing and dispensing [13, 14], mifepristone product monograph [5, 15], and the SOGC clinical practice guidelines [9, 10]. We mapped these publications to relevant survey sections and individual questions. The survey section on first trimester MA required major revisions as the mifepristone medication regimen only became available in Canada in 2017 and, therefore, was not included in the 2012 survey. We reviewed the USA first trimester MA section of the 2012 survey for questions we could use or adapt for our survey [16]. Due to the complexity of provincial variations in abortion related policies and service delivery between provider types (physicians, NPs, midwives, and administrators) some of the survey topics and questions required an iterative process of meetings and revisions until consensus was reached. The participants of the working-groups ensured we created relevant and concise questions, placed in appropriate sections of the survey and in a logical order to ensure understanding.

Additionally, we collaborated with the authors of the validated stigma and harassment scale we had used in the 2012 survey to update our survey section on stigma and harassment which included questions beyond the original scale [17, 18] exploring resilience and retention of providers.

Further considerations of the initial survey development included moving from a combined paper- and web-based survey sent to known abortion facilities to a web-based survey accessible via a generic survey link in order to reach a wider and largely unknown study population. This decision would also allow us to use branching logic that enabled respondents to only read questions that applied to them based on their prior answers, to minimize survey fatigue. Our instrument was available through the REDCap platform [12], a secure web-based meta-data-driven web application, housed at our partner research institute, BC Children’s Hospital Research Institute.

Phase 2: content validation via a modified Delphi method with a panel of clinical experts

Panel assembly

We virtually convened clinical and research expert panels from across Canada, consisting of representatives of the study population, to examine face validity of the first version of the survey instrument, using a modified Delphi method (described below) [19,20,21]. Experts were primarily co-investigators or knowledge users of our study team, and were asked to invite colleagues to participate through a snowball sampling approach if we thought we needed further expert representation [22]. We held panels for each type of abortion care and each provider type and ensured representation of rural and urban, as well as hospital versus non-hospital based, specialists and generalist clinicians in order to optimize the survey. Depending on their clinical expertise, some experts attended multiple panel meetings.

The Delphi method

We used a Delphi method [19, 21] to consult a representative group of Canadian abortion providers to discuss and validate the initial survey questions and sections. We modified each of the rounds of Delphi expert input by revising the survey instrument after each of the panel sessions to reflect the input provided by that group before meeting with the next panel. Some of the survey sections (e.g., demographics) were reviewed by multiple panels, some only applied to one of the panels (e.g., second trimester surgical abortion). As some of the experts sat on multiple panels, they reviewed subsequent versions of some sections again as suggested in the Delphi method [20]. A final version of the survey instrument was created after all expert panel input was incorporated.

Each panel consisted of 5–8 clinicians and researchers who met through videoconference. At least two research team members (RR, WVN, SD, EG) moderated each meeting. The moderator, following a standard semi-structured interview guide, provided a project description followed by survey instrument information highlighting the changes compared to the 2012 survey instrument and subsequently guided the content validation process (see Table 1 for content validation and Supplement `1 for complete interview guide). Expert panelists reviewed each question for clarity and relevance. We invited them to give feedback on wording of questions and response options including their rationale, suggest non-essential questions to be removed, and provide feedback on whether the proposed questions and response options would feasibly capture the abortion related regulations and care delivery specific to their provider type (physician versus NP versus midwives versus administrator) and province. We reserved time for open-ended discussion, panelists to share their perceptions and experiences with care provision and implementation of mifepristone. The meetings were audio-recorded and transcribed.

Table 1 CAPS 2019 development panelist instructions for question analysis

Response analysis

Our panel meeting facilitators (RR, SD, WVN, EG) independently reviewed transcripts to identify emerging themes. Together they conducted a content analysis of the transcripts in a recursive fashion to identify common and conflicting viewpoints in a stepwise analysis. We examined stratified groups (Physician’s versus NPs versus midwives versus administrators and groups from different provinces) for similarities and differences. This entire analysis was conducted to refine the scope of the abortion workforce and service delivery we would feasibly be able to capture in this survey, to remove unneeded questions and modify existing ones, and to identify important topics missed in the first version of the survey.

Our usability analysis involved an iterative process to change the format of survey questions. For example, we had asked panelists to provide feedback on preferences between free text answers versus check boxes with number intervals or Likert scales. We ensured standardization of language between survey sections, consistent use of language, and consistency between Likert scale directions throughout the survey, i.e., from strongly disagree to strongly agree or from less to more often. Discussions further aimed at clarifying ambiguous terms or concepts and the possible consequence of different answers from respondents; i.e., regarding service delivery models, such as being the “most responsible provider” versus working in a “shared care model”, or working in a clinic versus office versus facility.

In the next step of our analysis, we used a general inductive approach to identify needed content modifications to questions and answer options. For example, in the context of regulations not allowing independent midwifery abortion care, the midwifery service delivery models across the country proved to be too heterogeneous to be feasibly captured in a survey rather than in an interview. We therefore removed questions related to abortion practice by midwives from our survey. In the context of the evolving COVID-19 pandemic during the subsequent piloting phase, co-investigators emphasized the importance to clarify that our survey asked about 2019 and pre-COVID care, but also to add a question at the end of the survey exploring how the pandemic affected respondents’ abortion care.

Finalizing the bilingual survey instrument and creating a web-based survey

Following the response analysis, the core co-investigator working group who had developed the initial survey draft reviewed a revised version of the survey instrument and iteratively suggested further modifications until we reached consensus. Thereafter, we professionally translated the survey into French, and the Francophone investigators on our team (MSW and EG) reviewed the French survey for correct translation of medical terminology and relevance for Francophone respondents.

The bilingual survey instrument was entered into REDCap. We programmed questions that determined the branching logic or were considered essential as mandatory and respondents could not continue until they had answered that question, while other questions could be skipped by the respondents if they chose. Detailed programming in collaboration with the REDCap technical team was further needed so that the electronic survey allowed respondents to choose answer options in the intended fashion and to avoid mistakes; i.e., radio buttons if only one answer was asked for versus an option to check all answer options that applied to a respondent. We further programmed “check all that apply”, and answer options that were incompatible with other answer options to the same question as an “action tag” which automatically unselected the other answer options to that question. When applicable, we displayed multiple sub-questions and answer options to a combined question stem in a table format called “matrix”. We used the options of soft and hard coding of number answers to decrease the chance of non-sensical answers to questions involving percentages or numbers such as gestational age of a pregnancy. “Soft-coding” allowed respondents to override a pop-up window with a warning that the answer was outside of the expected range (for example usual upper gestational age limit for a clinical care aspect), while “hard coding” did not allow entering the number at all (for example percentage greater than 100). Generally, we aimed to format the survey in a web user-friendly design, e.g., minimize the need to scroll on a screen, and allowed respondents to create a return code if they wanted to complete the survey in multiple sittings.

To protect respondent confidentiality as requested by our IRB, the main study section did not include personal identifiers. However, in a separate survey section that was not linked to the data, respondents could provide their email address for remuneration purposes ($50 gift certificate) or to be contacted with study results or future research.

Web-based surveys increase the risk of fraudulent responses, especially when anonymous and recruiting via a generic link, which pose a threat to their data integrity [23, 24]. Offering financial incentives as well as conducting research on socially or politically sensitive topics are additional risk factors for fraud [25]. Therefore, we took measures to decrease the risk for fraud with careful choices of our study design [23, 24]. To help prevent fraud from robots we added a CAPTCHA to the beginning of the survey. In our initial survey question, we asked respondents to confirm that they had not taken the survey before. We planned to monitor incoming responses for possible fraud. Based on the literature, this consisted of confirming sensical combinations between a few select questions in the demographics [26]. We describe our detailed fraud detection strategy in a separate manuscript [27].

Phase 3: pilot testing of the draft survey

We conducted extensive piloting in order to assess usability, face validity, and clarity of language throughout the online survey as well as usability of advanced REDCap functionality to display questions and response options, and feasibility of complex survey components such as skip pattern functionality. First, we distributed the link for the web-based draft survey among co-investigators and study team members, each of whom represented one or more aspects of the target study population. This included individuals who were a part of phase 2, as well as individuals who had not yet seen the survey instrument. Along with the survey link we provided instructions to respond with specific recommendations or issues that arose. In an iterative approach, as we received suggestions, a team of 6 study investigators and research assistants (RR, MSW, MG, MM, MV, and ME) regularly met via videoconference to reach consensus on how to revise the online survey. Thereafter, the same team systematically piloted all REDCap functionality on multiple internet browsers and corrected errors. This cycle of testing, revising, and retesting the survey instrument in REDCap continued until we identified no further concerns.

Results

Phase 1: development of the preliminary survey sections and questions

Between April and August 2019 we held 4 multisectoral (including approximately 20 clinicians, workforce specialists and researchers) co-investigators (many of which were clinician experts in abortion care themselves) and knowledge user meetings to iteratively refine the survey population, inform the initial survey draft and agree to the changes compared to the CAPS 2012. Table 2 highlights the differences between the surveys’ study populations, types of abortion care and survey design. In 2012, the majority of abortion services were provided in facilities that focused on abortion care. We primarily surveyed administrators of these facilities in CAPS 2012, and asked them to pass on the survey to at most 5 clinicians in their facility. Due to the availability of mifepristone for first trimester MA since 2017, we anticipated a move of abortion care from facilities with a focus on abortion care into primary care clinics. In order to capture this change, we considered the clinician as the primary unit of recruitment rather than the clinic administrator and expanded the survey sections for individual providers while shortening the clinic administrator section. Data on second and third trimester MA which are provided in hospitals was scarce in our 2012 survey. The 2012 survey section on second trimester MA had not included many details and due to our recruitment focus on non-hospital-based facilities in 2012, we did not capture this type of abortion care well. To include this care more comprehensively in 2019, we expanded the second trimester MA section, added specific third trimester language and placed the questions in a clinician rather than administrator section. We further modified our recruitment strategy to include individual clinicians in both hospitals and non-hospital based settings.

Table 2 CAPS sections, topics and design in 2012 compared to 2019

The first version of the survey instrument consisted of 6 sections and up to 58 questions per sub-section (however, due to the branching logic respondents would not see all questions). The survey sections were demographics (Sect. 1), clinical abortion practices (Sect. 2; divided into subsections of first trimester MA, first trimester surgical abortion, second trimester surgical abortion, and second/third trimester MA), administrator (Sect. 3), diverse populations (Sect. 4), stigma and resilience (Sect. 5), and remuneration and future research (Sect. 6). We used questions from our CAPS 2012 survey when possible; however, the majority required modifications due to the changes in the workforce, new regulations, availability of mifepristone for MA and updated guidelines.

Phase 2: content validation via a modified Delphi method with a panel of clinical experts

In September and October 2019, we held 7 expert panel meetings, composed of 5–8 participants each. These represented abortion providers who were: physicians, NPs, midwives, and administrators; from both urban and rural communities across Canada in addition to researchers. Table 3 depicts a list of meetings, their topics, panelist characteristics and key findings of the content analysis including the survey components found to be not feasible. Recurring themes between panels were (1) the desire to capture as much information as possible, including constructs such as cultural or structural competence [28, 29], while avoiding survey fatigue; (2) balancing pros and cons of open-ended questions as a tool to explore respondent’s experiences and attitudes that however might increase survey fatigue and would also require qualitative data analysis; (3) offering questions and answer options that were applicable to variations in service delivery between provinces, rural and urban respondents as well as different provider types; and (4) reaching consensus on clarity of language around terminology and their definitions.

Table 3 Expert panel meetings in preparation of the CAPS 2019

Phase 3: pilot testing of the draft survey

In January 2020, we piloted the web-based survey with approximately 25 co-investigators and knowledge users, all of whom were clinicians and within the target population. Based on their feedback we completed minor modifications, including some wording changes for further clarity, correcting spelling errors and changing the order of questions in February 2020. From March to June 2020 we piloted and optimized REDCap programming and found that advanced program functionality made it feasible to achieve a user friendly survey display and experience including complex skip pattern logic. The CAPS 2019 was originally set to launch in the spring 2020, as information and experiences related to 2019 abortion care would be readily recalled. However, due to the COVID-19 pandemic, our team recognized that clinicians would not have the time and energy to respond to a survey. Therefore, the survey launch date was postponed to July 2020. The final survey instrument of our cross-sectional, national, bilingual, web-based CAPS 2019 including the anticipated time it would take to complete the individual sections is summarized in Table 4.

Table 4 Final CAPS 2019 survey instrument sections and time to complete them

Discussion

We developed a cross-sectional, national, bilingual, web-based survey instrument that we piloted and revised in a rigorous and iterative approach until face validity and usability were assured. The CAPS 2019 explored the abortion provider workforce, the care they provide and their experiences with stigma and harassment. It was developed in order to capture changes compared to 2012 and after the implementation of mifepristone for first trimester MA in Canada in 2017 as well as other regulatory changes and updated clinical practice guidelines [5, 9, 10, 14, 15]. We navigated the challenge that, while we aimed to build on the 2012 survey to preserve question validity and to be able to directly compare our results, the survey instrument needed considerable adaptation and inclusion of additional content to reflect the Canadian context and changes in abortion care since 2012.

Our study process resulted in a more comprehensive survey than either our CAPS 2012 [2, 3], or prior U.S. iterations [16, 30, 31] from which the CAPS 2012 had been adapted. We moved the primary unit of recruitment and most survey questions from the abortion facility administrator to the individual clinician. We included NPs providing first trimester MA and expanded the questions on second and third trimester MA. As in the CAPS 2012, we relied on a validated scale to explore respondents’ experience with stigma and harassment [17]. Our national and multisectoral team allowed us to recognize that some topics, such as interdisciplinary collaborative care between midwives or nurses and physicians were too heterogeneous and complex to be feasibly captured by a survey, but rather required other research methods such as interviews, which could be conducted in a separate project.

Limitations: Overall our option to use previously fielded or validated questions was limited. The anticipated shift from abortion facility-based services to primary care, which differs from the service delivery in the USA [16]. Limited the degree to which we could use the previously fielded U.S. survey mifepristone first trimester MA questions. Similarly, the shift from administrator to clinician questions required modification of prior questions. Phase 3 pilot testing focused on the usability and face validity of the online survey instrument. We chose not to include specific measurements such as Cronbach alpha coefficients to assess internal consistency, test–retest reliability testing, or factor analysis to describe variability among items. While the co-investigators participating to pilot test our survey were clinical experts in the field and within the target population, it is possible that their feedback was influenced by their role on the study team, and that they were not naïve to the survey. We think, however, that our extensive panel group meetings and piloting in addition to using previously [32] fielded or validated scales when possible provided good reassurance in regard to question validity. Additionally, we previously successfully developed and piloted several effective survey instruments using a multiphase approach usually forgoing quantitative testing of specific question parameters [32,33,34,35].

Strengths of our study included the expansion of the study population to include previously understudied aspects of abortion care such as second and third trimester MA. Further, the web-based functionality allowed us to use branching logic to mitigate the length of our survey and to limit survey fatigue. REDCap additionally offered the opportunity to program individual answer options or combinations of answers in a way that limited respondent errors.

Conclusions

The CAPS 2019 development study used a rigorous multiphase approach to develop a comprehensive and nationally representative survey instrument of the abortion workforce, their clinical care and experience with stigma and harassment. It may provide the foundation for surveys in other countries. Results from our survey will provide data to inform knowledge translation activities of decision-makers in Canada aiming to improve equitable access to high quality abortion care.