- Study Protocol
- Open Access
Leveraging behavioral economics and implementation science to engage patients at risk for suicide in mental health treatment: a pilot study protocol
Pilot and Feasibility Studies volume 8, Article number: 181 (2022)
Primary care is an ideal setting to connect individuals at risk for suicide to follow-up care; however, only half of the patients referred from the primary care attend an initial mental health visit. We aim to develop acceptable, feasible, low-cost, and effective new strategies to increase treatment initiation among at-risk individuals identified in primary care.
We will conduct a multi-phase, mixed-methods study. First, we will conduct a chart review study by using administrative data, including medical records, to identify characteristics of primary care patients at risk for suicide who do or do not attend an initial mental health visit following a referral. Second, we will conduct a mixed methods study by using direct observations and qualitative interviews with key stakeholders (N = 65) to understand barriers and facilitators to mental health service initiation among at-risk individuals. Stakeholders will include patients with suicidal ideation referred from primary care who do and do not attend a first mental health visit, primary care and behavioral health providers, and individuals involved in the referral process. We also will collect preliminary self-report and behavioral data regarding potential mechanisms of behavior change (i.e., self-regulation and social support) from patients. Third, we will leverage these findings, relevant frameworks, and the extant literature to conduct a multi-arm, non-randomized feasibility trial. During this trial, we will rapidly prototype and test strategies to support attendance at initial mental health visits. Strategies will be developed with subject matter experts (N = 10) and iteratively pilot tested (~5 patients per strategy) and refined. Research will be completed in the Penn Integrated Care Program (PIC), which includes fourteen primary care clinics in Philadelphia that provide infrastructure for electronic referrals, patient communication, and data access.
We will leverage frameworks and methods from behavioral economics and implementation science to develop strategies to increase mental health treatment initiation among individuals at risk for suicide identified in primary care. This project will lead to an evaluation of these strategies in a fully powered randomized trial and contribute to improvements in access to and engagement in mental health services for individuals at risk for suicide.
Suicide is a major global public health problem. In 2018, there were approximately 800,000 suicide deaths worldwide  and over one million people attempt suicide each year . The effects of suicide are far reaching, affecting family members, friends, and communities, and costing the USA more than $90 billion annually . Despite increased attention to suicide as a significant public health concern, current suicide rates are approximately 30% higher than at the turn of the millennium . Although evidence-based practices (EBPs) for suicide prevention exist and individuals are less likely to die by suicide if they engage in services [5,6,7], individuals at risk for suicide often have difficulty engaging in mental health treatment. To date, most research in this area has examined mental health treatment initiation following acute services for patients at risk for suicide [8, 9]. Studies have not yet developed or tested strategies to increase mental health treatment initiation among individuals at risk for suicide identified in primary care, even though a large proportion of these individuals do not go on to initiate mental health treatment following referral [10,11,12].
Primary care is a logical place to identify and connect individuals at risk for suicide to follow-up care because most adults visit primary care annually . Approximately two thirds of people who die by suicide interact with a primary care clinician in the year prior to death . While mental health screening and referral are common in primary care, even when patients are referred to a mental health provider within the same practice only half attend an initial visit [15, 16]. Strategies such as reminder calls and texts, motivational and informational interventions, and case management have only demonstrated small to moderate effects . Even when these interventions are implemented, about 40% of patients do not initiate treatment , indicating the need for more effective methods for increasing treatment initiation.
Behavioral science theories and methods hold promise for improving patient engagement . Behavioral economics, which combines classic economic theory with insights from psychology to understand common, predictable factors that influence decision making , can be leveraged to improve treatment engagement . Applying behavioral insights effectively requires two steps. First, it is essential to understand the target problem, context, and behavioral and structural barriers and facilitators. Principles from implementation science provide guidance for achieving this understanding and can enhance knowledge of the ways in which organizational factors and stakeholder perspectives influence innovation adoption [21, 22]. Second, it is important to understand key mechanisms that underlie changes in behavior. The Science of Behavior Change (SOBC) provides an experimental therapeutic framework that guides the identification of mechanisms of change to better understand how and why certain strategies work . Factors related to self-regulation (e.g., temporal discounting) and interpersonal and social processes (e.g., emotional social support) may impact the decision to initiate treatment. Thus, it is important to target putative mechanisms, such as temporal discounting and social support, when designing strategies and to attend to contextual factors that impact implementation success (see Fig. 1).
The EAST framework
As noted above, this study draws on principles from behavioral economics, implementation science, and experimental therapeutics. Behavioral economists have identified a wide range of heuristics in human decision making (i.e., cognitive biases), many of which contribute to poor health outcomes. The EAST Framework asserts that, to encourage behavior change and reduce people’s reliance on cognitive biases, the desired behavior should be Easy, Attractive, Social, and Timely . The first of the EAST principles, make it Easy to change behavior, includes strategies such as utilizing default options (e.g., auto enrollment in 401k), reducing the effort required to perform an activity, and simplifying messaging. The second principle is to make the desired behavior Attractive (e.g., use of attention-grabbing messaging and incentives). The third principle, make it Social, includes demonstrating that most people engage in the desired behavior, harnessing the power of networks, and encouraging people to express commitment to others. The final principle, make it Timely, focuses on careful consideration of when to implement strategies, such as providing prompts when people are most likely to be receptive or engaging in collaborative planning for action.
Four steps are recommended to apply these insights : (1) define the outcome (i.e., increase attendance at first mental health visit), (2) understand the context (i.e., what stops people from attending), (3) build the intervention (i.e., develop implementation strategies), and (4) test, learn, and adapt (e.g., evaluate implementation strategies and engage in rapid prototyping). Principles from both experimental therapeutics and implementation science can be leveraged to execute these steps with maximum effectiveness and inform our conceptual model (Fig. 1). For example, in order to understand the context (step 2), it is essential to examine why people have difficulty engaging in a desired behavior (i.e., the context) and why or how an intervention works to change behavior (i.e., mechanisms). Principles and methods from implementation science, such as conducting contextual inquiries to identify barriers and facilitators to evidence-based practice implementation, can be applied to better understand the context and determinants affecting treatment engagement.
The science of behavior change (SOBC) framework
The SOBC Common Fund Program, which applies an experimental therapeutics approach to behavior change, has identified key domains of mechanisms relevant to multiple health behaviors and clinical outcomes that informed our conceptual model and measurement strategy . Mechanisms, critical to advancing the science of implementation, refer to “processes that are responsible for change” or the specific way by which implementation strategies affect implementation success and patient outcomes. Identification of mechanisms can lead to more efficient, effective, and tailored strategies based on the practice of interest and the implementation context .
Self-regulation and interpersonal and social processes, two SOBC domains, may mediate the relationship between engagement strategies and treatment initiation. Within the domain of self-regulation, temporal discounting—the cognitive process of discounting rewards that are not immediate—may be a relevant mechanism (e.g., choosing to turn off the alarm clock to gain the immediate reward of sleeping versus attending an initial therapy appointment, which promises the future reward of symptom reduction). A potential strategy to target this mechanism might be a financial incentive to attend treatment sessions . A second putative SOBC mechanism is social-emotional processes (e.g., emotional social support), one type of interpersonal, or social process. Caring Contacts, for example, are brief, non-demanding letters or texts expressing care or concern that have been associated with decreases in suicidal ideation and deaths and may target this mechanism [27, 28].
The promise of rapid prototyping
To address this public health crisis, we must expedite the rate at which innovations to engage individuals at risk for suicide in treatment enter routine practice [29, 30]. Traditionally, intervention effectiveness is tested by expensive, lengthy trials, followed by dissemination and implementation activities. Rapid prototyping involves a series of brief, rigorous tests to optimize operations in early-study stages. Using rapid prototyping increases the likelihood that interventions tested in trials will be acceptable, feasible, and effective, thereby shortening the process of developing and implementing interventions. Industries outside of health care commonly use this approach to learn quickly and “de-risk” decision-making prior to a large rollout [31, 32].
Leveraging the work of the Penn ALACRITY (Advanced Laboratories for Accelerating the Reach and Impact of Treatments for Youth and Adults with Mental Illness) Center for transforming mental health delivery through behavioral economics and implementation science , we will develop, pilot, and optimize strategies to increase treatment initiation for persons at risk for suicide and accelerate the pace at which promising strategies enter routine care. We will first develop strategies in collaboration with a team of experts in suicide, implementation science, and behavioral economics. We will then iteratively pilot test and refine these preliminary strategies using rapid prototyping. Through this process, we will uncover additional barriers and facilitators to implementing strategies that will allow us to further refine and optimize them prior to testing them in a future randomized trial.
The main objective of this study is to develop acceptable, feasible, low-cost, and effective strategies that increase patients’ treatment initiation (i.e., attendance at a first mental health visit) following identification of suicide risk in primary care. To complete this objective, we will partner with a large and diverse health system to conduct a multiphase, mixed-methods study (see Fig. 2). We will first conduct a chart review study to identify characteristics of patients at risk for suicide who do or do not attend an initial mental health visit following a referral from primary care and variation in patient attendance among referring providers and practices (Aim 1). We will then conduct a mixed methods study by using qualitative interviews and direct observations with stakeholders to identify barriers and facilitators to mental health treatment attendance for individuals at risk for suicide (Aim 2). In accordance with an experimental therapeutics approach to behavior change, we will also collect preliminary data on mechanisms of action that may impede treatment attendance (i.e., self-regulation and social support) from patients using self-report and behavioral measures [34,35,36,37]. Finally, using procedures from implementation science  and behavioral economics , we will leverage the information gleaned from Aims 1 and 2, relevant theories and frameworks (e.g., EAST ;), and the extant literature to develop preliminary strategies to support attendance at first mental health visits. We will then conduct a multi-arm, non-randomized feasibility trial to rapidly prototype and test strategies to optimize engagement by using a series of rigorous, iterative tests (Aim 3). To maximize generalizability, we will develop strategies for increasing attendance in both collaborative care and outpatient specialty mental health.
In 2018, a collaborative care program was developed through a partnership between Penn’s Department of Psychiatry and Primary Care Service Line and is referred to as the Penn Integrated Care (PIC) program (a detailed description is provided by Wolk and colleagues ;). Fourteen primary care clinics in Philadelphia and the surrounding counties that range in size and include family medicine and internal medicine practices have PIC services embedded in their clinics. These practices provide infrastructure for electronic referrals, patient communication, and data access. All practices use a sophisticated electronic health record (EHR) system and patient registries that enables the capture of episodes of care and billing for services using Collaborative Care billing codes . A master’s level mental health clinician is embedded within each practice and delivers traditional Collaborative Care services in collaboration with a consulting psychiatrist , as well as brief evidence-based interventions, including behavioral activation, motivational interviewing, and problem-solving therapy, for those with mild to moderate mental illness . The PIC model also includes a centralized “Resource Center” staffed by bachelors-level mental health intake coordinators who are supervised by licensed clinicians. This Resource Center assesses, triages, and refers primary care patients in need of mental health services to the health system and community providers.
Aim 1 (chart review study)
Aim 2 is to identify the characteristics of patients at risk for suicide who do or do not attend an initial mental health visit following referral from primary care and variation in patient attendance among referring providers and practices.
The sample will consist of adults referred for mental health services from primary care in PIC practices, including all Medicare, Medicaid, and commercial insurance enrollees. Based on preliminary data, we anticipate that our study sample will include 916 adults with a score of ≥1 on the PHQ-9 suicide item who were referred to PIC between 2018 and mid-2021. By comparison, there were 2470 adults with a score of ≥1 on the PHQ-9 suicide item who were referred to specialty care and/or did not have insurance coverage for PIC.
We will use administrative data, including medical records and insurance claims data, to identify which patient characteristics (e.g., diagnoses, demographics) are associated with attendance at mental health visits and examine variations in patient attendance among referring providers and practices. Our primary data source will be medical records; when available, we will also use insurance claims data. These analyses are exploratory.
We will rely primarily on medical records to examine treatment engagement among patients with suicidal ideation or behaviors. Treatment engagement will be measured by attendance at a mental health visit following a referral. For a subset of patients, we will review insurance claims data to examine whether they utilized specialty care outside of Penn Medicine. The patients selected will depend on access to insurance claims and approval to analyze these claims.
Medical records will be de-identified and sent securely to our study team by program leadership and the Data Analytics Center at Penn Medicine. They will include patient demographic and clinical information, including age, gender, race/ethnicity, insurance type, any utilization of care at Penn Medicine (e.g., specialty care, hospitalizations), prescriptions, patient acuity (on a 1–4 scales), and behavioral health screenings (e.g., PHQ-9 for depression, GAD-7 for anxiety [43, 44];).
Insurance claims will provide similar demographic and utilization data as medical records but will additionally include any care utilization occurring outside of Penn Medicine. We anticipate that insurance claims will be accessible for Medicaid enrollees from Philadelphia and/or enrollees of one major commercial insurer.
We will compare characteristics across (1) patients who screen positive for suicidal ideation (have a score of ≥1 on item 9 of the PHQ-9, which is administered at least annually to all Penn patients), and attend the first mental health visit following referral , to (2) patients who screen positive for suicidal ideation and do not attend the first mental health visit following referral. We will detect differences in group means using t tests, then use logistic regressions to identify characteristics that are associated with engagement.
Our equation of interest is Ai = α + Xiβ + εI, where A is a binary variable that indicates whether patient i attended their initial follow-up appointment after the suicide screening. The vector X includes covariates, including age, gender, and race/ethnicity. The unit of analysis is the patient level. We will cluster standard errors at the clinic level and assess model appropriateness and fit using the relevant diagnostics.
We will also compare the two groups’ service use by examining outpatient, inpatient, and pharmacological treatment in the year prior to screening. For patients with accessible insurance claims, we will measure service utilization after the PHQ-9 screen to track mental health treatment engagement outside of Penn Medicine . This will serve as a validity check of medical records to measure engagement because the claims will capture relevant primary or specialty care accessed outside of Penn Medicine.
Aim 2 (mixed methods study)
Aim 2 is to use qualitative interviews and direct observations with stakeholders, as well as self-report and behavioral measures from patients, to identify barriers and facilitators to mental health treatment attendance for individuals at risk of suicide.
Following the initial inspection of the EHR data in Aim 1, we will identify patients referred from Penn primary care to mental health services following a positive screen for suicidal ideation on the PHQ-9 who did or did not attend their scheduled mental health visit over the past 3 months . All patients must (1) be 18 years and older, able to communicate in English, and able and willing to provide informed consent and (2) present with elevated suicidal ideation per item 9 of the PHQ-9 (item score ≥ 1) completed during a primary care visit. Exclusion criteria include (1) an imminent risk of suicide (i.e., plan with the method and intent to act) and (2) currently experiencing a psychotic episode requiring emergency services and/or precluding abilities to provide informed consent.
Additionally, we will interview primary care and behavioral health providers, as well as individuals involved in the implementation of integrated care at Penn. We will include Penn primary care providers who have referred one or more patients with a positive screen for suicidal ideation for mental health services over the past 3 months. Behavioral health providers will include those practicing at Penn and in community clinics to which Penn patients are routinely referred. Once we identify patients and providers, we will randomly select individuals to invite to participate in qualitative interviews. Individuals involved in the implementation of integrated care at Penn include leaders in Primary Care and Psychiatry, supervisors of behavioral health providers, and mental health intake coordinators. All providers, leaders, and intake coordinators must (1) be 18 years and older, able to communicate in English, and able and willing to provide informed consent and (2) be employed in an appropriate primary care or behavioral health role.
We anticipate conducting a total of 65 interviews including with patients who do (n = 12) and do not (n = 12) attend a first mental health visit, primary care providers (n = 12), behavioral health providers (n = 12), intake coordinators (n = 5), and leaders (n = 12). While we anticipate conducting 65 interviews, we will continue until saturation is achieved.
Guided by EAST, we will use observation and semi-structured qualitative interviews with key informants to understand stakeholder experiences and perspectives around mental health treatment engagement. Interviews will include questions about barriers and facilitators of treatment initiation and attendance. For patient stakeholders, we also will assess self-regulation and social support with self-report and behavioral measures. Participants will be compensated $30 an hour for participating. Finally, we will observe intake coordinators in the Resource Center as they assess and schedule patients with mental health providers. Although we are underpowered to detect significant effects, we hypothesize that lower self-regulation and lower social support will predict patient non-attendance at initial mental health visits.
Measures (Table 1)
We will measure self-regulation, specifically temporal discounting, using two SOBC-recognized measures: (1) The Kirby Delay-Discounting Task (KDDT)  and (2) The Consideration of Future Consequences (CFC) Scale .
The KDDT is a behavioral task that assesses the tendency to devalue rewards as they become more temporally remote (known as delay-discounting) that has demonstrated reliability and validity [34, 35, 45]. The KDDT has 27 items that provide two choices for each question: an immediate reward or a larger but delayed reward (e.g., $19 today or $25 in 53 days). Acceptable test-retest reliability has been established at 5 weeks (r = .77) and 1 year (r = .71; 37).
The CFC scale is a 12-item self-report measure of consideration of future consequences or forward-thinking behavior . Each CFC item (e.g., “Often I engage in a particular behavior in order to achieve outcomes that may not result for many years”) is rated on a 1-5 Likert-type scale ranging from “extremely uncharacteristic” to “extremely characteristic.” The total score ranges from 12 to 60, with higher scores indicating greater consideration of future consequences. The CFC has demonstrated strong internal consistency (Cronbach’s alpha .80–.86) and acceptable test-retest reliability (r = .72) .
Interpersonal processes (emotional social support)
We will measure perceived emotional social support using the 12-item Multidimensional Scale of Perceived Social Support (MSPSS) . The MSPSS, a self-report measure recognized by the Research Domain Criteria Project (RDoC), is rated on a 1–7 Likert-type scale ranging from “Very Strongly Disagree” to “Very Strongly Agree” in response to each statement (e.g., “I get the emotional help and support I need from my family”). The total score ranges from 12 to 84, with higher scores indicating greater perceived social support. Internal consistency (Cronbach’s alpha = .88) and test-retest reliability (r = .85) are well established . Divergent validity of the MSPSS has also been established; the measure has been found to be inversely correlated with measures of depression and anxiety .
Structural barriers and provider attitudes (qualitative interviews)
Individual, semi-structured interviews will be conducted to understand cognitive biases and contextual factors that may impact treatment engagement broadly and attendance at initial mental health visits in particular. We developed interview guides for each stakeholder group to ensure uniform inclusion and sequencing of topics and to allow for valid comparison across interviews. The first section asks about general views related to mental health services, including perceptions of (1) what constitutes mental health care, (2) why mental health care is or is not important, (3) barriers and facilitators to attending mental health visits, (4) use of and experience with mental health services, and (5) engagement strategies used or recommended (for providers, intake coordinators, leaders).
The second section asks about relevant cognitive biases, including (1) what is appealing and satisfactory about mental health care, (2) what is unappealing and unsatisfactory about mental health care, and (3) perceived expectations of others related to mental health care. Finally, patients will be asked to recount the screening, referral, and appointment process and, if they attended a mental health visit, steps implemented or supports utilized to attend. For patients who did not attend, we will similarly ask them to recount any steps they attempted to take and barriers they encountered.
Provider behavior (observations)
Study staff will spend one full week observing intake coordinators in the centralized mental health assessment, triage, and referral management center (i.e., PIC Resource Center). Observations will include detailed field notes of processes and procedures to understand leverage points for increasing engagement.
We will upload interviews and field notes into NVivo for data management and analysis. Analysis will be guided by an integrated approach that includes identification of a priori attributes of interest (i.e., EAST principles) and modified grounded theory, which provides a rigorous, systematic approach to identifying emergent codes and themes . This approach uses an inductive process of iterative coding to identify recurrent themes, categories, and relationships. After initial exploration of data, a comprehensive coding scheme will be developed and applied to all data. The product will be a fine-grained descriptive analysis of the challenges to engagement in mental health services, including issues related to attendance at first visit following referral. Two members of the research team will separately double code a sample of transcripts to assess reliability of the coding scheme. Disagreements will be resolved through team discussion.
To explore potential mechanisms, we will use descriptive statistics to examine the distribution of self-report (CFC and MSPSS) and behavioral task (KDDT) scores. We will use one-way ANOVAs to examine the relationships between attendance and scores on the CFC, MSPSS, and KDDT. These analyses are exploratory to inform future trials; the study will not be powered to detect effects.
Mixed methods analysis
We will integrate quantitative data from Aim 1 and interviews using the following taxonomy: the structure is Quan—QUAL, the function is to expand upon the quantitative findings to understand the process of engagement as experienced by stakeholders, and the process is connecting . To integrate the quantitative and qualitative methods, we will follow the NIH guidelines for best practices . We will use the quantitative data to identify patterns in the qualitative data. We will enter quantitative findings into NVivo as attributes of each participant (i.e., did/did not attend; referral to PIC or community; KDDT, CFC, and MSPSS scores). These attributes will be used to categorize and compare important themes among subgroups.
Aim 3 (multi-aim, non-randomized feasibility trial)
Aim 3 is rapidly prototype and test strategies to optimize engagement by using a series of rigorous, iterative tests.
In Aim 3, we will develop preliminary strategies to support attendance at initial mental health visits and iteratively test and refine these strategies through rapid prototyping. Secondarily, we will explore the role of two potential mechanisms, self-regulation and social support, in the use of engagement strategies and attendance at initial mental health visits.
Each strategy will be tested with approximately 5 patients. All patients must be (1) 18 years and older and able to communicate in English and (2) present with elevated suicidal ideation on item 9 of the PHQ-9 (item score ≥1) completed during a primary care visit. In addition, patients must not (1) currently be experiencing a psychotic episode requiring emergency services and/or precluding their ability to provide informed consent, (2) have a documented diagnosis of dementia in the past 2 years, (3) have primary care provider notes demonstrating that participation is not indicated, and (4) have already received a study engagement strategy following a different primary care visit. We will leverage the infrastructure of the Resource Center to recruit from the pool of patients who were referred for a mental health intake following a positive screen for depression or suicide risk in primary care based on the EHR. This provides a mechanism for recruiting patients while also ensuring that there is appropriate clinical screening and expertise to triage individuals in need of acute services. We anticipate that 25 patients will complete the temporal discounting and social support behavioral task and self-report measures, though additional patients may receive the strategies. Attendance data will be available from any individual who receives a strategy as part of routine care quality assurance.
A team of experts in implementation science (n = 2), behavioral economics (n = 2), suicide (n= 2), and primary care and behavioral health (n = 4, total N = 10) will convene a half-day retreat to guide Aim 3 activities and develop strategies for rapid prototyping. This team will review key barriers and facilitators to attendance at initial mental health visits identified in Aims 1 and 2 and then generate strategies for facilitating treatment initiation that address barriers and leverage facilitators. The EAST framework will guide the selection of strategies . We will also favor simple, established strategies  with freely available resources (e.g., Caring Contacts templates from Now Matters Now) to promote scalability , but will also consider other potential implementation strategies as indicated . We will explore both low- and high-tech (e.g., EHR integration) strategies to maximize relevance across contexts. Based on the literature, previous work, and our conceptual model, we hypothesize that strategies that counteract delayed discounting tendencies (e.g., incentives) and foster emotional social support (e.g., Caring Contacts) will be needed .
We will then use rapid prototyping to test and refine the strategies identified by the team of experts , a process that has been applied successfully at Penn over the past 6 years [51,52,53,54]. The goal of rapid prototyping is to determine the best strategies for broader implementation in a subsequent trial. Prototype strategy designs will be validated through rapid-cycle tests in this real-world context and in a manner that is quick and inexpensive (see Table 2 for examples). This will allow us to fail fast and learn quickly. As we deploy strategies and learn from our experiences, we will adapt and iterate to refine promising strategies.
Strategies will be rolled out in the PIC Resource Center over 6 months. During this period, strategies will be implemented by an embedded mental health intake coordinator as part of routine care. Consent to receive strategies is, therefore, not required. However, because participants will complete measures assessing acceptability and feasibility of these strategies, as well as temporal discounting and social support, we will employ an IRB-approved modified consent procedure prior to implementing strategies that includes a waiver of written consent. The modified consent procedure will consist of (1) a brief description of the study purpose, procedures, and potential risks; (2) notification that the strategy is part of a research project for which participation is voluntary; and (3) instructions on declining further participation. This modified approach will allow us to circumvent potential confounds associated with limiting a trial examining engagement to those who provide active, written informed consent to receive engagement strategies. By waiving the requirement of traditional, written informed consent, we will minimize the likelihood of a biased sample and maximize the validity and generalizability of our results [57, 58]. We anticipate that each strategy will be tested with ≥5 patients, but this is flexible; additional tests will be implemented as needed.
After testing each strategy, we will quantitatively assess patients’ perceptions of its feasibility, acceptability, and appropriateness using three brief, validated questionnaires: Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM), that were developed to better evaluate and monitor implementation outcomes . Each questionnaire consists of four items (total of 12 items) rated on a 1–5 Likert-type scales ranging from “Completely disagree” to “Completely Agree.” The total score ranges from 4 to 20 for each measure, with higher scores indicating greater acceptability, appropriateness, or feasibility, respectively. Reliability was demonstrated with Cronbach’s alphas of .85 for acceptability, .91 for appropriateness, and .89 for feasibility, and test-retest reliability was demonstrated with correlations of .80 for acceptability, .73 for appropriateness, and .88 for feasibility.
Patients will then complete the temporal discounting and social support behavioral task and self-report measures (the KDDT, CFC, and MPSS) described in Aim 2.
Descriptive statistics will be used to describe feasibility, acceptability, and appropriateness of strategies as assessed with the AIM, IAM, and FIM. We will also use logistic regressions to examine the impact of engagement strategies on attendance at initial mental health visits using attendance information derived from the administrative data (did/did not attend). These analyses will provide us with a preliminary estimate of the effectiveness of each engagement strategy. Finally, we will use descriptive analyses to examine scores on the temporal discounting and social support self-report and behavioral tasks, as well as relationships between scores, engagement strategies, and attendance at initial mental health visits. These analyses are exploratory to inform mechanisms to investigate in future trials.
Consistent with recommendations for pilot studies, we have not conducted power analyses for Aims 1–3. We instead focus on exploring the data at this stage , as well as examining the feasibility and acceptability of strategies to support attendance at initial mental health visits. Analyses will be oriented towards identifying promising strategies and mechanisms to focus on in subsequent studies.
Primary care has been identified as a crucial setting for connecting individuals at risk for suicide to follow-up care, yet only half of patients referred to mental health treatment by primary care providers attend an initial visit [15, 16]. Surprisingly, no studies have yet developed or tested strategies to increase mental health treatment initiation for individuals at risk for suicide identified in primary care. This study aims to use frameworks and methods from behavioral economics, implementation science, and experimental therapeutics to develop acceptable, feasible, low-cost, and effective strategies to increase treatment initiation in this population. To do this, we will conduct a multiphase, mixed-methods study to identify characteristics of patients at risk for suicide who do and do not initiate mental health treatment, examine barriers and facilitators to treatment initiation among patients at risk for suicide, and rapidly prototype and test strategies to increase engagement.
This study has multiple strengths. First, we will partner with a large, diverse health system to identify barriers and facilitators to treatment initiation and develop and test potential solutions. Second, we will leverage rich administrative data to capture clinical characteristics and mental healthcare utilized both inside and outside the health system. This approach addresses key limitations of medical record use, which does not capture utilization outside the health system, and insurance claims, which lack verified diagnoses . Third, unlike most studies of patient engagement, we will both quantitatively and qualitatively examine engagement strategies. Fourth, we will concurrently assess the perspectives of multiple stakeholder groups, including providers, patients, and health system leaders, which is a valuable approach that is rarely taken. Fifth, we will integrate methods and measures from behavioral economics, implementation science, and experimental therapeutics to develop and refine implementation strategies and elucidate generalizable mechanisms of engagement. Finally, we will rapidly prototype and test engagement strategies, which increases the potential for successful implementation and decreases the research-to-practice timeline.
Despite these strengths, the current study also has several limitations. In particular, the study will be conducted in a single, well-resourced health system with an existing integrated care program; the extent to which the results will generalize to other health systems is unknown. However, the demographic and economic diversity of the Penn Medicine patients may offset threats to external validity. In addition, given our focus on establishing feasibility and acceptability of implementation strategies, our sample size for testing strategies will be small. As such, statistical power will be limited, precluding our ability to draw firm conclusions about the strategies’ effectiveness. Furthermore, although we will examine associations between mechanisms of behavior change (i.e., self-regulation and social support) and treatment initiation among individuals at risk for suicide, our small sample size will preclude us from testing whether these mechanisms mediate the relationship between engagement strategies and treatment initiation. The current research, however, will be pivotal in paving the way for larger, confirmatory trials testing the effectiveness of these strategies and relevance of these mechanisms of behavior change. Finally, the current study is focused on treatment initiation rather than ongoing treatment engagement. Given that this is a formative trial, we chose to operationalize treatment engagement in this way to maximize parsimony, feasibility, and generalizability of our findings. It is possible, however, that effective strategies may differ for initial and ongoing treatment engagement. Examining strategies to improve ongoing treatment engagement is beyond the scope of the current trial, but we aim to broaden our operationalization of engagement in a subsequent larger, adequately powered trial.
In sum, this study will produce novel information about factors that enable and prevent primary care patients at risk for suicide from following up on mental health treatment. It will also produce a menu of strategies likely to increase treatment initiation among these individuals, along with potential mechanisms that may account for changes in treatment initiation. We anticipate that this project will lead to future studies testing the strategies developed to increase treatment initiation in a randomized, multi-site implementation trial. Our ultimate goal is to improve access to and engagement in mental health services for individuals at risk for suicide. This goal has the potential to reduce mortality among the majority of individuals with suicidal ideation who regularly attend primary care but not specialty mental health care.
Availability of data and materials
Acceptability of Intervention Measure
Advanced Laboratories for Accelerating the Reach and Impact of Treatments for Youth and Adults with Mental Illness
Analysis of variance
Consideration of Future Consequences scale
Easy, Attractive, Social, and Timely
Electronic health records
Feasibility of Intervention Measure
General Anxiety Disorder-7
Intervention Appropriateness Measure
Kirby Delay-Discounting Task
Multidimensional Scale of Perceived Social Support
National Institutes of Health
Patient Health Questionnaire-9
Penn Integrate Care Program
Research Domain Criteria Project
Science of Behavior Change
Centers for Disease Control and Prevention, National Center for Health Statistics. National Ambulatory Medical Care Survey (NAMCS) 2014; https://www.cdc.gov/nchs/ahcd/index.htm
American Association of Suicidology. U.S.A. Suicide 2017: official final data. 2018. www.suicidology.org. Accessed 10 Dec 2018.
Shepard DS, Gurewich D, Lwin AK, Reed GA Jr, Silverman MM. Suicide and suicidal attempts in the United States: costs and policy implications. Suicide Life Threat Behav. 2016;46(3):352–62.
Centers for Disease Control and Prevention. Web-based Injury Statistics Query and Reporting System (WISQARS) [online]. 2005; http://www.cdc.gov/ncipc/wisqars.
Alonzo D, Moravec C, Kaufman B. Individuals at risk for suicide: mental health clinicians’ perspectives on barriers to and facilitators of treatment engagement. Crisis. 2017;38(3):158–67.
Dixon LB, Holoshitz Y, Nossel I. Treatment engagement of individuals experiencing mental illness: review and update. World Psych. 2016;15(1):13–20.
Erlangsen A, Lind BD, Stuart EA, Qin P, Stenager E, Larsen KJ, et al. Short-term and long-term effects of psychosocial therapy for people after deliberate self-harm: a register-based, nationwide multicentre study using propensity score matching. Lancet Psychiatry. 2015;2(1):49–58.
Miller IW, Camargo CA, Arias SA, Sullivan AF, Allen MH, Goldstein AB, et al. Suicide prevention in an emergency department population: the ED-SAFE study. JAMA Psychiatry. 2017;74(6):563–70.
Doupnik SK, Rudd B, Schmutte T, Worsley D, Bowden CF, McCarthy E, et al. Association of suicide prevention interventions with subsequent suicide attempts, linkage to follow-up care, and depression symptoms for acute care settings: a systematic review and meta-analysis. JAMA Psychiatry. 2020;77(10):1021–30.
Kessler R. Mental health care treatment initiation when mental health services are incorporated into primary care practice. J Am Board Fam Med. 2012;25(2):255–9.
Younes N, Passerieux C, Hardy-Bayle MC, Falissard B, Gasquet I. Long term GP opinions and involvement after a consultation-liaison intervention for mental health problems. BMC Fam Pract. 2008;9:41.
Cunningham PJ. Beyond parity: primary care physicians’ perspectives on access to mental health care. Health Aff. 2009;28:w490–501.
Blackwell D, Lucas J, Clarke T. Summary Health Statistics for U. S. adults: National Health Interview Survey, 2012. Atlanta, GA: Vital Health Stat National Center for Health Statistics; 2014.
Ahmedani BK, Simon GE, Stewart C, Beck A, Waitzfelder BE, Rossom R, et al. Health care contacts in the year before suicide death. J Gen Intern Med. 2014;29(6):870–7.
Solberg LI, Crain AL, Jaeckels N, Ohnsorg KA, Margolis KL, Beck A, et al. The DIAMOND initiative: implementing collaborative care for depression in 75 primary care clinics. Implement Sci. 2013;8(1):135.
Whitebird RR, Solberg LI, Jaeckels NA, Pietruszewski PB, Hadzic S, Unützer J, et al. Effective implementation of collaborative care for depression: what is needed? Am J Manag Care. 2014;20(9):699–707.
Oldham M, Kellett S, Miles E, Sheeran P. Interventions to increase attendance at psychotherapy: a meta-analysis of randomized controlled trials. J Consult Clin Psychol. 2012;80(5):928–39.
Lowenstein G, Asch DA, Volpp KG. Behavioral economics holds potential to deliver better results for patients, insurers, and employers. Health Aff. 2013;32(7):1244–50.
Volpp KG, Loewenstein G, Asch DA. Behavioral economics and health. In: Glanz K, Rimer BK, Viswanath V, editors. Health behavior: theory, research, and practice. San Francisco: Jossey-Bass, Inc./Wiley; 2015.
Keller PA, Harlam B, Loewenstein G, Volpp KG. Enhanced active choice: a new method to motivate behavior change. J Consum Psychol. 2011;21(4):376–83.
Sunstein CR. Nudges that fail. Behavioural Public Policy. 2017;1(1):4–25.
Wu SJ, Paluck EL. Designing nudges for the context: golden coin decals nudge workplace behavior in China. Organ Behav Hum Decis Process. in press.
Nielsen L, Riddle M, King JW, Aklin WM, Chen W, Clark D, et al. The NIH science of behavior change program: transforming the science through a focus on mechanisms of change. Behav Res Ther. 2018;101:3–11.
Service O, Hallsworth M, Halpern D, Algate F, Gallagher R, Nguyen S, et al. EAST: four simple ways to apply behavioral insights; 2014.
Kazdin AE. Mediators and mechanisms of change in psychotherapy research. Annu Rev Clin Psychol. 2007;3:1–27.
Post EP, Cruz M, Harman J. Incentive payments for attendance at appointments for depression among low-income African Americans. Psychiatr Serv. 2006;57(3):414–6.
Comtois KA, Kerbrat AH, DeCou CR, Atkins DC, Majeres JJ, Baker JC, et al. Effect of augmenting standard care for military personnel with brief caring text messages for suicide prevention: a randomized clinical trial. JAMA Psychiatry. 2019;76(5):474–83.
Reger MA, Gebhardt HM, Lee JM, Ammerman BA, Tucker RP, Matarazzo BB, et al. Veteran preferences for the caring contacts suicide prevention intervention. Suicide Life-Threat Behavior. 2019;49(5):1439–51.
McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635–45.
Crossing the Quality Chasm. A new health system for the 21st century. Washington, DC: National Academy Press; 2001.
Dopp AR, Parisi KE, Munson SA, Lyon AR. Integrating implementation and user-centred design strategies to enhance the impact of health services: protocol from a concept mapping study. Health Res Policy Syst. 2019;17(1):1.
Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implement Sci. 2015;10(1):21.
Beidas RS, Volpp KG, Buttenheim AN, Marcus SC, Olfson M, Pellecchia M, et al. Transforming mental health delivery through behavioral economics and implementation science: protocol for three exploratory projects. JMIR Res Protoc. 2019;8(2):e12121. https://doi.org/10.2196/12121.
Kirby KN, Marakovic NN. Delay-discounting probabilistic rewards: rates decrease as amounts increase. Psychon Bull Rev. 1996;3(1):100–4.
Kirby KN. One-year temporal stability of delay-discount rates. Psychon Bull Rev. 2009;16(3):457–62. https://doi.org/10.3758/PBR.16.3.457.
Strathman A, Gleicher F, Boninger DS, Edwards CS. The consideration of future consequences: weighing immediate and distant outcomes of behavior. J Pers Soc Psychol. 1994;66(4):742–52.
Zimet GD, Dahlem NW, Zimet SG, Farley GK. The multidimensional scale of perceived social support. J Pers Assess. 1988;52(1):30–41.
Fernandez ME, ten Hoor GA, van Lieshout S, et al. Implementation mapping: using intervention mapping to develop implementation strategies. Front Public Health. 2019;7(158)1–15.
Altman M, Huang TTK, Breland JY. Design thinking in health care. Prev Chronic Dis. 2018;15:E117.
Wolk CB, Last BS, Livesey C, Oquendo MA, Press MJ, Mandell DS, et al. Addressing common challenges in the implementation of collaborative care for mental health: the Penn integrated care program. Ann Fam Med. 2021;19(2):148–56. https://doi.org/10.1370/afm.2651.
Press MJ, Howe R, Schoenbaum M, et al. Medicare payment for behavioral health integration. N Engl J Med. 2017;376(5):405–7.
Archer J, Bower P, Gilbody S, Lovell K, Richards D, Gask L, et al. Collaborative care for depression and anxiety problems. Cochrane Database Syst Rev. 2012;10:Cd006525.
Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–13.
Spitzer RL, Kroenke K, Williams JB, Lowe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092–7.
Odum AL. Delay discounting: I'm a k, you’re a k. J Exp Anal Behav. 2011;96(3):427–39.
Zimet GD, Powell SS, Farley GK, Werkman S, Berkoff KA. Psychometric characteristics of the multidimensional scale of perceived social support. J Pers Assess. 1990;55(3-4):610–7.
Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007;42(4):1758–72.
Creswell J, Klassen A, Plano Clark V, Smith K. Best practices for mixed methods research in the health sciences. In: the Office of Behavioral and Social Sciences Research, ed. Washington, DC: National Institutes of Health; 2011.
Mehta SJ. Scaling and spreading innovation in health care delivery. Jt Comm J Qual Patient Saf. 2018;44(10):564–5.
Center for Health Care Innovation. 2019; https://healthcareinnovation.upenn.edu/. Accessed 28 Jan 2019.
Thaler RH, Sustein CR. Nudge: improving decisions about health, wealth, and happiness. Revised and expanded edition. New York: Penguin Books; 2009.
Whiteside U, Lungu A, Richards J, Simon GE, Clingan S, Siler J, et al. Designing messaging to engage patients in an online suicide prevention intervention: survey results from patients with current suicidal ideation. J Med Internet Res. 2014;16(2):e42. https://doi.org/10.2196/jmir.3173 Erratum in: J Med Internet Res. 2015;17(4):e69.
Asch DA, Terwiesch C, Mahoney KB, Rosin R. Insourcing health care innovation. N Engl J Med. 2014;370:1775.
Asch DA, Rosin R. Innovation as discipline, not fad. N Engl J Med. 2015;373(7):592–4.
Vlaev I, King D, Darzi A, Dolan P. Changing health behaviors using financial incentives: a review from behavioral economics. BMC Public Health. 2019;19(1):1059.
Schwebel FJ, Larimer ME. Using text message reminders in health care services: a narrative literature review. Internet Interv. 2018;13:82–104.
Zelen M. Randomized consent designs for clinical trials: an update. Stat Med. 1990;9(6):645–56.
Simon GE, Beck A, Rossom R, Richards J, Kirlin B, King D, et al. Population-based outreach versus care as usual to prevent suicide attempt: study protocol for a randomized controlled trial. Trials. 2016;17(1):452.
Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108.
Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res. 2011;45:626–9.
Heintzman J, Bailey SR, Hoopes MJ, Le T, Gold R, O’Malley JP, et al. Agreement of Medicaid claims and electronic health records for assessing preventive care quality among adults. J Am Med Inform Assoc. 2014;21(4):720–4.
The authors wish to thank their partners in the Penn Integrated Care program including Matthew Press, MD, MSc, and Eleanor Anderson, MD for their collaboration.
This study is funded by the National Institute of Mental Health (R21MH123851). This article was also supported by the Office of Academic Affiliations, Advanced Fellowship Program in Mental Illness Research and Treatment, Department of Veterans Affairs (Dr. Khazanov), and the VISN 4 Mental Illness Research Education and Clinical Center (Dr. Oslin). The funding source had no role in the design of the study and will not have any role in the conduct of the study, data analysis, interpretation of results, or the writing of study reports.
Ethics approval and consent to participate
This project, “Leveraging behavioral economics and implementation science to engage suicidal patients in mental health treatment”, was approved by the University of Pennsylvania IRB (protocol number 844601).
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Shari Jager-Hyman is the co-first author.
About this article
Cite this article
Khazanov, G.K., Jager-Hyman, S., Harrison, J. et al. Leveraging behavioral economics and implementation science to engage patients at risk for suicide in mental health treatment: a pilot study protocol. Pilot Feasibility Stud 8, 181 (2022). https://doi.org/10.1186/s40814-022-01131-y
- Treatment engagement
- Treatment initiation
- Behavioral economics
- Science of behavior change