Referrals to Mental Health Services in Ohio: An Exploration of Time to First Referral After Completion of Mental Health First Aid Gatekeeper Training

Background: The college population is particularly vulnerable to mental health challenges. In 2020 only 46.2% of people with a mental illness received mental health services. Mental Health First Aid (MHFA) is a training course designed to teach people how to connect individuals in need of professional services to the appropriate resources. Methods: Mental Health First Aid (MHFA) trainings were offered to students, faculty, and staff at Kent State University. Data from 343 individuals, who completed the MHFA gatekeeper training, were analyzed to explore the impact of time and participant characteristics on the likelihood of first referral to mental health services after completion of the MHFA. Participants completed a pretraining and posttraining paper questionnaire on the day of MHFA training and received a monthly online follow-up survey to assess self-reported referrals over time. Results: After completing MHFA, the average time until first referral was approximately 3 months. Several participant characteristics were significantly associated with referral to mental health services. African American and Black participants who completed the training were more likely to make a referral as compared to White participants. Extraversion was associated with increased likelihood of making a referral, while emotional stability was associated with a decreased likelihood of making a referral. Conclusion: Participants were 5.7% less likely to first report referring with each passing month following the MHFA training, suggesting that there may be cause for an MHFA or similar gatekeeper “booster” course to highlight the importance of making referrals.


INTRODUCTION
Mental illnesses affect thinking, mood, and/or behavior and include illnesses such as anxiety disorders, depression, and substance use disorders. 1 In 2020 an estimated 52.9 million adults in the United States, nearly 1 in 5, had a mental illness, and only 46.2% of those adults received mental health services in 2020. 2 The college population is particularly vulnerable to mental health challenges due to the stress of coursework and potential separa-friends (19%) and not having enough time (19%). 5 In addition, in most states, at age 18 years individuals become responsible for making their own decisions regarding mental health, and a significant proportion of college students with preexisting mental health conditions may stop taking their mental health medication when they arrive at college. 6 Therefore, it becomes vital for college campuses to break down barriers to access professional services (eg, lack of knowledge of mental health challenges, unsure of accessibility and stigma) 7 and utilize mental health gatekeeper trainings to teach individuals how to assist a person in need of mental health support.
Mental health gatekeeper trainings are programs that train individuals to recognize the signs of someone experiencing psychological distress, engage with that person, and help to connect them to services as appropriate. Many gatekeeper trainings focus on recognizing signs of suicide (eg, QPR, LivingWorks safeTALK), but others can be broader in terms of recognizing psychological distress (eg, Kognito). One mental health gatekeeper training is the standardized Mental Health First Aid (MHFA) training, which was created to alleviate some of the potential barriers to seeking mental health treatment. 8 The 8-hour MHFA training provides education on signs and symptoms of mental health challenges and attempts to reduce mental illness stigma. An action plan, which uses the acronym ALGEE, is taught to participants and can be used to assist people experiencing mental health challenges. 8 Researchers have studied various aspects of the effectiveness of MHFA. Increases in mental health knowledge and confidence in providing help to others, in addition to a reduction in stigmatizing attitudes, have been found after completion of the MHFA training. [9][10][11] However, research exploring referrals to mental health services after completing the training is limited and mixed. [12][13][14] For example, Eisenberg and colleagues found that after university residential advisors completed MHFA, the number of students they referred to mental health services did not change. 14 Furthermore, Lipson and colleagues conducted a randomized control trial with 32 colleges and found MHFA trainings for residential advisors had no impact on students' mental health help seeking. 13 However, there was an increase in the likelihood of residential advisors seeking mental health services for themselves. Trainee's self-perceived knowledge, self-perceived ability to identify students in distress, and confidence to help also increased. Additional research is needed to further explore referrals to mental health services after MHFA training.
Mental Health First Aid training provides tools to make referrals to mental health services; however, there is no existing empirical research analyzing the amount of time from MHFA training completion until the first referral is made. Since providing mental health information and resources is a main component of the MHFA action plan, understanding the amount of time it takes to begin making referrals is warranted to assess the effectiveness of the training. Furthermore, studies have not explored MHFA partic-ipant factors (eg, demographics). These factors need to be studied to determine their potential effect on referrals as well as their overall influence on the effectiveness of the training. Since this is an exploratory study, a range of variables were assessed that might potentially be linked with making referrals to mental health service (ie, race, sex, faculty/staff or student status, personal and/ or family experience with a mental health or substance abuse problem, average number of students interacted with face-to-face per day, average number of students interacted with electronically per day, and contact with a student on campus in need of help related to mental health or substance abuse prior to training). These variables were selected from prior research and their potential relationship with making referrals. 13,[15][16][17] The current study aims to: (1) determine the impact of time on the likelihood of first referral to mental health services after completion of the MHFA training and (2) assess the role of MHFA participant characteristics (eg, sex, MHFA knowledge) in making referrals.

Participants and Procedure
Beginning in the spring of 2016 and ending in the spring of 2018, free MHFA trainings were offered to all students, faculty, and staff at Kent State University. At the beginning of each training, all individuals taking the training were invited, but not required, to participate in a research study. Prior to data collection, the study was approved by the university's institutional review board. The study is a nonexperimental longitudinal design. This means a comparison group was not utilized, and data were collected from participants repeatedly over time. Those consenting to participate in the study were asked to complete the paper-and-pencil pretraining questionnaire immediately prior to the start of the MHFA training. The pretraining questionnaire included questions on demographics (eg, age, race, sex), personality characteristics, personal and family experience with mental health or substance abuse illnesses, MHFA knowledge, personal stigma, confidence to refer to mental health services, previous contact on campus with someone in need of mental health or substance abuse help, average number of students interacted with face-to-face per day, and average number of students interacted with electronically per day.
The majority of the MHFA trainings were led by 2 trainers who were university staff or university faculty or community trainers. Immediately following completion of the MHFA training, individuals were asked to complete a paper-and-pencil posttraining questionnaire that included identical measures of personal stigma, MHFA knowledge, and confidence to refer to mental health services from the pretraining questionnaire.
Information regarding whether participants provided referral information to anyone in the previous 30 days was collected using a monthly online follow-up survey sent through email. Participants received the follow-up survey every month following the date that they completed MHFA training unless they requested to be reojph.org Ohio Public Health Association 43 moved from the study or were no longer enrolled/employed at the university. Participants could opt out of the follow-up at any time. Surveys were sent every month regardless of whether the person answered previous monthly follow-up surveys. Therefore, an individual may have responded to follow-up surveys every month or responded sporadically. The follow-up survey was sent electronically using Qualtrics ® .
Of the 730 individuals who completed the MHFA training, 633 (86.7%) individuals consented to participate in the current study. Among the 633 participants who consented to participate in the study, 182 (28.8%) did not respond to any follow-up surveys and were removed from the sample. A total of 108 (17.1%) participants had missing covariate data, which excluded their data from the analysis. The final sample consisted of 343 participants (54% of those who consented to the study) who responded to at least one monthly follow-up survey and replied to each covariate of interest.

Measures
Pretraining Questionnaire Only Participant Characteristics. Several items were included to assess participant demographic information and assess experience with mental health or substance use. Items also assessed the type and frequency of contact that participants had with students. Question topics included race, sex, faculty/staff or student status, personal and/or family experience with mental health or substance abuse, average number of students interacted with face-to-face per day, average number of students interacted with electronically per day, and contact with a student on campus in need of help related to mental health or substance abuse prior to training (Table 1).
Prosocial Personality Battery (PSB). The 30-item version of the PSB was administered to participants for the present study. 18 The PSB consists of 7 individual scales: social responsibility, empathetic concern, perspective taking, personal distress, mutual moral reasoning, other oriented reasoning, and self-reported altruism. Previously, these 7 individual scales have been determined to create 2 separate factors: helpfulness and empathy. 18 Empathy combines the sum of social responsibility, empathic concern, perspective taking, mutual moral reasoning, and other oriented reasoning. Helpfulness combines reverse scoring of the personal distress and standard scoring of self-reported altruism scales. Confirmatory factor analysis was used to confirm the fit of these factors for the current study. Participants indicated how frequently they carried out each item (eg, "carrying a stranger's belongings") in the past ("Never" = 1 to "Very Often" = 5).
Ten-Item Personality Inventory (TIPI). The TIPI is a short, 10-item measure of the Big Five personality dimensions, including extraversion, agreeableness, conscientiousness, emotional stability, and openness to experiences. Participants indicate the extent to which personality traits apply to them ("Disagree strongly" = 1 to "Agree strongly" = 7). 19 Convergence with the Big Five Inventory 20 has been shown (mean r = .77) across the 5 dimensions, and testretest reliability for the TIPI is mean r = .72. 19 Cronbach α scores were low for some several constructs (extraversion, α = 0.70; agreeableness, α = 0.27; conscientiousness, α = 0.55; emotional stability, α = 0.63; openness to experiences, α = 0.35) because the TIPI was created to maximize content validity and only contained a few items per construct. 19 Pretraining and Posttraining Questionnaire Depression Stigma Scale. The Depression Stigma Scale, 21 an 18item measure comprised of two 9-item subscales, was used to assess stigma. One subscale assesses perceived stigma and the other assesses personal stigma. Cronbach α for the current study was 0.78 for the personal stigma items, 0.76 for the perceived stigma items, and 0.82 for the total scale. 21 Seven items from the personal stigma subscale were used for the current study, with the items showing good internal consistency (α = 0.84). Two question items were not included in the current study, because they were not applicable to the study participants regarding the individuals they interact with directly. Personal stigma questions ask participants how much they agreed ("Strongly Agree" = 1 to "Strongly Disagree" = 5) to 7 statements about "John," a fictional person described in a vignette as having depression. These statements included that John could snap out of it, showed signs of personal weakness, did not have a real medical illness, was dangerous, that it is best to avoid people like John, he is unpredictable, and if I felt like John, I would not tell anyone. Scores from these 7 items were averaged to provide a measure of personal stigma. Mean scores could range from 1 to 5, with higher mean scores indicating greater personal stigma.
Mental Health First Aid Knowledge. Knowledge as a result of completing the MHFA training was assessed by asking participants to define each letter in the mnemonic ALGEE, the 5-step action plan taught during MHFA training. 8 The letters in ALGEE stand for Assess for risk of suicide or harm, Listen nonjudgmentally, Give reassurance and information, Encourage appropriate professional help, and Encourage self-help and other support strategies. 22 An MHFA knowledge score was computed by summing the total number of action steps correctly defined. Internal consistency of the knowledge items was α = 0.67 for the current study. The knowledge score could range from 0 to 5, with higher scores indicating greater knowledge.
Confidence in Making Referrals. To assess confidence in making referrals, 5 novel question items were developed by the authors for the purposes of the study. Participants were asked to rate their level of confidence in making each referral type: give an informational card/pamphlet, provide a link to a website, provide information about 24-hour hotlines, provide assistance in contacting an on-campus behavioral health provider, and provide assistance in contacting an off-campus provider. Participants could rate each referral from 1 (not at all confident) to 5 (extremely confident). A confidence in making referrals score was computed by averaging the ratings of the 5 items. Internal consistency for the scale was high (α = 0.91).

Monthly Follow-Up Survey
Referrals Made. To assess number of referrals made each month, participants were asked, "Did you provide referral information about services for mental health or substance abuse problems to [university] students in the past 74 days (for example, information about the on-campus health center)?" Respondents selected 'yes' or 'no.' Respondents selecting 'yes' were then asked to report the number of referrals that they made in the past 30 days.

Analytic Plan
A logistic regression model was conducted to assess the likelihood of referral to mental health services after completing MHFA training ( Table 2). Because of the exploratory nature of the study, a stepwise method of variable inclusion was utilized, thus reducing the likelihood of overparameterizing the model. Given that the following analyses focus on outcomes after the training, posttraining scores for personal stigma, MHFA knowledge, and confidence in our models were used. Referral outcomes were derived from responses to the follow-up surveys in the first 12 months after participants completed training. Possible selection variables were race, sex, faculty/staff or student status, empathy, helpfulness, extraversion, agreeableness, conscientiousness, emotional stabil- ity, openness to experiences, personal and/or family experience with a mental health or substance abuse problem, personal stigma, average number of students interacted with face-to-face per day, average number of students interacted with electronically per day, MHFA knowledge, confidence in making referrals, contact with a student on campus in need of help related to mental health or substance abuse prior to training, and number of months since MHFA training occurred. The entry and exit criteria for each variable was a P value of 0.15. 23 All analyses were conducted in SAS 9.3. 24

RESULTS
After stepwise iteration, variables with a P value of 0.15 or less were included in the model: race, sex, faculty/staff or student status, average number of students interacted with face-to-face per day, contact with a student on campus in need of help related to mental health or substance abuse prior to training, extraversion, emotional stability, conscientiousness, helpfulness, MHFA knowledge, and number of months since MHFA training occurred.
Participant characteristics are presented in Table 1. Most participants are racially White participants (79.9%), and most are females (90.1%). The average age of participants is 32.5 years, and participants reported interacting with an average of 15.9 people in-person and 11.0 people electronically each day. Importantly, among participants, 245 (71.4%) referred someone to services, with a mean time to referral of 3.1 months (SD = 2.7).
The significant covariates in the logistic regression model were race, faculty/staff or student status, emotional stability, extraversion, conscientiousness, helpfulness, average number of students interacted with face-to-face per day, contact with a student on campus in need of help related to mental health or substance abuse prior to training, and number of months since MHFA training occurred (see Table 2). Covariates in the model that were not significant were sex and MHFA knowledge.
Many participant characteristics increased the odds of referral (Table 2). African American and Black participants were approximately 4 times as likely to refer compared to White participants. A person who had contact with a student on campus in need of help related to mental health or substance abuse prior to training was 3 times as likely to refer as compared to someone who had not seen someone in need of help before training. For every additional student someone interacted with face-to-face, the likelihood of referral increased by 1%. For every unit increase in extraversion and helpfulness, participants were 24.2% and 21.1% more likely to make a referral, respectively. Each unit increase in emotional stability decreased the likelihood of referring by 15.4%, and staff/ faculty were 33.8% less likely to make a referral than students. Each unit increase in conscientiousness decreased the likelihood of referring by 21.6%, and each additional month after completion of the MHFA training, decreased the likelihood of first-time referral by 5.7%.

DISCUSSION
The purpose of the current study was to explore the impact of time and the role of participant characteristics on the likelihood of making a referral to mental health services after completing an MHFA gatekeeper training. The analytical model demonstrates the average time until first referral was approximately 3 months after completing MHFA, and with each passing month the likelihood of a first-time referral decreases. The decrease in referrals may signal a need for a booster component to the MHFA course to reinforce information taught in the initial MHFA training. It is well-established that to maintain the performance of a specific task, such as referring someone to services, it is better learned over a long period rather than a short amount of time. However, in opposition to this is the established conclusion that an amassed amount of knowledge or practice leads to much better initial performance. 11 Self-efficacy, one construct of the Social Cognitive Theory, refers to a person's confidence about their abilities to successfully execute a task. 25 Additional opportunities to practice making referrals could increase the chances of a person making referrals. The juxtaposition of these 2 concepts may be balanced by introducing a booster component to the current MHFA gatekeeper training.
The current study suggests that several participant characteristics predict whether an individual who completed an MHFA gatekeeper training will refer someone in need to mental health resources. African American and Black participants were more likely to make referrals compared to White participants. While we were unable to find other research that directly supports this finding, a large national study examining confidence related to mental health literacy found that African American and Black participants who completed an MHFA training reported higher mental health literacy as compared to White participants who had completed the training. 26 Further, according to the National Institute on Minority Health and Health Disparities, African American and Black participants are more likely to experience serious psychological distress than White participants. 27 Taken together, greater experience with mental health issues and greater mental health literacy may increase the likelihood of making a referral. To explore this further, a subset of the original analysis composed of only African American and Black participants was conducted to compare results to the overall model. Mental Health First Aid knowledge was not significantly related to referrals among African American and Black participants. Further, the covariables of agreeableness and personal stigma were not significant for the overall model. Therefore, the association of these variables with referral may be a reason for the increased association of African American and Black participants making more referrals in comparison to White participants.
Individuals who reported knowing someone in need of mental health services prior to the gatekeeper training were more likely to make a referral following the training. It is quite possible that these individuals signed up for the training specifically to learn ojph.org Ohio Public Health Association 46 how to assist those in need who were already a part of their social interactions. An increase in referrals among students, compared to staff/faculty, and among those who see more students face-to-face suggests that students and those in direct contact with larger numbers of students may be an important target population for gatekeeper trainings such as MHFA. Additionally, a future direction is to analyze these results separately for faculty/staff and students.
It is not surprising that extraversion and helpfulness were participant characteristics found to be associated with an increased likelihood of referring an individual to mental health services. Extraverted individuals (as compared to introverted individuals) would be more inclined to engage with another individual, especially an individual they are not very familiar with (eg, an acquaintance). While those who indicate higher levels of helpfulness would be more likely to assist a person in need of mental health help and take action in the event of an emergency or crisis. Both emotional stability and conscientiousness have been found to be positively associated with better mental health, [28][29][30] and research has demonstrated both characteristics to be positively related to increased internal locus of control. 31 Consequently, the current study's findings that conscientiousness and emotional stability are associated with less likelihood of making a referral runs counter to our expectations. That said, perhaps individuals who are conscientious and/or emotionally stable, may be primarily focused internally and thus may fail to recognize others around them who may be in psychological distress and may be in need of mental health services. The addition of empathy training may be beneficial and lead to an increase in referrals. Further study is needed to explore this finding.
The present study adds to the existing research on the MHFA gatekeeper training by investigating referrals to mental health services based on time and participant characteristics and experiences. Specifically, the current study contributes by examining how personal characteristics and experiences of gatekeepers are associated with the likelihood to refer an individual for mental health services. A further contribution is this is one of the first studies to examine the permanence of making referrals over time.
Although our findings are very preliminary, knowing if and when referrals decrease over time and what factors influence the decrease is important to understand as it relates to any mental health gatekeeper training and its impact to help individuals in need.
One limitation of this study is that the amount of exposure to people in need of mental health assistance and referral could not be controlled. For instance, some participants may have much more contact with individuals, and thus more opportunity to make referrals. Additionally, the analysis does not contain a control group who did not receive the MHFA training. Further, participants who completed the MHFA training but chose not to participate in the study were not evaluated for self-selection bias. In addition, num-ber of referrals made required participants to recall and estimate information retrospectively, which potentially introduces error. Finally, social desirability may have led to inaccurate reporting from participants.

PUBLIC HEALTH IMPLICATIONS
Even with limitations, the study provides practical implementation applications. Monthly decreases in referrals suggest a need for a potential booster after the initial MHFA training. Since MHFA knowledge was not significant, a booster class focusing on the curriculum of the MHFA would not be necessary. Instead, a booster highlighting the importance of making referrals could be delivered, potentially through email. However, future research is needed to explore potential booster options (eg, length of booster session, delivery format), and the feasibility of using email as the delivery method, which would reduce costs compared to a class session. If resources limit the number of individuals who can be trained as MHFA gatekeepers, results suggest potentially targeting the MHFA training to those with increased face-to-face interactions, which was correlated with increased referrals. Future research and programming can explore ways to increase referral rates based on individual-level characteristics. For example, research tools and educational components can be included in the training or provided after the training to increase the likelihood of those with introverted personalities referring people in need to mental health services. Further research is needed to continue to explore the effectiveness of MHFA on referring individuals to mental health services.