Strategies to Enhance Qualitative and Mixed Methods Research Instruction to Prepare MPH Students for Public Health Practice

to ABSTRACT Background: The 2016 Council on Education for Public Health (CEPH) accreditation guidelines for Master of Public Health (MPH) programs describe skills, including qualitative and quantitative research skills while removing the requirement to structure programs around 5 designated core areas. These revised guidelines provide an opportunity to integrate qualitative and mixed methods instruction in multiple courses. The purpose of this paper is to describe process and outcomes of a project aimed at integrating qualitative and mixed methods content into core courses within the Kent State University MPH program. Methods: Content development work was conducted from May through October 2019. The work consisted of content analysis of current core course content, research texts, practicum presentations, and feedback from alumni working in public health practice. Five key qualitative processes that support CEPH competencies and reflect current public health practice in Ohio were articulated to form a framework for new course content. Results: New content was developed for each of the 6 current core MPH courses to address CEPH competencies, incorporate the identified 5 key processes, and to emphasize cross-methodological comparison and the complementary nature of qualitative and quantitative approaches to research questions or practice issues. Initial student responses to content were positive; further evaluation efforts are planned. Conclusion: New content provides MPH students with skills practice associated with qualitative and mixed methods approaches to research and applied public health. To address complex public health challenges, current and future public health professionals will benefit from being able to flexibly move across methodological boundaries.


INTRODUCTION
Public health has a tradition of affiliation with epidemiologic and quantitatively analyzed research designs, often used with the aim of identifying causal relationships 1 to address a variety of concerns ranging from illness and injury to neighborhood patterns of violence. 2 Evidence-based practice is a related concept in that preferred evidence tends to be associated with conventionally quantitative designs such as randomized controlled trials, 3 while case studies and other uncontrolled designs that typify qualitative inquiry are depicted as less desirable, and by implication, less useful.
Similarly, public health graduate education in some institutions may emphasize quantitative research instruction. The 2011 Council on Education in Public Health (CEPH) guidelines for Master of PUBLIC HEALTH EDUCATION research instruction balanced quantitative coursework was determined at program level, and most likely was associated with faculty skills and interests.
In contrast, the 2016 CEPH update, which reflects the most profound adjustment to guidelines for public health curricula since the 1940s, 5 includes increased emphasis on qualitative and mixed methods research in public health education. Mixed methods typically refer to a combination of qualitative and quantitative data streams to address a common issue. 6 The evolution of CEPH guidelines is consistent with interest in and appreciation of qualitative inquiry which has grown steadily in the social sciences and other fields from the 1980s into the first decades of the 2000s. 7,8 In the 2016 guidelines for MPH degrees, CEPH eliminated the 5 core areas and replaced these with a list of 22 described competencies. Two of 4 competencies associated with "Evidence-based approaches to public health" 9 include explicit mention of qualitative research:  Select quantitative and qualitative data collection methods appropriate for a given public health context  Analyze quantitative and qualitative data using biostatistics, informatics, computer-based programming and software, as appropriate. 9(p17) An additional competency specifies students must "interpret results" 9(p17) which qualitative research methods yield.
The stated goals of the updated CEPH guidelines were to better prepare students for practice and to prepare them to empower communities to achieve improvements in public health. 5 Specific aims of the 2016 update included providing students with an integrated education, granting programs greater flexibility in curricula, and substituting focus on enhancing practice skills in place of presenting topical knowledge. 5 Although it is likely the CEPH 2016 guidelines resulted in the need for modifications to many accredited MPH degree programs, a literature search suggests few scholars have disseminated reports describing their efforts. Published research about CEPH accredited programs tends to describe development and success of novel concentrations or emphases, 10,11 or to compare academic program accreditation with other credentialing, 12,13 or to describe the role of field placement requirements. 14,15 Authors of one paper provided recommendations for development of competency-based public health education although the work pre-dates the 2016 CEPH guidelines. 16 These recommendations are still relevant and include advice to develop courses to meet competencies, rather than to match competencies to pre-existing courses. 16 At Kent State University, the MPH degree has traditionally emphasized statistical and mathematical reasoning within courses and their associated competencies, consistent with the 2011 CEPH guidelines. 4 The original 5 MPH concentration areas (presently reduced to 4 reflecting suspension of the environmental health concentration) tracked with CEPH's 5 prescribed core areas. The quantitative focus also reflected the expertise of faculty recruited for development of this now 10-year-old college. In summary, neither core nor concentration requirements of the Kent State University MPH include a dedicated qualitative or mixed methods course, as neither of these was an essential component of instruction in the 5 core knowledge areas.
In response to the 2016 guidelines, content was integrated into one of the core courses, Community Health Needs Assessment, to ensure Kent State University MPH students were provided with adequate resources and ample skills practice opportunities to meet the "and qualitative" portions of the 2016 competencies, described above. While this content ensured the MPH was compliant with CEPH guidelines, research methods instruction across the concentrations continued to be relatively segregated and emphasized quantitative methods. Students were not provided with extensive opportunities within their courses for comparison or blending of qualitative and quantitative methodologies.
There are several reasons to encourage an inclusive approach to qualitative and quantitative research methods education. First, a joint presentation facilitates comparison of the strengths, weaknesses, and applicability of each approach, so that students are more likely to select the most appropriate and not just most familiar methods when they enter public health professional practice. Another important reason is that quantitative and qualitative knowledge in combination provide more complete information. Quantitative data might identify the scope of a problem, whereas qualitative data might contribute to understanding the complex reasons underlying a problem encountered in public health practice. 1 In clinical trials and other experimental designs, qualitative inquiry can enhance and clarify quantitative results by facilitating in-depth exploration of dropout, noncompliance, and other inconsistencies in data or results. 17,18 Qualitative data is also useful for practitioners when implementing previously developed programs to a new context, to ensure processes are refined as appropriate. 19 The desire to make better use of the flexibility provided by CEPH 2016 guidelines by developing inclusive research methods instruction motivated an MPH curriculum review and development of recommendations for content refinement across the 6 courses required for all concentrations, that at the time of the review emphasized quantitative methods and examples. The purpose of this report is to describe results of an internally funded project undertaken to identify ways to integrate qualitative and mixed methods educational content into the 6 MPH core courses at Kent State University.

Setting and Program Description
At the time this project was undertaken, there were approximately 225 students enrolled in the Kent State University MPH program.

PUBLIC HEALTH EDUCATION
This included students in 4 concentration areas: Biostatistics; Epidemiology; Health Policy and Management; Social and Behavioral Sciences, and included full-time and part-time, fully online, and traditional students. Students do not take all courses in the same sequence, and there are few courses that are designated as a specific prerequisite for another course. These circumstances mandated that content in a given course must be complete enough and focused enough to offer a meaningful learning experience without requiring courses be taken in a specific sequence. There are presently 6 core courses required for all MPH students.

Content Development
The project activities took place primarily between mid-May and the end of October 2019. Content development consisted of integration of information derived from content analysis of several sources. These included existing core course syllabi, multiple qualitative and mixed methods textbooks, 20-23 reports from culminating experience/practicum projects, and information informally solicited from 6 former students and a similar number of others working in public health practice. The overall goal of the content analysis was to develop a specific list of subskills that contributed to the CEPH competencies and could be used to inform specific course content.
Additional focused aims were associated with content analysis of each information source. Information taken from syllabi included course catalog description, specific learning outcomes, and current course assessments that focused on research processes such as design, data processing, and analysis. Other aims of this segment of content analysis were to identify current quantitatively-focused course assessments that might be modified to include an associated qualitative component, and to identify course-specific outcomes that suggested qualitative inquiry, such as an assignment to consider context-specific adaptations to programs. Qualitative and mixed methods research texts were consulted for definitions, examples, and directions for data processing and analysis. Other aims of this segment of content analysis included identification of high-quality resources for use as reading and instructional content in courses. Culminating experience or practicum reports that reflected qualitative or mixed methods designs were consulted to determine methods used to gather and analyze data and to identify typical questions and methods used for field placement research on behalf of agencies. After initial integration of information from content analysis, alumni, current students, adjuncts, and others who currently or formerly worked in public health practice were informally surveyed regarding common skills used in jobs, to identify skills beyond those used for field placement in a variety of public health settings.
All content analysis was conducted primarily by the author. Doctoral students additionally assisted with review of syllabi and practicum reports and provided feedback at various stages that informed development of the final list. Content of interest was extracted from course syllabi and practicum reports on an individual basis and then combined in order to develop a master list of subskills. The initial list was discussed informally with alumni and others who had public health practice experience, and this information facilitated further refinement of the list. Lastly the developed list was compared with text content to check availability of instructional resources and to identify what additional information needed to be located or created. This initial content assembly and review took approximately 6 weeks and resulted in development of the following list of 5 key processes: (1) Preparing data and conducting qualitative coding for data analysis, (2) Quantitizing qualitative data (compiling unstructured/ qualitative data into categorical or count data), (3) Developing questions or prompts to elicit useful information in qualitative interviews, (4) Describing one or more ways to use mixed methods to address a single purpose, and (5) Using qualitative evaluation methods to identify unintended consequences of public health programs.
Following identification of these processes, current syllabi, assessments, and course readings were reviewed again in greater detail to ensure coherence between content to be developed and the tone of each course. Draft versions of new content, including assessments and readings, were reviewed for fit, consistency, and comprehensiveness of information by a doctoral student who possessed extensive experience in instruction, content development, and qualitative inquiry. Recommendations from this content review, which included ways to improve clarity of instructions and identification of supplemental resources, were incorporated into the recommended version of developed content.
Content development for each of 6 core courses took roughly 2 weeks per course, with another several days to one week for the review and revision process. Preliminary recommendations were presented twice: once for public health faculty and once at a university teaching and learning event. The project funding provided 10 weeks of full-time (40 hours/week) support for the author. The remainder of the work was completed using a portion of assigned teaching hours. Graduate student participation was provided through a combination of voluntary and college-supported time.
Notably these processes, with the possible exception of the first item, are not exact duplicates of CEPH competencies. However, process 4 is consistent enough with the "select methods" competency that both aims could be met in a single course. Process 3 goes beyond CEPH competencies by providing instruction in gathering data, which is not included in CEPH MPH competencies with respect to qualitative or quantitative research but is an important skill in public health practice. the 5 processes while emphasizing cross-methodological comparison, and ways to take advantage of the complementary nature of qualitative and quantitative approaches to research questions or practice issues. All content reflects an addition to existing course content. Additional content detail is available by contacting the author.

RESULTS
The constructivist nature of qualitative research, meaning that researchers and practitioners respect the role of each individual's perceptions and values in shaping his or her view of the world 21 is an important attribute of qualitative research and therefore discussed within developed course content. That said, the focus of much instructional content was more closely aligned with the pragmatic paradigm, 7,24 that specifies that researchers prioritize use of methods and designs most likely to address the purpose of a given research project, over other considerations including philosophical orientation. Because integration of qualitative and quantitative results is an advanced research skill, emphasis was on identifying uses of mixed methods rather than on instruction in fully integrated analyses.
To date, students have been given opportunities to complete online and in person modules that provide opportunities to learn about and practice basic qualitative data analysis, all within a sin-

PUBLIC HEALTH EDUCATION
gle course. New content was originally intended to be incorporated into all existing core courses in association with the scheduled, staggered course revision schedule. Traditionally, this has followed a 3-year timeline between major course revisions, although challenges associated with alternative content delivery for Fall 2020 will most likely impact the major revision schedule. Under normal circumstances, the span to incorporate content would run from Fall 2020 to Spring 2023.
Initial student assessments are primarily positive, with some students, including working professionals, enthusiastically recognizing clear applicability of qualitative or mixed methods approaches in their current public health practice. Instructor assessment of student work suggests the need for further refinement of content to emphasize the value of nuance in transcripts and other practice data. For example, some students have demonstrated a tendency to assign broad concrete codes (eg, "environmental issues") rather than being attuned to context-specific aspects of concerns presented in the data (eg, "poor air quality exacerbates asthma").
Of 52 students surveyed who received in person qualitative content, just over half (n = 26 or 54%) indicated the information was "very useful," with another 40% (n = 21) indicating the information was "somewhat useful." Open response student comments were not extensive but included positive statements (eg, "I enjoyed learning about this."). Additionally, 2 students reached out to course instructors shortly after completing qualitative assignments to begin to plan qualitative practicum projects. Similar data were not initially gathered from the slightly larger number of online students as cohort evaluation is planned at a later date, as described below. Additional planned evaluation processes include the following:  Survey of students on sufficiency of research training on the initial MPH exit survey. Survey items track with the CEPH competencies and include both a quantitative/rating and qualitative/commenting component. This process is intended to begin when the first cohort who completed qualitative content graduates in Spring Semester 2021.
 Survey of instructors regarding new content. This will be given at the end of the academic year when a given course has been revised, so will reflect experiences from at least one and up to 4 sections of a given course. Items assess quality and clarity of content, ease of assessing, and student feedback.
 Tracking proportion of practicum projects with qualitative data streams, alone or in combination with quantitative data streams.

DISCUSSION
This public health education paper described content modifications developed for the Kent State University MPH program. These modifications were aimed to exceed 2016 accreditation guidelines related to qualitative and quantitative research processes. Aspects of this content will allow students to begin to explore integration of qualitative and quantitative methods and results and to strive toward mixed methods approaches to inquiry in their professional practice. Importantly, the focus of this work has not been to deemphasize quantitative methods in favor of qualitative inquiry, but rather to train student researchers to understand and appreciate the role and value added by both flavors of inquiry. It is important to acknowledge the role of the Kent State University Teaching Council and College of Public Health administration for their support for this work. Institutional and academic unit support not only makes the work possible but also demonstrates to students and faculty the institution's commitment to ongoing enhancement of qualitative research instruction.
Scholar/educators have recommended developing courses to fit competencies, rather than fitting competencies to courses. 16 With this project, an alternative to either of these approaches was used. In essence, 3 competencies were reinterpreted in various ways across 6 courses with different emphases. This is meant to create a learning experience consistent with the theory that expertise is gained not through theoretical knowledge but through participating in and processing multiple context-specific experiences. 25 It is hoped that this new content resonates more deeply with students by presenting related ideas in different ways rather than by repeatedly reinforcing the same idea. This also points to a strength and efficiency of this process which is revision and refinement of existing content rather than creation of new courses. An associated advantage is that refinements developed through this process were not substantial enough to require curriculum action, instead these are easily processed as information items. Additionally, use of existing assessments as a starting place for revisions means that new content is more likely to seamlessly integrate with existing content.
Incorporating these or similar content modifications presents opportunities to contribute to student knowledge but is associated with multiple challenges. These include that fewer faculty in some public health degree programs are well versed in both qualitative and mixed methods research when compared to those with proficiency in epidemiology or biostatistics; many programs and associated faculty reflect the priorities from the 2011 and prior CEPH guidelines. 4 This is a key challenge for this specific program as the majority of faculty have little to no qualitative research experience or training. For fully online courses, this is partially navigable by blending content from multiple faculty and developing comprehensive instructor guides to assist faculty in grading assessments. Realistically, in any accredited public health program, faculty skills and unit-specific workload expectations may facilitate or limit integration of additional qualitative content into existing courses.
Limitations of the described processes include that informal feedback provided by alumni and practitioners was not solicited systematically, and the completion of this project during a compressed time frame prevented contact with some individuals who might have provided different views or recommendations. The

PUBLIC HEALTH EDUCATION
staggered course revision schedule extends the timing of content evaluation, and there is a chance that different needs may emerge between the time of content development and scheduled revisions for a given course, necessitating revision of the revision(s), prior to integration.
There is an additional challenge associated with assessing the actual impact on public health practice. Professionals working in public health and other fields have engaged in qualitative and mixed methods research when necessary to do so, sometimes with extensive and sometimes with little to no academic training. It is difficult to support the assertion that MPH graduates who were exposed to additional content are more capable or confident researchers. The best approach to assess outcomes is most likely a mixed methods evaluation to assess research productivity, whether via publications, evaluations, or other outputs associated with practice in combination with self-assessment of research skills. This evaluation would be enhanced by including feedback from alumni, collaborators, and other colleagues and might potentially be incorporated into current information gathering required for accreditation.
One area that warrants further exploration is development or refinement of directed elective course offerings to facilitate students' ability to dive more deeply into integration of qualitative and quantitative methods to address their particular research or practice interests. To this end, 2 doctoral level public health qualitative research courses were approved through the Kent State University curriculum process for modified delivery as 7-week online electives in the MPH program. These courses will be offered for the first time for MPH students during the 2020-2021 academic year. Further 1-hour dedicated content modules, to be offered online as 5-week courses, are approved or in the process of approval, and are planned to be made available for doctoral or MPH students. While some students will be motivated to explore the philosophy and myriad alternatives associated with qualitative inquiry, for other students acquisition of basic skills in interviewing, transcribing, and coding data via single credit-hour courses might meet their current needs.

PUBLIC HEALTH EDUCATION IMPLICATIONS
Prior to the 2016 guidelines, MPH programs had more flexibility to develop competencies although curriculum was limited by the need to structure around 5 designated core areas. The CEPH 2016 accreditation guidelines provide opportunities to meet competencies related to qualitative and quantitative research across multiple courses. To address increasingly complex issues, future public health professionals will benefit from improving appreciation and understanding of qualitative, quantitative, and mixed methods approaches.
The current coronavirus disease 2019 (COVID-19) crisis illustrates a good example of a health concern that is communicated largely in quantitative means (numbers, rates, and trends) while virus spread is related to individual and behavioral processes, and investigated through contact tracing, that relies largely on qualitative interviews. Optimally prepared public health professionals understand the strengths and weaknesses of each approach and can visualize how qualitative and quantitative methods can be used in complementary ways.