Research Design Flaws Identified in Study Suggesting That Chief Diversity Officers “Hurt” Diversity

Last week, numerous media outlets highlighted a working paper submitted to the National Bureau of Economic Research contending that Chief Diversity Officers have no influence on faculty diversification and that their presence inversely influences the diversity of tenured faculty. The articles were widely shared in social media and received a visceral response from diversity experts who questioned the motivation for the study, analytic approach, its theoretical underpinnings, and interpretation of the results.

As leaders who have extensive expertise on the roles of Chief Diversity Officers (CDOs) and issues directly in line with the work of CDOs, our attention was immediately drawn to methodological and conceptual concerns that are worthy of note. But first, here is an overview of the study.

Dr. Luke Wood

Four researchers from Baylor University used data from the Integrated Postsecondary Education Data System (IPEDS) and harvested webpage information regarding CDO hire dates to examine the effectiveness of having a CDO at the institution. Their primary variable for examining the influence of CDOs on faculty success was a dichotomous measure of whether or not the institution had a CDO in a given year and its correlation with diversity hiring in subsequent years. They conducted a litany of analyses examining the effect of having a CDO on student, faculty, and administrative diversity by rank, while controlling for the proportion of existing students, faculty, and administrators who are members of an underrepresented minority, as well as the growth in undergraduate applications. Largely, the study found that there was no effect of CDO on diversity, except with the finding that CDOs have an inverse relationship on the hiring of already tenured faculty. As critical researchers, we continue to advocate, encourage, and pursue research in areas where limited or no work has been undertaken. However, in this case, we immediately recognized flaws in the study design that undoubtedly influenced the integrity of the results.

More to the point, though, the framing of the issue in the first place as one of a pipeline issue that CDOs might influence by their presence narrowly captures the issues that CDOs are tasked with in the first place in relation to diverse faculty hiring. We suggest that the findings are the byproduct of a series of underspecified models. These issues are representative of the common pitfalls that occur when research topics on diversity are engaged with limited insight as to the goals and nature of the work that CDOs do. Our different perspective on the methodological approach could be more disciplinary specific and preferential. However, there are four critical flaws based on the analysis that require attention for those placing stock in the results:

1).  Missing Contextual Factors

Dr. Adriel A. Hilton

The analysis did not control for any contextual factors that have a direct bearing on the influence of CDOs. These factors include measures of the campus racial climate, workplace satisfaction, relational dynamics between faculty and administration, and racial diversity within the surrounding community that is changing in drastically different ways across states and regions. Thus, the analytic approach employs an inherent assumption that these factors have no influence on CDO success and that their influence is the same across time and across colleges. There are a number of data sources (e.g., National Survey of Student Engagement, Beginning Postsecondary Student Longitudinal Study, the Higher Education Research Institute’s College Senior Survey and Faculty Survey, extant surveys of CDOs, U.S. Census Bureau’s American Community Survey) that could have been utilized to account for these factors and this did not occur.

Other missing contextual factors stem from the socio-historical nature of institutions and their various missions in relation to the community. For instance, the authors note that “perhaps surprisingly, the proportions for underrepresented groups has been consistently higher in universities without a CDO than with a CDO…” In reality, though, this is not surprising at all if one looks at the array of socio-historical settings of colleges included in the analysis. For instance, the roughly 400 institutions within the Carnegie classification M1 (master’s colleges and universities – larger programs) tend to be regionally or locally-serving institutions that are relatively more engaged with local communities and more representative of the populations they serve than their research-intensive counterparts.

2). No Account For the Varying Operational Structures of CDOs

Dr. Raul A. Leon

The analysis examines CDOs using a constrained binary (yes or no) measurement. Given this, there is no consideration for the varying operational structures that CDOs operate under. Research focused on CDO configurations demonstrates that CDOs are not all created equal, and their configurations can have a pervasive impact not only on the breadth of strategies and tasks CDOs engage with, but also the depth of how CDOs can pursue strategic goals (Leon, 2014). Damon Williams and Katrina Wade-Golden extended a taxonomy of CDO models (e.g., collaborative, unit, portfolio) which could have easily been appropriate to make apples-to-apples comparisons. As such, this means that they compared a CDO who has limited or no supporting staff, lacks a reporting structure, and often has access to a limited budget (collaborative model) to CDOs who have a portfolio of divisions, programs, and staff that can aid them in accomplishing institutional diversity goals (portfolio model). Researchers have previously (Williams & Wade-Golden, 2007, 2013; Leon 2014) drawn attention to the importance of accounting for these differences when designing CDO roles. Furthermore, they have emphasized the significance of not ignoring the CDO models when conducting research. This study ignores that recommendation, and sets the ground to further disregard the mismatch between the design of the CDO configuration and the size of institutions. This is a significant design flaw.

3). No Acknowledgment of Assumption Violations

The authors inject some nuance into the analysis using what is called a Herfindahl-like index to account for structural diversity among students, faculty, and administrators. This index is maximized when all racial/ethnic groups are, in the author’s words, evenly represented. But we ask in relation to what? There are widely divergent trends across states in terms of equitable representation of minority groups among students, faculty, and administrators in relation to changes in local demographics (Hatch, Garcia, & Sáenz, 2016; Hatch, Mardock-Uman, & Garcia, 2016). For instance, research shows that among Latina/os, the absolute changes in the numbers and proportion of student, faculty, and administrators within colleges do not necessarily keep up with demographic changes within local communities. In fact, the evidence shows higher rates of Latino population growth are related to decreasing equity rates despite numerical growth. Equitable local representation is not the same as numerical diversity. This shortcoming in the analysis matters since the type of analysis the authors use specifically depends on an assumption of unchanging circumstances, something observably untenable given how quickly some parts of the country are transforming demographically relative to others.

Dr. Deryl K. Hatch-Tocaimaza

4). No Account for Variation in Job Duties

The analysis does not acknowledge that CDOs have varying responsibilities, instead, the authors treated all CDOs as the same regardless of whether they were charged with diversifying students, faculty, staff or all of the above. Given this, much attention has been drawn to the findings suggesting that the presence of CDOs has a negative effect on the diversification of tenured faculty. It should be noted that many CDOs are not charged with faculty diversification as a responsibility or priority. Indeed hiring decisions are typically the prerogative of departments with a minimal level of administrative oversight that they follow proper procedures. At a minimum, the analysis should have included a control for the time CDOs spent engaging in the respective tasks that they were assessed on. What does this mean? It means that a CDO who may have a specific charge around student diversity was examined in the same pool with the assumption that they were also charged with addressing faculty diversity despite ample research demonstrating that this is not always the case.

The aforementioned represent significant design issues. Therefore, we suggest that their study results be reviewed with caution and, in the words of the authors, as a rough “first look” at a more complex topic. We are not arguing that all CDOs are successful, or that all institutions with CDOs are capable of reaching their diversification goals merely by hiring these individuals – however, we do argue that this analysis ignores key analytic and theoretical conceptualizations that obscure our understanding of the work of CDOs and their impact on diverse faculty hiring.

Dr. Deryl Hatch-Tocaimaza is an assistant professor in the College of Education and Human Sciences at the University of Nebraksa-Lincoln. Dr. Adriel A. Hilton is the dean of students and diversity officer at Seton Hall University. Dr. Luke Wood is the associate vice president for faculty diversity and inclusion and distinguished professor of education at San Diego State University. Dr. Raul A. Leon is an associate professor of educational leadership at Eastern Michigan University.

References

Hatch, D. K., Garcia, C. E., & Sáenz, V. B. (2016). Latino men in two-year public colleges: State-level enrollment changes and equity trends over the last decade. Journal of Applied Research in the Community College, 23(2), 73–92.

Hatch, D. K., Mardock Uman, N., & Garcia, C. E. (2016). Variation within the “new Latino diaspora”: A decade of changes across the U.S. in the equitable participation of Latina/os in higher education. Journal of Hispanic Higher Education, 15(4), 358–385.

Leon, R. A. (2014). The Chief Diversity Officer: An examination of CDO models and strategies. Journal of Diversity in Higher Education, 7(2), 77.

Williams, D. A., & Wade-Golden, K. C. (2007). The chief diversity officer: A primer for college and university presidents. Washington, DC: American Council on Education.

Williams, D. A., & Wade-Golden, K. C. (2013). The chief diversity officer.  Sterling, VA: Stylus.