- If your head hurts just thinking about adverse impact. Cheer up, you are not alone.
A long time ago, I had an international student ask me why my class was not titled Thoughts of Professor Doverspike; in her country, all classes were simply listed as Thoughts of Professor _____. I have always thought that was a wonderful idea and so this month’s blog is Thoughts on Adverse Impact (actually, I originally titled it Thought on Adverse Impact, but I was unsure whether that was a typo, a Freudian slip, or an accurate depiction of my limited cognitive ability).
I have spent most of my professional life, which now amounts to over 35 years, dealing with adverse impact issues. During this time period I have found that most human resource professionals, from the novice to the experienced employment attorney, struggle when it comes to understanding even the basics of adverse impact.
From the Practitioner Perspective, adverse impact involves a technology, not a science. Thus, in my opinion, the ways in which we prepare for, calculate, and interpret the results of an adverse impact analysis are guided primarily by the Uniform Guidelines and case law, as opposed to strict principles of statistics. Therefore, after offering a brief definition and thoughts on technology, I first offer up my suggestions on how to plan and think about your adverse impact study. In my next blog, I will deal with more commonly discussed issues, such as various approaches to quantifying adverse impact, the sequencing of tests, and the interpretation of results.
As a serious caution, this blog does contain my opinions on a controversial topic. It would not be difficult to find other experts who might disagree with some of my recommendations and conclusions.
For an excellent and comprehensive discussion of the topic of adverse impact see Considerations in Addressing Adverse Impact by Dianna E. Belman, which is available as a free download at the IPMA-HR Assessment Services site.
Adverse Impact can be defined as practical or significant differences in selection rate as a function of protected group status. Terms such as hiring rate or passing rate can be substituted for selection rate, and the term selection rate can refer to any type of personnel decision that results in a change in a person’s employment status. Thus, if I am using a physical ability test with firefighters and the assessment results in a significantly higher passing rate for Males than for Females, I would interpret this difference as being consistent with a finding of adverse impact as a function of sex.
Although dated, and it could be argued outdated, the Uniform Guidelines state that:
The use of any selection procedure which has an adverse impact on the hiring, promotion, or other employment or membership opportunities of members of any race, sex, or ethnic group will be considered to be discriminatory and inconsistent with these guidelines, unless the procedure has been validated in accordance with these guidelines, or the provisions of section 6 of this part are satisfied. Furthermore, the Uniform Guidelines offer the famous 4/5ths rule: A selection rate for any race, sex, or ethnic group which is less than four-fifths (4/5) (or eighty percent) of the rate for the group with the highest rate will generally be regarded by the Federal enforcement agencies as evidence of adverse impact, while a greater than four-fifths rate will generally not be regarded by Federal enforcement agencies as evidence of adverse impact.
Be Wary of an Over Reliance on Technology
Thanks to advances in technology, it is relatively easy to go online, Google, and find a number of adverse impact calculators. Thus, an excellent question would be – Doverspike, why do I need to pay attention to your ramblings when I can just take my numbers and enter them into a statistical adverse impact calculator online and get the desired answer? My reply would be for the same reason you should be cautious in medical self-diagnosis online; that is, I would argue that preparing for and interpreting the results of an adverse impact study requires an experienced eye, an understanding of one’s assumptions, and careful consideration of those variables impacting the results. It is far too easy to simply have a clerk input data into an online calculator that spits out adverse impact ratios; even though adverse impact is best seen as a technology, expertise is still needed in the application and analysis of the online tools available to the assessment professional. One of my purposes in this blog is to start you thinking about what you are really doing when you conduct a study of adverse impact; that is, to carefully plan your analysis strategy.
Planning for an Analysis of Adverse Impact
A professional analysis of adverse impact involves the following steps and associated questions:
- A specific event, decision, or assessment must be identified. Although it may seem obvious, in order to analyze adverse impact we must first identify the appropriate associated personnel decision, selection event, or test. This corresponds to asking the question as to what is the decision-making event or employment practice being considered. Although this may seem like a simple matter, it is often highly controversial and contested in court cases. In public sector assessment one of the main issues is whether the analysis should reflect only one hurdle or test in a battery, or if the analysis corresponds to the outcome from a series of tests or decisions. For example, in police selection, are we just looking at adverse impact from our cognitive ability measure, or are we studying the whole selection process from application to admission to the police academy. This corresponds to the bottom-line issue, which was addressed by the Supreme Court in Connecticut v. Teal, 1982. Another example occurs in age discrimination cases involving reductions in force, where the question is whether the decision process involved the new organization rehiring employees or the old organization laying off employees.
- A group of applicants must be identified. Who is an applicant? If someone withdraws from the process, are they still an applicant? If an individual is disqualified because they lied on their resume, are they still an applicant? Is everyone who sent in a resume an applicant? If a person does not meet the minimum qualifications, are they an applicant? Again, what on the surface might seem to be a simple question, often becomes quite complicated. My recommendation is that every agency should be proactive in developing official definitions of different types or levels of applicants; for example, you might have initial candidates, qualified applicants, and preferred applicants, based on the needs and definitions adopted by your agency.
- It must be possible to identify the protected group status of the individuals. In order to compute adverse impact ratios we need to categorize individuals based on their protected group status. No doubt, many readers would be surprised at how often an adverse impact analysis is based on conjectures regarding the sex or race of individuals. As an expert, I have often been given a list of candidates and told that I have to make a guess as to their sex based on their first name. As you might imagine, this is even more difficult with race or ethnicity. Creative approaches aside, if you are going to calculate adverse impact, you need information on the protected group status of individuals. You should consult with your employment attorney in order to determine the best way to collect demographic information without biasing decisions made during the selection process.
- Based on the decision-making event, some individuals from the group of applicants will have received a favorable decision and some individuals will have received an unfavorable decision. Seemingly pretty straightforward, we must be able to determine whether individuals received a favorable decision (e.g., usually passed the test or were hired), or an unfavorable decision (e.g., failed the test or were not hired). Of course, this type of decision is related to whether the individual is considered an applicant. How do we classify an individual that was offered a job but turned it down? What about a candidate for promotion who retires three days before they would have received a promotion, or retires three days after receiving their promotion? Again, my recommendation is to be proactive in determining ahead of time how to classify or define borderline cases.
- There must be an appropriate hypothesis or question. There are large differences of opinion between experts on the question actually being assessed through an adverse impact analysis. However, at least in the case of public sector agencies, I believe the question of interest is related to merit. That is, I would argue that the hypothesis we are testing is: whether two individuals of equal merit will have the same probability of receiving a favorable decision (for example, passing a test) regardless of the protected group status of the individuals. Thus, if I have a Hispanic applicant and a White applicant of equal merit, I would expect both individuals to have the same probability of being hired or passing a test.
- An appropriate statistical test must be applied. There are a number of options in terms of potential statistical tests and also the order of tests. We will consider this topic in more detail in the next blog.
- Explore alternative explanations. Although bias or discrimination is a possible reason for adverse impact, it is not the only possible explanation. Following a finding of adverse impact, there should be a search for possible reasons for the obtained finding. One possible explanation is that there are real group differences on the test or underlying construct. In promotional testing, groups may differ in their ability to find time to study for a job knowledge test. In entry level testing, groups may have large differences in their educational or experience levels. The consideration of potential explanations for a finding of adverse impact is critical both in terms of justifying the results of previous screenings and in developing more effective selection systems so as to reduce future adverse impact.
Conclusion Adverse impact is at once both a simple and an extremely complex concept. If you are reading this blog as a relative newcomer to assessment, then you may be interested in a discussion of the basics. If you are more experienced, then you already know how difficult it is to fully understand adverse impact. I hope my thoughts have added some clarity for both beginners and experts. In my next blog, I will continue the discussion of adverse impact by addressing the topics of the common statistical tests.