At last, we have reached the end of the employment law series. It is important to note that the four cases discussed in this series are in no way exhaustive of all the legal cases that exist with implications for test development and usage. They are simply the major and most often referenced cases.

As test users, if you ever find yourself asking “Am I allowed to do this” before, during or after a test administration, chances are there are legal cases and/or professional guidelines that can answer your question. If you’re still feeling uneasy, try asking the test developer or your legal team for advice. The most common questions I personally receive are:

Although this exam was designed for X position, could I also use it for a similar Y position?

Where should I set my cut score?

How should I determine who passes the exam?

Test validation can be a confusing topic, and it is my hope that this series of posts has clarified the purpose and importance of test validation for test users and developers. I like to think of test validation as the process of gathering evidence that your test is job-related. What does “job-related” mean and how do you gather this evidence? This is where the court cases and legal/professional guidelines really come in handy. For example, the case of Washington v. Davis (1976) involved debate over the strategy used to provide support for job-relatedness — reviewing the Court’s decision on this case can help provide insight into what is likely to be accepted in Court. Other useful resources will be listed at the end of this post, but for now, let’s get started with Ricci v. Destefano (2009).

Ricci v. Destefano, 557 U.S. 557 (2009)

New Haven, Connecticut’s Fire Department required all firefighters seeking promotion to captain or lieutenant to pass a written examination that was developed and validated by a test development company. The African American and Hispanic/Latino firefighters received significantly lower scores on the exams than the White candidates for the position, which would have resulted in none of the minority test takers being eligible for promotion based on their test scores. The Black and Hispanic/Latino applicants then threatened to sue the city of New Haven if they used the results from the test to make promotional decisions and demanded that the city invalidate the test on the basis of adverse impact. Fearing litigation, the city of New Haven complied and settled the case without litigation. The test results were ignored, and no candidates were promoted based on the results.

The White firefighters who would have received promotions based on their passing score were not happy with the city’s decision to invalidate the results and argued that this was unfair due to the exam having been properly validated and administered. They brought their case to the U.S. Supreme Court for verdict.

The Decision: The U.S. Supreme Court ruled that although the results of the exam would have resulted in disparate impact against Black and Latino/Hispanic firefighters, it was unlawful for the city of New Haven to invalidate the results due to the exam having been properly validated and administered.

The Implications: Needless to say, this court decision in particular was controversial and perhaps confusing, but let’s try to clear some things up. First, the Court made it clear that adverse impact alone is not an indication of an invalid test or that a test should not be used for hiring or promotional purposes. This is important because “adverse” or “disparate impact” are words that cause many test users to shudder due to the negative connotation. To avoid using the results of an examination that has sufficient evidence to support it’s use due to fear of adverse impact litigation is therefore unwise for test users, and additional steps need to be taken.

The Court also made clear that when adverse impact is found, the burden of proof shifts to the test user (the city of New Haven) to defend the use of the test as a “business necessity.” That is, the city of New Haven needed to prove that they needed to use the test to be able to properly determine who is qualified and eligible for promotion. Once business necessity has been proven, the city then needs to prove that they investigated alternate means to make the promotional decision that could have achieved the same result without adverse impact. If no alternate means (or alternate test) were found that could achieve the same result without adverse impact, then the Court is likely to rule in favor of the test user.

Test developers and test users should attempt to conduct adverse impact analyses when data permits. Test developers should make this information available to test users, and test users would also benefit from submitting their candidate data to the test development company. These analyses will help to first determine if adverse impact has historically been indicated. If your agency will be using an examination that is known to have adverse impact, be sure to maintain the evidence you need to prove that using the exam is a business necessity (job analysis information, technical reports, etc.). Test users are also encouraged to investigate many alternate testing means before making their final decision on which test to use — doing so also provides support for using your chosen exam.

References

Ricci v. DeStefano, 557 U.S. 557 (2009)

Other useful resources

SIOP’s Principles for the Validation and Use of Selection Procedures (2019)

APA, AERA, and NCME’s Standards for Educational and Psychological Testing (2014)

Uniform Guidelines on Employee Selection Procedures (1978)

Adverse Impact: Implications for Organizational Staffing and High Stakes Selection (2010) (Edited by James L. Outtz)

It’s Friday, and I’m sitting at my desk singing “Woahhh, we’re halfway there. Woahhh! Livin’ On a Prayer.” Why? Because we are halfway through the employment law series! In the first two posts, we reviewed the cases of Griggs v. Duke Power Co. (1971) and Albemarle Paper Co. v. Moody (1975) and brushed up on the important decisions that affect test development and usage. For the final two posts of this series, we will review the cases of Washington v. Davis (1976) and Ricci v. DeStefano (2009). Let’s get started!

Washington v. Davis, 426 U.S. 229 (1976)

The Metropolitan Police Department of the District of Columbia (MPD) required all applicants seeking entry into the police academy to pass an exam intended to assess cognitive traits such as verbal ability, vocabulary and reading comprehension. At their set cut score, approximately four times as many Black applicants failed the test compared to Whites. African American test-takers who failed the exam at the set cut score then sought judgment that the exam unlawfully discriminated against Black applicants.

The case first went to two lower courts for verdict. The District Court initially ruled that the exam was “directly related to a determination of whether the applicant possesses sufficient skills requisite to the demands of the curriculum a recruit must master at the police academy.” In other words, the District Court ruled that the exam had been validated and shown to be related to “the demands a recruit must master at the police academy.”

However, the DC Court of Appeals then ruled the opposite: “the adverse impact was sufficient to establish a constitutional violation, absent proof by petitioners that the test was an adequate measure of job performance in addition to being an indication of probable success in the training program.” In other words, the DC Court of Appeals ruled that it was not enough to show that the exam was related to the demands of the police academy, but also to the job itself.

The Decision: The Supreme Court made the final decision that the District Court was correct and the DC Court of Appeals erred. The test “was directly related to the requirements of the police training program and … a positive relationship between the test and training-course performance was sufficient to validate the former, wholly aside from its possible relationship to actual performance as a police officer.”

The Implications: In a nutshell, the conflict between the District Court and the Court of Appeals concerned the appropriateness of the validation strategy used. When candidates challenge hiring decisions based on disparate impact, the test development and validation strategy are usually reviewed by the courts — and with great scrutiny — to investigate the job-relatedness of the exam.

MPD attempted to provide support for the validity or job-relatedness of the exam by presenting the significant relationship found between exam scores and police academy performance. The Court of Appeals challenged this validation strategy as inappropriate on its own due to the criterion measure of “police academy performance.” Instead, the Court of Appeals argued that a relationship needed to be established between exam scores and on-the-job performance. Ultimately, the Supreme Court ruled that the validation strategy was appropriate and the relationship between exam scores and police academy performance was sufficient to provide support for job-relatedness.

The implications for test developers are huge! The criteria used in validation studies are not limited to job performance. You do not absolutely have to establish a relationship with the exam and job performance for it to be valid or constitutionally acceptable.This is great news for test developers because adequate measures of job performance are not always available, and not being limited to this sole criterion measure allows many other avenues for investigating the relationship between test scores and aspects of the job.

Test users should be sure to read the test’s technical report for details on any studies investigating the relationship with the exam and aspects of the job. Look at the type of job aspects used in the studies (e.g., job performance, academy scores, training performance) and how it was measured (e.g., supervisory ratings of performance, attendance records, negative customer service reports) to see if you measure the same job aspects in your agency. This information can help you gauge the appropriateness of using the exam at your agency.

References

Washington v. Davis, 426 U.S. 229 (1976)

In my previous post, Thoughts on Adverse Impact Part 1, I offered my suggestions on how to plan and think about an adverse impact study. Summarizing and reviewing some of my main points:

From the Practitioner Perspective, adverse impact involves a technology, not a science.

• The ways in which we prepare for, calculate, and interpret the results of an adverse impact analysis are guided primarily by the Uniform Guidelines and case law, as opposed to strict principles of statistics.
• Adverse Impact can be defined as practical or significant differences in selection rate as a function of protected group status.

In this month’s blog, I deal with more commonly discussed issues, such as various approaches to quantifying adverse impact and the sequencing of tests. Again, as a caution, this blog does contain my opinions on a controversial topic. In addition, although I have tried to simplify the discussion, this is a complex topic and it would be difficult enough to explain in a long article or book, let alone a blog.

Distinguishing Between Adverse Impact and Subgroup Differences

Although adverse impact and subgroup differences between protected classes are distinct concepts, the two terms are often used interchangeably, not only in the practitioner literature but by experts who should know better. In order to appreciate some of the issues in the analysis of adverse impact, one must first develop an understanding of the difference between these two frequently confused concepts. (more…)

• If your head hurts just thinking about adverse impact. Cheer up, you are not alone.

A long time ago, I had an international student ask me why my class was not titled Thoughts of Professor Doverspike; in her country, all classes were simply listed as Thoughts of Professor _____. I have always thought that was a wonderful idea and so this month’s blog is Thoughts on Adverse Impact (actually, I originally titled it Thought on Adverse Impact, but I was unsure whether that was a typo, a Freudian slip, or an accurate depiction of my limited cognitive ability).

I have spent most of my professional life, which now amounts to over 35 years, dealing with adverse impact issues. During this time period I have found that most human resource professionals, from the novice to the experienced employment attorney, struggle when it comes to understanding even the basics of adverse impact.

From the Practitioner Perspective, adverse impact involves a technology, not a science. Thus, in my opinion, the ways in which we prepare for, calculate, and interpret the results of an adverse impact analysis are guided primarily by the Uniform Guidelines and case law, as opposed to strict principles of statistics. Therefore, after offering a brief definition and thoughts on technology, I first offer up my suggestions on how to plan and think about your adverse impact study. In my next blog, I will deal with more commonly discussed issues, such as various approaches to quantifying adverse impact, the sequencing of tests, and the interpretation of results.

As a serious caution, this blog does contain my opinions on a controversial topic. It would not be difficult to find other experts who might disagree with some of my recommendations and conclusions. (more…)