At last, we have reached the end of the employment law series. It is important to note that the four cases discussed in this series are in no way exhaustive of all the legal cases that exist with implications for test development and usage. They are simply the major and most often referenced cases.

As test users, if you ever find yourself asking “Am I allowed to do this” before, during or after a test administration, chances are there are legal cases and/or professional guidelines that can answer your question. If you’re still feeling uneasy, try asking the test developer or your legal team for advice. The most common questions I personally receive are:

Although this exam was designed for X position, could I also use it for a similar Y position?

Where should I set my cut score?

How should I determine who passes the exam?

Test validation can be a confusing topic, and it is my hope that this series of posts has clarified the purpose and importance of test validation for test users and developers. I like to think of test validation as the process of gathering evidence that your test is job-related. What does “job-related” mean and how do you gather this evidence? This is where the court cases and legal/professional guidelines really come in handy. For example, the case of Washington v. Davis (1976) involved debate over the strategy used to provide support for job-relatedness — reviewing the Court’s decision on this case can help provide insight into what is likely to be accepted in Court. Other useful resources will be listed at the end of this post, but for now, let’s get started with Ricci v. Destefano (2009).

Ricci v. Destefano, 557 U.S. 557 (2009)

New Haven, Connecticut’s Fire Department required all firefighters seeking promotion to captain or lieutenant to pass a written examination that was developed and validated by a test development company. The African American and Hispanic/Latino firefighters received significantly lower scores on the exams than the White candidates for the position, which would have resulted in none of the minority test takers being eligible for promotion based on their test scores. The Black and Hispanic/Latino applicants then threatened to sue the city of New Haven if they used the results from the test to make promotional decisions and demanded that the city invalidate the test on the basis of adverse impact. Fearing litigation, the city of New Haven complied and settled the case without litigation. The test results were ignored, and no candidates were promoted based on the results.

The White firefighters who would have received promotions based on their passing score were not happy with the city’s decision to invalidate the results and argued that this was unfair due to the exam having been properly validated and administered. They brought their case to the U.S. Supreme Court for verdict.

The Decision: The U.S. Supreme Court ruled that although the results of the exam would have resulted in disparate impact against Black and Latino/Hispanic firefighters, it was unlawful for the city of New Haven to invalidate the results due to the exam having been properly validated and administered.

The Implications: Needless to say, this court decision in particular was controversial and perhaps confusing, but let’s try to clear some things up. First, the Court made it clear that adverse impact alone is not an indication of an invalid test or that a test should not be used for hiring or promotional purposes. This is important because “adverse” or “disparate impact” are words that cause many test users to shudder due to the negative connotation. To avoid using the results of an examination that has sufficient evidence to support it’s use due to fear of adverse impact litigation is therefore unwise for test users, and additional steps need to be taken.

The Court also made clear that when adverse impact is found, the burden of proof shifts to the test user (the city of New Haven) to defend the use of the test as a “business necessity.” That is, the city of New Haven needed to prove that they needed to use the test to be able to properly determine who is qualified and eligible for promotion. Once business necessity has been proven, the city then needs to prove that they investigated alternate means to make the promotional decision that could have achieved the same result without adverse impact. If no alternate means (or alternate test) were found that could achieve the same result without adverse impact, then the Court is likely to rule in favor of the test user.

Test developers and test users should attempt to conduct adverse impact analyses when data permits. Test developers should make this information available to test users, and test users would also benefit from submitting their candidate data to the test development company. These analyses will help to first determine if adverse impact has historically been indicated. If your agency will be using an examination that is known to have adverse impact, be sure to maintain the evidence you need to prove that using the exam is a business necessity (job analysis information, technical reports, etc.). Test users are also encouraged to investigate many alternate testing means before making their final decision on which test to use — doing so also provides support for using your chosen exam.

References

Ricci v. DeStefano, 557 U.S. 557 (2009)

Other useful resources

SIOP’s Principles for the Validation and Use of Selection Procedures (2019)

APA, AERA, and NCME’s Standards for Educational and Psychological Testing (2014)

Uniform Guidelines on Employee Selection Procedures (1978)

Adverse Impact: Implications for Organizational Staffing and High Stakes Selection (2010) (Edited by James L. Outtz)

It’s Friday, and I’m sitting at my desk singing “Woahhh, we’re halfway there. Woahhh! Livin’ On a Prayer.” Why? Because we are halfway through the employment law series! In the first two posts, we reviewed the cases of Griggs v. Duke Power Co. (1971) and Albemarle Paper Co. v. Moody (1975) and brushed up on the important decisions that affect test development and usage. For the final two posts of this series, we will review the cases of Washington v. Davis (1976) and Ricci v. DeStefano (2009). Let’s get started!

Washington v. Davis, 426 U.S. 229 (1976)

The Metropolitan Police Department of the District of Columbia (MPD) required all applicants seeking entry into the police academy to pass an exam intended to assess cognitive traits such as verbal ability, vocabulary and reading comprehension. At their set cut score, approximately four times as many Black applicants failed the test compared to Whites. African American test-takers who failed the exam at the set cut score then sought judgment that the exam unlawfully discriminated against Black applicants.

The case first went to two lower courts for verdict. The District Court initially ruled that the exam was “directly related to a determination of whether the applicant possesses sufficient skills requisite to the demands of the curriculum a recruit must master at the police academy.” In other words, the District Court ruled that the exam had been validated and shown to be related to “the demands a recruit must master at the police academy.”

However, the DC Court of Appeals then ruled the opposite: “the adverse impact was sufficient to establish a constitutional violation, absent proof by petitioners that the test was an adequate measure of job performance in addition to being an indication of probable success in the training program.” In other words, the DC Court of Appeals ruled that it was not enough to show that the exam was related to the demands of the police academy, but also to the job itself.

The Decision: The Supreme Court made the final decision that the District Court was correct and the DC Court of Appeals erred. The test “was directly related to the requirements of the police training program and … a positive relationship between the test and training-course performance was sufficient to validate the former, wholly aside from its possible relationship to actual performance as a police officer.”

The Implications: In a nutshell, the conflict between the District Court and the Court of Appeals concerned the appropriateness of the validation strategy used. When candidates challenge hiring decisions based on disparate impact, the test development and validation strategy are usually reviewed by the courts — and with great scrutiny — to investigate the job-relatedness of the exam.

MPD attempted to provide support for the validity or job-relatedness of the exam by presenting the significant relationship found between exam scores and police academy performance. The Court of Appeals challenged this validation strategy as inappropriate on its own due to the criterion measure of “police academy performance.” Instead, the Court of Appeals argued that a relationship needed to be established between exam scores and on-the-job performance. Ultimately, the Supreme Court ruled that the validation strategy was appropriate and the relationship between exam scores and police academy performance was sufficient to provide support for job-relatedness.

The implications for test developers are huge! The criteria used in validation studies are not limited to job performance. You do not absolutely have to establish a relationship with the exam and job performance for it to be valid or constitutionally acceptable.This is great news for test developers because adequate measures of job performance are not always available, and not being limited to this sole criterion measure allows many other avenues for investigating the relationship between test scores and aspects of the job.

Test users should be sure to read the test’s technical report for details on any studies investigating the relationship with the exam and aspects of the job. Look at the type of job aspects used in the studies (e.g., job performance, academy scores, training performance) and how it was measured (e.g., supervisory ratings of performance, attendance records, negative customer service reports) to see if you measure the same job aspects in your agency. This information can help you gauge the appropriateness of using the exam at your agency.

References

Washington v. Davis, 426 U.S. 229 (1976)