The idea for this primer series germinated from a simple question – “Could you do an article looking at the validity of tests used in public safety assessment?” In response, I decided to do a series of articles aimed to inform, but also intended to keep things simple. The blogs in this series are intended to cover:

  1. What are the characteristics of a good test?
  2. What are some authoritative references human resource and assessment professionals can rely upon in evaluating the worthiness of tests?
  3. What is validity?
  4. Are public safety assessments good tests and are they valid?

The first article in the primer series has been published and is available on the IPMA-HR Assessment Services Review page.

This is the second in the series and is intended to answer the question as to where can the reader turn for guidance in addition to that offered in this series of blogs. My suggested list is fairly short and includes:

The Guardians Court Case 

For my money, the most useful resource for public sector testing professionals is not a document or article, but a court case – The Guardians Association of the New York City Police Dept. v. Civil Service Commission of the City of New York (1980). This case lays out a very practical approach to content validity; it distills from The Uniform Guidelines a set of critical principles. As so eloquently stated by the court, those principles include:

  1. There must be a suitable job analysis. Of course, what constitutes a “suitable” job analysis can be open to judgment. If you do not feel confident in your ability to conduct a job analysis, you might want to consider attending an IPMA-HR workshop on job analysis.
  2. The test constructors must have a reasonable level of competence. This would apply both to the testing professionals in charge of the test as well as any subject matter experts. As far as the test constructors are concerned, they should understand the characteristics of a good test, as well as the principles of effective item writing.
  3. The content of the test must be related to the content of the job. This means the items or exercises required as part of the examination should have logical relationships to important tasks performed on the job.
  4. The content of the test must be representative of the content of the job. This is probably the most difficult of the principles to understand, as it really has two different meanings. First, it could be seen as indicating that the overall assessment battery should tap most of the important knowledge, skills, and abilities. Second, it could be interpreted as meaning that the format used for the test has a logical relationship to the job. For example, a test that required reading for a job where no reading was required, would be unlikely to meet the representativeness requirement.
  5. A scoring system that selects applicants who can better perform the job. Of course, the best way to assure an adequate scoring system is to meet the characteristics of a good test described in the previous blog. That is, the test should provide a reliable and valid measure of the characteristics in question.

The Uniform Guidelines

Released in 1978, The Uniform Guidelines were a product of the Equal Opportunity Commission, the Civil Service Commission, the Department of Labor, and the Department of Justice. Although a series of questions and answers were soon released, no update or revision has been published. The lack of a meaningful revision is a topic frequently mentioned by critics. In particular, The Uniform Guidelines were written before the introduction of validity generalization and during a period where many professionals still did not have access to calculators or computers for producing statistical analyses. Of particular value and note are Sections 14, which provides technical standards for content, criterion-related, and construct validity studies, and 15, which describes how to document evidence of adverse impact and validity.

Given the ubiquitous nature of The Uniform Guidelines, I am always surprised that testing professional do not pay closer attention and make greater use of The Uniform Guidelines. I am not a fan nor encouraging slavish adherence, yet The Uniform Guidelines lay out specifics on how to validate, and more critically, how to document the validity of test. 

The American Psychological Association (APA) Standards 

Although often shortened to the APA Standards, this document is more correctly referred to as The Standards for Educational and Psychological Testing and is a product of the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. First issued in 1966, the newest revision was in 2015. Although often regarded as a “gold standard” for testing, my own opinion is that the nature of The Standards has shifted quite a bit over the years. In particular, while once it could be seen as more similar to essential guidelines or basic professional protocols, I would argue it now serves as more of a textbook on testing.

Much of the material may be too technical for the beginning user and too impractical for many other uses. One problem we have in our field is that psychometrics is designed for very large numbers of people and very large numbers of items. This is not a situation encountered by the typical testing professional, especially in the public sector. Having said that, any individual involved in testing should be familiar with The Standards.

The Society for Industrial and Organizational Psychology (SIOP) Principles

The caveat here would be that there is a new edition, the 5th, which is under the final stages of review and should be released in 2018. Thus, the Fourth Edition of the Principles for the Validation and Use of Personnel Selection Procedures is somewhat dated, having been published in 2003. However, it is still a valuable resource, although it may be on the technical side for those new to testing and validation. As with the APA Principles, I tend to view the SIOP Principles as more of a primer, guide, or textbook, than a strict set of guidelines or professional protocols.  

US Department of Labor (DOL) and Training Administration, Testing and Assessment: An Employer’s Guide to Good Practices

Although somewhat dated, having a 1999 copyright, this DOL guide is free for downloading. In addition, there is practical value in being able to say “we read and complied with the DOL Guide to Good Practices.” Intended as a basic primer, the Guide presents the essential concepts of employment testing in easy-to-understand terms. This includes topics such as administering and scoring tests, interpreting results, and understanding the professional and legal standards.

IPMA-HR Assessment Services Publications

IPMA-HR Assessment Services offers a number of free publications that provide basic information on administering tests and validating assessments. (Note: Add link). A very useful summary of a number of selection topics is provided by Considerations in Implementing Selection Procedures (Note: Add link). Of course, all tests from IPMA-HR Assessment Services, including the safety forces examinations, are designed to be consistent with the professional principles for the development of tests and are carefully developed so as to be high-quality tests. If you have questions regarding pre-employment or promotional testing, please feel free to contact the staff at IPMA-HR Assessment Services.

Final Thoughts

In addition to the documents above, the more experienced testing professional will want to be familiar with applicable laws including the Civil Rights Act, 1964, 1991, the  Age Discrimination in Employment Act, 1967, and the Americans with Disabilities Act Amendments Act, 2008. Additional knowledge can be obtained by attending conferences or workshops offered by IPMA-HR.

Our first two blogs have covered:

  1. What are the characteristics of a good test?
  2. What are some authoritative references human resource and assessment professionals can rely upon in evaluating the worthiness of tests?

Our next two blogs will cover:

  1. What is validity?
  2. Are public safety assessments good tests and are they valid?

On the personal side, as you read this I will be in a state of at least temporary retirement from The University of Akron. So, my standard biography write-up has changed slightly. In reality, I plan on working harder than ever, unless someone has an island where I can lay in the sun. In the meantime, if you have any questions or thoughts for me, please, email Dennis Doverspike at dennisdoverspike@gmail.com. As always, if you have a question you would like to see addressed in a future blog, please let us know.