As indicated in the first article in this series, a thorough job analysis should be the foundation for most of the technical work performed in Human Resources. We also discussed that while analysts in the field today may not necessarily need to be able to design their own job analysis systems and create written exams from the results, they should have an understanding of the process and the ability to recognize whether or not products and vendors meet professional standards and can stand up to court scrutiny.

Our focus, as suggested above, will be the utilization of job analyses to create content valid selection instruments with other uses for job analyses results being discussed in the next article. In addition, it is important to stress the Uniform Guidelines on Employee Selection Procedures (UGESP 1978) along with the Society for Industrial and Organizational Psychology (SIOP) Principles for the Validation and Use of Personnel Selection Procedures (Principles, 2003) are still the guiding documents for determining the adequacy of content validation procedures (see references at the end of this post). It is also important to note that the UGESP (1978) don’t apply only to written exams, but to all selection instruments. That includes oral exams, physical fitness tests, background investigations and one-on-one hiring interviews.

In addition to meeting these requirements, we should never lose sight of the fact that we want our selection instruments to be valid for their intended use. Yes, we want to be able to win any challenges, but primarily utilizing valid selection instruments enables us to select the best people available for the jobs we are filling. After all, that is the bottom line in test validity.

One of the best ways to evaluate the adequacy of a product or, in this case, a test developed by a consultant, is to review the test and the report prepared from their analyses using the UGESP and the SIOP Principles as a yardstick. In that regard, the IPMA-HR test development team provided me with a copy of one of their reports and I can say that it is one of the best and most comprehensive reports I have ever seen. The report includes the steps taken to perform a comprehensive job analyses and the steps involved in the test development process. Reading and re-reading reports like these is a valuable practice in gaining an understanding of what job analyses are all about and what they are designed to accomplish.

In particular, some of the key issues include the linkage we discussed in the first article. Essentially, the flow starts with the job, which is broken down into individual Tasks, the Knowledge, Skills, Abilities and Personal Characteristics (KSAP’s) are generated from the Tasks and linked back to the Tasks demonstrating that particular KSAP’s are necessary to perform the Tasks identified. Then actual selection instruments are developed using the KSAP’s that are necessary to perform the Tasks. Thus, when this process is done correctly, as it was in this case, there is a flow connecting the job to the test and the test back to the job.

In the IPMA-HR report, the table of contents demonstrates this linkage just as it is prescribed in the Guidelines. In addition, another point that I made in the first article is reflected in this report. Specifically, that is the benefit of having broader participation and more subject matter experts available when studies are done by test developers as opposed to conducting studies in house. In this case, the study included 44 cities that represented 13 states and 750 Job Analysis Questionnaires were provided to SME’s. The significant value is that all the results are more stable and thus more reliable than studies done with a limited number of SME’s which would be the case when forced to rely on only those available within your own department.

Walking through the table of contents, the areas that are noted as being discussed in the body of the report demonstrate conformance to the appropriate Guidelines. In particular, the report documents the job analysis questionnaire development and distribution along with a supplemental job analysis development and distribution to confirm some of the results obtained from the questionnaire. The job analysis results are discussed and the focus is on Tasks along with their Importance and Frequency Ratings and their linkage to KSAP’s that have been rated regarding their importance. Essentially this shows that the KSAP’s that have been identified as important for success on the job are also linked to Tasks that are an important part of doing the job.

In that regard, these ratings for Tasks and KSAP’s demonstrate compliance with the UGESP in that they state, “Any job analysis should focus on the work behaviors and the Tasks associated with them…The work behaviors selected for measurement should be critical work behaviors and/or important work behaviors constituting most of the job.” Further, the Guidelines state, “For any selection procedure measuring a knowledge, skill, or ability, the user should show that (a) the selection procedure measures and is a representative sample of that knowledge, skill, or ability and (b) that knowledge, skill, or ability is used in and is a necessary prerequisite to performance of critical or important work behaviors.”

In this report, it also shows that an additional step was taken to group all the Tasks involved in performing a related function into duty areas. This is extremely beneficial in identifying the duty areas performed in a job as it assists in test development as well as providing a basis for broader uses of the job analyses which we will discuss in the next article.

Also available in the report provided by IPMA-HR is a comprehensive list of tables that show the results obtained from the job analysis questionnaires. Tables like these strengthen the report and give individuals considering hiring a test developer or using the products created by one, tremendous insight into what was collected and how it was used. Reviewing the narrative of the report and studying the tables helps new professionals school themselves in job analysis and test development. In particular, tables can often introduce analysts to new terms, and if analysts that are confronted with new terms will take the time to research terms new to them, when they are encountered, they can become more knowledgeable in their field and better prepared to evaluate the work of consultants and test publishers.

Because of the nature of this particular study which involved several agencies in a consortium, the report, out of necessity, includes information about how it was determined that it was appropriate to include each agency that participated in the study. Since this report relates to the development of a police officer exam that would be used by multiple jurisdictions, it is critical to be sure that the jobs in these jurisdictions are essentially the same and the populations served are similar. Reviewing the Tasks and KSAP’s for each jurisdiction provides this information and is typically referred to as a transportability study. The UGESP indicate that a selection procedure which is supported on the basis of content validity may be used for a job if it represents a critical work behavior or work behaviors that constitute most of the important parts of the job. The report provided by IPMA-HR demonstrates that they went into great detail to ensure the transportability of the test they developed along with ensuring that each agency that participated in the study was appropriate for inclusion.

While it is difficult to provide a comprehensive review of a study as large as the one represented by the report provided to me, for our purposes, we can see that the essential aspects for supporting content validity were followed and documented. The relationship between the job and the test is demonstrated and the justification for including each jurisdiction is provided. In particular, one can find Tasks on the table where they are listed and then move to the KSAP list to see which elements on that list relate to the Tasks spelled out on the Task list. Then one can go to the test and look at the elements that comprise it and how they are designed to measure the important KSAP’s that are linked back to the Tasks on the original Task list. While it may sound a bit confusing spelled out here, with an actual report in your hands, you should be able to follow from the job to the test and back again which will help you determine the adequacy of a job analysis and test development process you are reviewing.

The report completed by IPMA-HR provides a good learning tool and introduces many concepts that a new analyst should familiarize himself or herself with. In addition the test itself represents a direct link between the behaviors performed to be successful on the test with behaviors that have to be performed on the job to be successful in its performance. Also of note is the fact that this report does not stop there and it does not rely totally on a content validity strategy for supporting the validity of the police officer test.

In previous articles, we have discussed the types of validity. Essentially, we have content, criterion and construct as the three types. We won’t review them all here, but just as a refresher, I want to note that criterion related validity involves demonstrating that there is a relationship between test performance and a measure of job performance. Ideally we want to show that as test scores go up so does a job related criterion such as job performance. The process typically involves comparing test scores with performance ratings from supervisors with the hope that the test developer can demonstrate that as test scores go up so do job ratings. This is what is known as a positive correlation. Correlations can also be negative which means that as one score goes up the other goes down. Therefore, correlation coefficients can range from -1 to +1. In this case, the higher the positive correlation the more related our test scores are to job performance.

Now, with that in mind, it is good to know that criterion related validity can be broken down into two types; predictive and concurrent. If we test applicants and then take performance ratings from the ones that are hired after they have been on the job for a reasonable period of time, we are essentially utilizing a form of predictive criterion related validity. If we use current data, that is, we give current incumbents our entry level test and collect data on their current level of job performance and compare the two, we are looking at concurrent criterion related validity. The report I got indicated that IPMA-HR not only developed their police officer test utilizing the content validity model, but they went an extra step to determine the exams concurrent criterion related validity.

So that is a whirlwind tour through job analysis and test development. Hopefully, this will help in your education as an HR analyst and encourage you to get more details regarding job analysis and test development. Technical reports are a good starting place for that education. The goal should be to understand the process and assist in your ability to evaluate the adequacy of consultants and test publishers you are considering.

References

Equal Employment Opportunity Commission, the Civil Service Commission, the

Department of Labor and the Department of Justice. (1978). Uniform guidelines on employee selection procedures. Federal Register, Volume 43, Number 166, 38290-38315. Uniform Guidelines .pdf

Society for Industrial and Organizational Psychology, Inc. (2003, 4rd ed.). Principles for the validation and use of personnel selection procedures. College Park, MD. Telephone Number: (708)640-0068 http://www.siop.org/_principles/principles.pdf