Now we’ll get into more of the details surrounding the development of a biodata instrument. To help illustrate how such instruments are developed — and how the collection and scoring of this type of information can enhance selection systems — we’ll review the development of IPMA-HR’s Correctional Officer Biodata Questionnaire (CO-BDQ) through its Technical Report, which describes in detail the steps IPMA-HR and Bruce Davey and Associates (BDA) took to develop and validate the 120-item test.
By the end of this article, you should have a clearer picture of what information is gathered in order to develop a biodata instrument, how it relates to the job, how candidates are tested and how their results determine how good of a “fit” they are with the profile developed of successful job performers. Keep in mind throughout that, as with any selection instrument, the CO-BDQ had to be two things: reliable (i.e., measure what it measures consistently) and valid (i.e., measure what it is supposed to measure).
Developers wanted the CO-BDQ to measure the potential for good job performance coupled with low probability of turnover — doing this consistently would prove its reliability. Because of the nature of this type of instrument and the type of information it gathers and utilizes, extra effort had to be made to demonstrate its efficacy.
Developing and validating selection instruments is an expensive, time-consuming and labor intensive process; those participating in the development of the test with the intention of using it as a part of their selection system had to be confident that the undertaking was indeed worthwhile. For the CO-BDQ, this meant its use would increase the predictive validity of the entire selection process — in this case, one utilizing IPMA-HR’s entry-level correctional officers’ cognitive test.
As a general rule, each instrument used in the selection process should increase the value of the entire process incrementally. In other words, if you have a good written exam and a good oral exam, both of which correlate with job performance, then any additional instrument should supply you with information not already provided by the written or oral exam — in addition to being valid, reliable, and correlated with job performance. Otherwise, you would be spending extra time and money on duplicate information rather than improving the process.
Early studies on the use of biodata (in general) for selection purposes looked at biodata instruments as a replacement for the cognitive parts of selection processes. It was hoped that biodata instruments would not reflect the same level of adverse impact as cognitive instruments had; however, their efforts were not consistently successful. Researchers then determined that using a biodata instrument in conjunction with other instruments was the best approach for selecting public safety officers as it offered the greatest amount of information on which to base hiring decisions.
One of the jurisdictions I worked with in the early 1980s was sued by the Department of Justice for patterns and practices of selection and promotion that reflected adverse impact. (The discriminatory practices that brought about the lawsuit occurred before my tenure with this particular agency; I was actually brought in to produce new batteries that were both valid and reliable.) As part of the settlement, we worked with Richardson, Bellows, and Henry, a Washington D.C.-based consulting firm, to create both a cognitive instrument and a biodata instrument. The work at that time suggested that the cognitive instrument still had adverse impact, but the biodata instrument did not.
The experience of working with numerous consultants along with my own staff gave me a much greater appreciation for the importance of validating selection procedures. It also taught me how to go about developing and validating selection instruments. This background has given me a good base for evaluating the work of others, as well as a true appreciation for those who consistently get it right.
I believe the work done by Bruce Davey and IPMA-HR is an example of getting it right. As indicated by the CO-BDQ Technical Report, the development and validation of such an instrument is similar to the development and validation of a written exam. In its simplest form, this means that there is a job analysis conducted. A job analysis includes the identification of tasks performed; the identification of related knowledge, skills, and abilities necessary; identification of how they can be measured; and finally development of the measurement instruments.
In the case of content validity, the process generally ends with trying out the exams and cleaning them up as indicated by item analysis and study of pass points and the potential for adverse impact. This would be where written exams and biodata instruments diverge in that other processes are necessary to support the validity of biodata instruments. The additional validation processes of the IPMA-HR/Davey study included a form of concurrent validation and predictive validation, both of which are types of criterion related validity.
As stated numerous times before, I believe that a thorough job analysis is the foundation of most human resources work — certainly all work involving selection. In the case of this study, the IPMA-HR staff completed a thorough job analysis that included on-site observations, interviews, and distribution of job analysis questionnaires to 19 correctional facilities in 15 states. From the information gathered, the staff identified 14 job categories, 122 tasks, and 12 ability areas. The complete job analysis is contained in the technical report for the CO-EL 101 examination (IPMA-HR 1991).
It could be argued that the trickiest part of utilizing biodata information is the creation of the questionnaire. The development of test items (aka “questions”) offers a special opportunity to conduct research and utilize creativity. In practice, such instruments combine features of application blanks and personality inventories while offering the potential to assess a wide range of applicant traits.
The CO-BDQ includes items requiring self-assessment of personal traits and abilities along with items related to interests and preferences. This provided the widest range of items having the potential to contribute to the validity of the final product. It also offered the opportunity to assess almost all of the traits identified in the job analysis phase. The researchers also made the decision to exclude items dealing with circumstances entirely beyond a candidate’s control as well as items related to criminal activity.
Ultimately, seven subtypes, or categories of biodata, were identified for inclusion in the CO-BDQ. These categories are outlined in detail in the CO-BDQ Technical Report. As indicated in the study, it was the goal of the consulting team to use the biodata questionnaire to evaluate areas that could not be effectively covered by a written ability test or, as I’ve been saying, a cognitive test.
Many of the test items included in the CO-BDQ were designed to assess integrity, interest patterns, perseverance, leadership, reliability and interpersonal functioning — items not covered in the written test. Again, this is the ideal situation: a jurisdiction can increase the predictive validity of its test battery by adding instruments that measure job-related traits not already being measured.
Several different sources were used in the development of test items for the CO-BDQ including: the Police Officer Background Data Questionnaire (PO-BDQ), developed by BDA for IPMA-HR (Davey and Jacobson, 1992); the book “A catalog of life history items” (Glennon, Albright and Owens, 1966) was used to generate ideas; and the job analysis data.
As mentioned previously, the most common form of criterion used in criterion validity studies is a job performance rating. Fortunately, IPMA-HR had already developed the Correctional Officer Performance Rating Form as part of their project to validate the C-1 cognitive ability test for correctional officers. The form was developed from the IPMA-HR job analysis and requires supervisors to rate correctional officers on 15 areas of performance.
Two types of criterion related validity are frequently found in criterion related validity studies: concurrent and predictive. In concurrent studies, the score on the first measure is compared to the second measure. In the case of the CO-BDQ, the first measure was the biodata instrument, and the second measure was the performance ratings of current employees. The process is highly technical and requires a great deal of analysis in order to establish a key for the biodata instrument that is an accurate reflection of the correlation between desired traits and job performance.
The same challenge exists when developing a key for scoring the biodata in a predictive study. Predictive studies compare candidates’ scores with their predicted future performance.
The CO-BDQ Technical Report goes into great detail regarding the processes used to develop the final scoring key. From the statistical comparisons that were made it was concluded that “the CO-BDQ has a significant relationship to both turnover and the performance scale in each of the two states studied. In addition, the key showed significant validity for both males and females, and for blacks and whites.”
The conclusion of the CO-BDQ Test Adverse Impact and Fairness study stated that “The most notable information from these Tables (reporting pass rates) is an almost complete lack of significant difference by race and sex in the study.”
Ultimately, resulting use of the biodata instrument CO-BDQ indicate that it correlates with job performance, predicts turnover, and does not have adverse impact. Certainly the addition of such an instrument to the selection battery of correctional officers would be well warranted.
