Robert Burd on the Assessment Services Review

Maximizing the Effectiveness of Your Applicant Screening Process

Third and final part of a three-part series on the topic of Validating Minimum Qualifications.

The two previous articles focused on developing accurate minimum qualifications that were supported by the content validity model. Today, we will introduce the concept of externally imposed minimum qualifications and then touch on the other components of the application screening process that are necessary to ensure it has the desired impact on this critical component of the selection system and, ultimately, the success of the organization.

Quite often in human resources, the focus shifts from developing procedures that are valid and reliable to developing systems that can withstand legal challenges. The paradox is that if systems are valid and reliable they can withstand legal scrutiny and, more importantly, they become effective tools for selecting the best possible work force available.

While we often focus on internally imposed and established minimum qualifications, it is also important to remember that most public entities have a number of minimum qualifications that are imposed on them by entities outside of human resources. Typically, these outside sources come in the form of laws. For example: in order to become a peace officer in the state of Nevada, an individual has to be a citizen of the United States and at least 21 years of age. There is no way to get around these requirements, so it is important to include them in the minimum qualifications. (more…)

By Robert Burd|2012-12-10T13:07:49-04:00August 28th, 2012|Assessment, Validating Minimum Qualifications|Comments Off

Supporting Internally Developed Minimum Qualifications

Part two of a three-part series on the topic of Validating Minimum Qualifications.

The previous article focused on the content validity model as a tool for developing and supporting minimum qualifications. That discussion focused primarily on establishing minimal levels of education, training and experience. However; it is important to recognize that there are other requirements that go beyond education and experience requirements such licenses, certificates, age, and U. S. citizenship. Many of the additional requirements placed on job classes are established by law and therefore are externally imposed, which distinguishes them from qualifications that are internally developed and imposed

This article will focus on support for internally developed and imposed minimum qualifications. The next article in the series will put together all the components of the application screening process to maximize its effectiveness which goes hand in hand with its reliability and validity. (more…)

By Robert Burd|2012-12-10T13:07:49-04:00August 21st, 2012|Assessment, Validating Minimum Qualifications|Comments Off

Utilize a Content Validity Strategy to Establish Minimum Qualifications

Part one of a three-part series on the topic of Validating Minimum Qualifications.

Screening applications is the single most common human resources activity performed by all entities, public and private, that are involved in selecting employees. This process is typically not considered a test in the common use of the word, but it is. Since it is part of the selection process, it is required by the Uniform Guidelines for Employee Selection Procedures (UGESP 1978) to be both valid and reliable.

Just as the job analysis serves as the foundation for most of the activities related to test development and classification and compensation, the systematic analysis of the job should also be used to write the actual class specification which would include the minimum qualifications. Class specifications should be the standard for documenting the tasks to be performed on the job along with the knowledge, skills and abilities necessary to perform those tasks. In addition, class specifications should include a summary statement of how the minimum knowledge, skills, and abilities may be obtained. In that regard, since the class specification flows from the job analysis, the document should be a condensed version of the information obtained in the job analysis. Similar to the job analysis, the class specification should reflect a flow from the work performed (tasks) to the knowledge, skills, abilities, and personal characteristics (KSAP’s) necessary to perform the job at entry and the education and experience necessary to obtain those entry-level KSAP’s. How the KSAP’s may be obtained, stated in terms of education, training, and experience, becomes the minimum qualifications requirement. (more…)

By Robert Burd|2012-12-10T13:07:50-04:00August 8th, 2012|Assessment, Validating Minimum Qualifications|1 Comment

Test Standardization

Along with establishing tests under guidelines that ensure their validity and reliability, it is important to administer tests in a consistent manner across candidates to avoid diminishing their validity and reliability. In other words, the conditions under which a test is given should be standardized. Standardization of test administration can mean many things to different people and can vary from jurisdiction to jurisdiction, but the key is making sure that all candidates are treated as consistently as possible. In my experience, I have seen jurisdictions err in both directions, with some being too lax and some being too strict. The key is to avoid changes in the way a test is administered that could actually negatively impact candidates’ ability to perform at their optimum.

In that regard, it is also important to distinguish the concept of standardizing the manner in which a test is administered from what candidates often term, “fairness.” It is not uncommon for test takers to confuse what they consider to be “fairness” with what test administrators know to be standardization since test takers are unfamiliar with what actually constitutes standardization. While the two concepts are related, they are not the same. Fairness is a concept that exists in an individual’s mind and can often include obscure factors that have nothing to do with standardization. It is important for test administrators to understand factors related to standardization to avoid getting dragged into debates about the “fairness” of a test. (more…)

By Robert Burd|2012-12-10T13:08:07-04:00June 27th, 2012|Assessment, Test Standardization|Comments Off

Test Security and Cheating: Part 2

As indicated in the first article on this topic, a great deal can be learned from cheating schemes perpetrated on other agencies. This learning pursues two avenues, with the first being what the jurisdiction itself learned, what it did in response to minimize the damage and prevent further incidences in the future. The second avenue provides the path that you as an individual can take in terms of “brain storming” how to prevent such an occurrence within your own jurisdiction. Beyond that process, we should rely on the cumulative information available in the HR profession that is designed to provide test security and prevent cheating as the foundation for an anti cheating and test security program.

Test security agreements such as those developed by IPMA-HR and an agencies civil service rules provide a good starting blue print for establishing an effective test security/anti cheating plan. While these documents usually represent the best thinking of those who developed them, they should not be seen as exhaustive. So in addition to incorporating these guides in your own anti cheating plan, you can also benefit by using them as a basis for brain storming additional ideas that can deter cheating. In that regard, to develop your anti cheating program it is important to systematically look at the points in the test development, administration, and maintenance process that may be vulnerable. The other half of developing a sound program is to look at the other side as well in regard to who might be motivated to cheat. Dissecting the test process and potential cheaters provides a systematic approach to developing an effective preventative program. (more…)

By Robert Burd|2012-12-10T13:08:33-04:00May 23rd, 2012|Assessment, Test Security & Cheating|1 Comment

Test Security and Cheating: Part 1

If your jurisdiction has ever been the target of a cheating scheme or scandal then you are very familiar with the costs of cheating. Writing new tests, conducting new recruitments and administering new selection procedures are time-consuming and costly, yet they represent only a portion of what could be considered the costs of cheating. Perhaps the biggest potential costs of cheating are those incidents where cheating goes undetected, resulting in incorrect selections for promotions or filling entry-level positions. Whatever the costs and whatever the long-term impacts may be, the sad truth is that cheating still occurs.

A cheating incident from 2009 underscores the impact of cheating scandals on agencies and should serve as a warning for individuals with the responsibility of preventing cheating, as well as any individual that may consider cheating. In this case, a member of the testing committee provided test questions to a lieutenant who subsequently provided them to an officer studying for the test. That officer went to internal affairs and an investigation was launched. In addition to scrapping the test, four police officials were temporarily removed from their duties pending the conclusion of the investigation. Two eventually left the department. The cloud of suspicion and distrust continues to hang over the department and the ultimate negative affects are incalculable. (more…)

By Robert Burd|2012-12-10T13:08:33-04:00May 16th, 2012|Assessment, Test Security & Cheating|2 Comments

Successive Hurdles, Test Weighting and Certification Rules: Part 4

The articles in this series when taken as a whole present a picture of the challenges and potential pitfalls presented in the development of effective selection instruments and test batteries. In addition to the need to make sure instruments are reliable and valid so that they support the selection of the best available work force, they must also withstand legal scrutiny. Unfortunately, experience has shown that local laws, statutes and/or civil service rules that provide the blue print for how HR work is to be done are many times in conflict with exam development and validation procedures. In particular, certification rules that dictate the number of candidates from a ranked list that can be certified for a hiring authority to consider for selection can be responsible for undoing the efforts made to conform to professional standards.

Many individuals tasked with writing civil service rules, particularly in the infancy of the development of merit systems, did not have the benefit of possessing a test development and statistical background. Many systems focused on fairness and avoiding abuses of differing forms of the spoils system or the good ol’ boy system, but they did not take into consideration statistical concepts related to test scores, and in particular whether or not meaningful differences existed between scores. Sometimes, certification rules narrowly defined the group eligible for certification and in other instances; rules were modified in an attempt to address equal employment issues. These modifications often took the form of certification of the whole list which meant the hiring agency could select anyone on the entire list to put through the final selection interviews. (more…)

By Robert Burd|2012-12-10T13:09:07-04:00April 25th, 2012|Assessment, Successive Hurdles, Test Weighting & Certification Rules|Comments Off

Successive Hurdles, Test Weighting and Certification Rules: Part 3

In the last article we focused on weighing the tests and subtests that comprise the total selection process. We identified some instruments that should only be used as pass fail and we identified others that can be used to rank candidates. Those tests and subtests suitable for ranking are those identified through the job analysis as assisting in differentiating potential job performance. We also identified an issue with weighting tests and sub tests if we rely on simply multiplying test results by the percentage we want them to weigh in our total. Tests with greater variance tend to impact ranking more than the desired weight. Simply put, tests tend to self weight based on their variance.

Given a simple illustration we can see that tests that spread test scores out (have greater variance) will have a greater impact on the final ranking of candidates than tests that tend to lump everyone together (have less variance). Taking this concept to its extreme, it can be seen that if a group of five people all got the same score on a multiple-choice exam but achieved widely divergent scores on a structured interview, the multiple-choice exam would weight zero in our final ranking and the interview would weigh one hundred percent. (more…)

By Robert Burd|2012-12-10T13:09:07-04:00April 18th, 2012|Assessment, Successive Hurdles, Test Weighting & Certification Rules|Comments Off

Successive Hurdles, Test Weighting and Certification Rules: Part 2

In the previous article, I introduced the concept of weighting exams that comprise the battery of instruments in a selection process. This article will explore that process more in depth. To begin with, some instruments lend themselves to being weighted and thus providing an impact on the final ranking of candidates and others do not. Determination of which instruments are appropriate for ranking and the weight given to those that are considered appropriate for that purpose should be established through the use of a comprehensive job analysis designed to support the content validity model for test development.

There are numerous published methodologies for conducting job analyses that are designed to comply with the Uniform Guidelines for Employee Selection Procedures (UGESP) even though they may differ in how they combine subject matter experts’ ratings on KSAP’s which ultimately determine the weight given to selection components. Typically, these systems will collect ratings on KSAP’s and then review them to determine which ones have received ratings that indicate that they are required at time of hire, are important for job success and are linked to performing important job tasks effectively. Often to make the system more manageable, the next step will involve grouping KSAP’s into domains, which is what is recommended by several job analyses procedures designed to conform with the requirements of the UGESP. (more…)

By Robert Burd|2012-12-10T13:09:07-04:00April 11th, 2012|Assessment, Successive Hurdles, Test Weighting & Certification Rules|Comments Off

Successive Hurdles, Test Weighting and Certification Rules: Part 1

Medical doctors and psychologists rarely rely on the results of one clinical test when making diagnoses. Similarly, selection experts recommend using a battery of selection instruments when making employee selections. The people in these professions realize that the accuracy and reliability of their conclusions are greatly enhanced when they have a broader range of information on those being evaluated.

In selection it is often critical to measure quite divergent knowledge, skills, and abilities, which necessitates the use of multiple selection instruments as part of a battery that comprises the selection process. Most jobs require cognitive abilities and some require a body of knowledge, which in many cases can be measured by a written exam. In addition, most jobs require some degree of ability to communicate verbally. Since written tests can not measure verbal communication, a second test, usually a structured interview is necessary to measure whether or not a candidate possess the verbal abilities required for the target job.

In addition to measuring these abilities in candidates, many positions require additional abilities which require the use of additional selection instruments. Many classes like police officer, fire fighter, corrections officer and park ranger require the measurement of candidates’ physical abilities, psychological stability, medical fitness and suitability of background. To utilize these instruments effectively and efficiently, they must be combined in a manner that provides the greatest support for administration of the selection process and maximization of each instrument’s validity. Combining the information from multiple instruments is where the employee selection model differs from the medical model. (more…)

By Robert Burd|2012-12-10T13:09:07-04:00April 4th, 2012|Assessment, Successive Hurdles, Test Weighting & Certification Rules|Comments Off

About Robert Burd