About Robert Burd

Robert is an HR professional with over thirty years of experience in public sector human resources specializing in selection including: test development and validation, designing and conducting oral boards, assessment centers and physical fitness testing.

It Pays to Take A Look: Item Analysis Part 3

In the two previous articles, we looked at the statistical and technical aspects of item analysis. Individual test developers will view the statistical computations and their value differently based upon their knowledge of statistics and their understanding of their application. However, a test developer or test user with a rudimentary understanding of item analysis can still make accurate decisions regarding the effectiveness of test items and therefore, written exams. As we emphasized previously, IPMA-HR conducts item analyses on potential test items in their test development process and maintains item analysis data from successive administrations of all exams. These practices ensure that only items that perform well continue to be utilized and, in addition, this practice reflects a standard that all test developers should employ. Also note that for our discussion, we will be focusing on typical four response multiple choice items and true false items.

Effective utilization of item analysis information for item and test revision is where science meets art. This process of “cleaning” up test items and tests requires utilization of the basic information from an item analysis, effective analysis of the applicant response data and application of the information available for developing good test items. There is an extensive amount of scholarly information available on item writing as well as response theory and the effective practitioner should take the time to review some of this information prior to writing or “repairing” test items. It should also be noted that even though the information provided in this article focuses on actual test developers, it can also be extremely valuable for those who purchase or lease tests since it can assist them in evaluating the quality of tests they are considering. (more…)

By |2012-12-10T13:09:33-04:00February 29th, 2012|Assessment, Item Analysis in Public Safety|Comments Off on It Pays to Take A Look: Item Analysis Part 3

It Pays to Take A Look: Item Analysis Part 2

In the previous article, we began our discussion of the valuable information available through conducting an item analysis and we focused on the two most readily available pieces of information. First, of course, is the Difficulty Index. Just as the name implies, this index is an indicator of the difficulty of the item. It is expressed as a percentage and reflects the number of candidates that got that item right out of the total number of candidates that responded to the item. That is if nine out of ten respondents answered an item correctly the index would be .9 or 90%. From this illustration, we can also see that the Index actually has an inverse relationship with the difficulty of the item. That is, the higher the index or the higher percentage the easier the item is.

The second Index we discussed was the Item Discrimination Index. Essentially, this index reflects how the candidates who performed best on the test responded to a specific item when compared to how the candidates who performed the worst on the test responded to that same item. The top 27% of test performers and the bottom 27% of test performers are used for calculating the Discrimination Index and it is expressed as a proportion or percentage of the number in the top group that answered the item correctly in relation to the number in the bottom group that answered the item correctly. (more…)

By |2012-12-10T13:09:33-04:00February 22nd, 2012|Assessment, Item Analysis in Public Safety|1 Comment

It Pays to Take A Look: Item Analysis Part 1

Any organization using written exams as part of their selection processes that doesn’t take the time to review the information provided by an item analysis is overlooking a treasure trove of information. Without performing an item analysis on a written exam and acting upon the information gained from such an analysis, a jurisdiction truly does not know how that exam is performing.

There are two fundamental concepts involved in test utility and they are test validity and reliability. Simply put, validity refers to whether or not a test measures what it is intended to measure and reliability refers to how consistently a test measures what it is intended to measure. Item performance directly impacts these two concepts. Test items are the basic element for gathering information about test participants. In order for a test to have the necessary validity and reliability to make it worth using, test items have to perform optimally and gather the best information possible as accurately as possible. If test developers do not utilize information available from an item analysis, that test developer has no idea how items are performing. If items are performing poorly then all other statistical analyses become less meaningful and the test is not suited for its intended purpose. Hiring decisions based on the use of the test are questionable since they become another layer of what is ultimately a house of cards that does not have an adequate foundation. In this case, the written exam has not played its role in terms of optimizing accurate selections and supporting the mission of the organization. (more…)

By |2012-12-10T13:09:33-04:00February 16th, 2012|Assessment, Item Analysis in Public Safety|1 Comment

Implementing and Maintaining Succession Planning in Public Safety

This is part three of a three part series on succession planning in public safety departments. Part 1 focused on the organizational review and design of a succession planning process. Part 2 focused on how selection systems play a part in succession planning. Part 3 focuses on implementation and maintenance.

In this series of articles we have been sharing information regarding the design, implementation, and maintenance of succession plans with our focus being on public safety organizations with a particular emphasis on the use of assessment centers to assist in selecting appropriate participants. We have also emphasized the necessity of determining the scope of the program and ensuring that the KSAP’s identified for measurement in the center cover the levels within the organization and provide diagnostic information regarding the strengths and weaknesses of all participants.

Creating and selecting the appropriate exercises for inclusion in the assessment center may require the assistance of a consultant if a sufficient level of expertise is not available within existing staff. Regardless of who does the work, it is critical that job related simulations that closely parallel the work to be performed are used to elicit the widest range of behaviors possible from candidates. Again, this is an area where people are tempted to put candidates on the spot or get tricky or clever and all these temptations need to be avoided. Whoever designs the center needs to include those elements that have been identified as constituting an actual assessment center, these generally include:

  • More than one assessment technique is used and at least one is a simulation.
  • More than one assessor is used and assessors receive extensive training.
  • The decision on who participates and receives training is based on a pooling of information.
  • Overall evaluation occurs after completion of the exercises.
  • All rating dimensions are created from a thorough analysis of the job
  • All techniques used relate back to the dimensions.

Administration of the assessment center along with the selection of candidates for inclusion would generally mark the implementation phase. Although, it should be noted that a good succession plan will most likely move back and forth between the phases since modifications should be made as information is gained. (more…)

By |2012-12-10T13:10:10-04:00November 2nd, 2011|Assessment, Succession Planning in Public Safety|Comments Off on Implementing and Maintaining Succession Planning in Public Safety

How Selection Systems Play A Part In Succession Planning

This is part two of a three part series on succession planning in public safety departments. Part 1 focused on the organizational review and design of a succession planning process. Part 2 focuses on how selection systems play a part in succession planning. Part 3 will focus on implementation and maintenance.

As emphasized in Part 1 of this series of articles, it is critical in succession planning, to identify the employee pipeline in terms of its source, entry level and end point. It was also established that while rather generic in general, this series of articles is written with public safety organizations in mind and is intended for their benefit. In my experience I haven’t worked with any public safety departments that give serious consideration to the promotional potential of entry level candidates and yet to be successful, succession planning, needs to recognize that this is where it starts.

Most public safety organizations have rather rigid promotional requirements that typically include a specified time in the grade occupied before being eligible to compete for vacancies in the next level. That is, police officers usually have to spend two to four years as an officer before they are eligible to test for or be considered for sergeant, sergeants typically have to be sergeants for two to four years before they can test for or be considered for lieutenant and so forth on up to the top position. Police departments vary, of course, in their time in grade requirements as well as their selection methodology; however, the requirements cited are typical. Other things, typical or relatively common within public safety organizations is that selections for lower levels are not made with higher levels in mind or with any consideration of candidates outside the organization, both of which are important factors impacting succession planning. (more…)

By |2012-12-10T13:10:10-04:00October 26th, 2011|Assessment, Succession Planning in Public Safety|Comments Off on How Selection Systems Play A Part In Succession Planning

Review and Design of A Succession Planning Process

This is the first part of a three part series on succession planning in public safety departments. Part 1 focuses on the organizational review and design of a succession planning process. Part 2 will focus on how selection systems play a part in succession planning and Part 3 will focus on implementation and maintenance.

Succession Planning is one of those tools that is talked about a lot and considered fashionable by most, but unfortunately, implemented by very few organizations. That in and of itself suggests that there are challenges in implementing and utilizing an effective succession plan. Essentially, succession planning is a systematic process of identifying the future talent needs of an organization and taking the appropriate and necessary steps to ensure that there is an internal applicant pool available to fill the vacancies created by the loss of talent.

From that simple view of the process, it can be seen that there are a number of critical components to designing, implementing and maintaining a viable succession plan. It is important to recognize too that the implementation portion of the plan includes the critical component of selecting participants. Through many years of experience in designing oral boards for employee selection, I had the opportunity to work with hundreds of subject matter experts that were typically managers and supervisors in the career area or job family for which the test was being designed, typically police and corrections sergeants, lieutenants and captains. It was common for those managers to skip the critical foundational design components of an oral board and jump directly to writing questions. My observations in the succession planning process indicates that a similar phenomenon occurs among managers who are eager to start naming participants for the program or listing positions they don’t believe they can accomplish their missions without rather than focusing on developing the program. In both instances, skipping the basics that form the fundamentals for a successful oral board or a successful succession program leaves the process without a foundation. (more…)

By |2012-12-10T13:10:10-04:00October 19th, 2011|Assessment, Succession Planning in Public Safety|2 Comments