Hello 2018: The End of Year Blog

As is my tradition, this is my end-of- year blog, an annual optimistic look forward. Usually, I concentrate on trends in assessment. Out of a large number of possibilities, I have narrowed it down to:

  • We are a Growth Industry
  • Professionalization
  • The Candidate Experience

Growth Industry

The year 2017 was the year of Industrial-Organizational (I-O) Psychology. Newspaper articles declared I-O Psychology a “hot” job for 2018, a fastest growing job, and a high paying job. Although there are other subfields subsumed under the I-O Psychology label, most of this enthusiasm has been fueled by the growth in the demand for assessment professionals. Diving down further, we can trace our “hotness” to the explosive expansion of unproctored internet-based testing, the gamification of assessments, and a general interest in the application of emerging technologies to preemployment testing. Of course, fame can be fleeting, and predictions often wrong, but as for now at least the future’s so bright, I gotta wear shades. Continue reading

Joint Agency Police Promotional Assessment Centers: An Opportunity for Interagency Cooperation

The current fiscal environment for many law enforcement agencies requires innovative change in practices and procedures.  While we may not be facing the stark challenges of retrenchment budgeting that were common in the 1980s, today many of us have insufficient budgets to achieve all that we seek to accomplish. A joint agency promotional assessment center may enable multiple agencies to realize the benefits of the assessment center at a substantial cost savings.

Background Brief

Human Resource Management costs are significant in public policing.  As a rule of thumb 70-90% of any law enforcement agency’s budget is committed to some aspect of Human Resource Management (HRM).  These costs have been predicted to grow in the future as a result of any number of factors.  But a common factor in any HRM cost equation is the identification, selection and retention of competent and capable supervisors and managers.

Across the nation, several law enforcement agencies have adopted the promotional assessment center as a defensible and fair tool in the selection of supervisors and managers.  However, as many readers know only too well – a functional promotional assessment is an expensive tool[i].   The costs present in any one of several forms – including the significant time commitment of both the candidates and assessors; the logistical support required for a successful center (video recording, appropriate exercise spaces, and scheduling trained actors) as well as the extensive and intensive planning required for a successful assessment center process. Continue reading

Participants Needed for Entry-Level Firefighter Test Study (Discounts Available!)

IPMA-HR is in the final stretch of finishing up the validation study on our entry-level firefighter (FF-EL) test series, but we still need your help!  We are seeking a few more agencies who have administered a test from the FF-EL series to provide us the scores received by firefighters who were selected for the job.

Once we’ve received these test scores from you, we ask that each firefighter’s supervisor complete a brief 9-question performance evaluation.  This will allow us to see whether our tests from the FF-EL series continue to be successful in predicting on-the-job performance.

For your help, you will receive $100 off your agency’s next IPMA-HR assessment order.  In addition, your participation in this project will provide direct evidence supporting the validity of these exams and their use in your agency.

If you would like to participate, please visit the following link to provide your test scores:

https://www.surveymonkey.com/r/FFELScores

Our Research Associate, Julia Hind-Smith, will then follow up with you to provide further instructions on completing the performance evaluations.  If you have any questions about the study, please feel free to reach out to Julia at jsmith@ipma-hr.org.

(Please Note: All information gathered for this study will remain strictly confidential.  The data gathered will be combined with data from other departments and only be reported in the form of group statistics.)

Should We Be Gamifying Our Assessments?

For this month’s blog, I will look at what is one of the hottest questions in private and public-sector personnel selection – “Should we be gamifying our assessments?” In my opinion, the answer is “Yes!” and I will take this blog to explain why.

“Gamification” falls within the larger topic of “how should I incorporate emerging technologies into my assessment strategies.” Now, one might legitimately ask how it is that someone who started doing math on a slide rule can claim to be an expert on emerging technologies. I will simply remind you that Steve Jobs, Steve Wozniak, Paul Allen, Bill Gates, and I were born at roughly the same time. So, despite huge differences in our net worth, we do share a similar generational zeitgeist.

What Does Gamifying Mean?

Gamifying is one type of Technology Enhanced Assessment (TEA). Related types of TEA include:

  • games (so obviously there is a difference between games and gamifying).
  • enhanced item types.
  • the use of avatars.
  • virtual reality.
  • big data and advanced algorithms.

Gamification (or gamifying) is defined as “the application of game mechanics, elements, and features to non-game environments,” or in this case “the application of game-type elements to assessments used in personnel selection.” This differs from the use of true games in selection, although the difference is probably more of a continuum than a sharp distinction, as both games and gamification can be used in personnel selection. The differences between games and gamification can be summarized as:

  • With games, the person knows they are playing a game, whereas with gamification, the applicant still knows they are taking a test.
  • Games are meant to be fun and are structured to have clear rules that define the game play; gamified tests are seen as a more serious activity.
  • Games have different play sections and winning is the goal; gamified tests are structured similar to traditional assessments and getting hired is the goal.

Continue reading

Participants Needed for Office of Fire Marshal Study

IPMA-HR is currently seeking participants for a nationwide Office of Fire Marshal/Office of Fire Prevention study. This study is the first step in the development of a new Fire Marshal Test. The first part of the study involves surveying current members of the Office of Fire Prevention to learn about the important duties and demands of their job.

Examples of applicable positions include, but are not limited to: Fire Marshal, Fire/Arson Investigator, Premise Officer, Deputy Fire Marshal, Fire Prevention Officer, Fire Inspector, and Code Enforcement Officer.

If you or someone you know holds one of the above positions or a similar position, we would greatly value your input in developing our newest test series.

Participants will be entered into a raffle with a $500 prize! Participating agencies will also receive a 15% discount toward a future IPMA-HR assessment product purchase. Interested parties can use the following link to participate:

 https://www.surveymonkey.com/r/FireMarshal

Questions can be emailed to our Research Associate, Julia Hind-Smith, at jsmith@ipma-hr.org.

Test Response Data Collection Form Update

We have recently made several changes to the layout of the blue Test Response Data Collection Form that is included with every test order.

What’s changed?

To more efficiently collect data for test validation studies, we have updated our Test Response Data Form to not only collect score and demographic data, but also collect candidate identification data (e.g. candidate name, candidate ID number). Continue reading

Are Tests Valid for Public Safety Jobs?

Number 4 in the Validity of Public Safety Assessments Series

The idea for this primer series germinated from a simple question – “Could you do an article looking at the validity of tests used in public safety assessment?” In response, I decided to do a series of articles aimed to inform, but also designed to keep things simple. The blogs in this series were intended to cover:

  1. What are the characteristics of a good test?
  2. What are some authoritative references human resource and assessment professionals can rely upon in evaluating the worthiness of tests?
  3. What is validity?
  4. Finally, trying to address the original question I was asked, are public safety assessments valid?

The first three blogs in the primer series have been published and are available by clicking the links above.

This is the fourth and final article in the series and is intended to answer the question regarding the validity of tests for public safety jobs. I define public safety jobs here as including police, fire, and emergency medical services (EMS). In addition, human resource professionals are usually interested in the use of tests in both entry level screening and for arriving at promotion decisions. Continue reading

RFP for Entry-level Police Officer Selection Test Due Date Extended to September 15

The International Public Management Association for Human Resources (IPMA-HR) invites fixed-price proposals from qualified consulting firms to develop a multiple-choice entry-level police officer selection test for its stock testing service.

This includes the creation of an exam plan from the job analysis data collected by IPMA-HR, and the utilization of questions from IPMA-HR’s test item bank. The consulting firm chosen is expected to conduct a criterion-based validation study that includes having participating police departments administer the test to incumbents and gathering performance evaluation ratings to statistically show that the test predicts performance on the job. This also includes a test item analysis to select the final test questions, criterion validation statistics, and a fairness analysis.

The firm is expected to prepare validation study materials and assist in getting police departments to participate by making calls and sending correspondence. Further, the firm will prepare a technical report summarizing the steps taken during the test development and validation study project.

Proposals are due by September 15, 2017, 5:00 PM, ET.

Learn more about the full scope of the project and obtain contact information on our website. Continue reading

What is Validity?

Number 3 in the Validity of Public Safety Assessments Series

The idea for this primer series germinated from a simple question – “Could you do an article looking at the validity of tests used in public safety assessment?” In response, I decided to do a series of articles aimed to inform, but also designed to keep things simple. The blogs in this series are intended to cover:

  1. What are the characteristics of a good test?
  2. What are some authoritative references human resource and assessment professionals can rely upon in evaluating the worthiness of tests?
  3. What is validity?
  4. Are public safety assessments good tests and are they valid?

The first two blogs in the primer series has been published and are available by clicking the links above.

This is the third in the series and is intended to provide a basic introduction into the various kinds of validity evidence. By validity evidence, I do not mean the obvious distinction between the big four of:

  1. Content
  2. Criterion-related
  3. Construct
  4. Transfer or Transportability

Understanding the distinctions between the four types of validity listed above is important. However, in this blog, I mean something different by types of validity evidence. As our ultimate purpose or goal is to respond to the question as to whether tests are valid for purposes of public sector assessment, we can consider the following five types of validity evidence as relevant:

  1. Local Validation Based on Criterion-Related Evidence
  2. Validity Generalization Evidence Based on Tests in General
  3. Validity Generalization Evidence Based on Specific Occupation
  4. Validity Generalization Evidence Based on Specific Test
  5. Other

Continue reading