Improving the Interview Part 2a: The Panel or Board Interview

In the last blog, we investigated possible improvements that could be made in the use of individual interviews in pre-employment or promotional screening.  This month we expand our discussion to include the panel or board interview, an approach used by many public sector organizations.

As is often the case, once I start on a topic I have trouble controlling myself and my word count quickly gets out of control (my students have learned that if you ask me a simple question it can easily turn into an hour-long response). So, I have divided this blog into a 2a and 2b. In 2a, which you are reading right now, I:

• Delineate the major characteristics of the panel interview;
• Offer a version of a panel interview checklist;
• Discuss the need for structure and training;
• Provide an overview of the IPMA-HR Police Structured Interview System (POSIS).

Then, in a soon-to-follow Part 2b, I will answer frequently asked or encountered questions regarding the panel interview.

Characteristics of the Panel Interview

The distinguishing feature of the panel interview is the simultaneous involvement of multiple interviewers, usually three to five subject matter experts, instead of a single interviewer.  Normally, the panel sits at a large table facing the lone interviewee. The panel asks questions of the interviewee and also scores or rates the interview.  Although in some cases there may be only one panel, for large test administrations there will be multiple panels, each composed of multiple interviewers, with interviewees assigned to different panels.

For example, consider a police entry test with 30 candidates. In such a situation, we could have one panel of 3 raters who meet with each of the 30 candidates. This would result in a substantial burden on our interviewers.

As an alternative, we might decide to have 9 expert raters assigned to 3 panels. Panel 1 would consist of rater A, B, and C. Panel 2 would consist of raters D, E, and F. Finally, Panel 3 would consist of raters G, H, and I.

Panel 1 would interview and score candidates 1-10. Panel 2 would interview and score candidates 11-20. Panel 3 would interview and score candidates 21-30.

There are of course advantages and disadvantages to the use of a single panel versus multiple boards, which we will discuss at a later point in this blog.

Steps in the Panel Interview Checklist

For your convenience, we have summarized the steps in preparing for and conducting a panel interview in Table 1.

Table 1. Panel Interview Checklist

Point of Emphasis: Structure and Training     

Before we start, it is important to note that our major points, or recommended practices, from the blog dealing with the individual interview apply as well to the panel interview. That is, the gremlin in the mix is the degree of structure. In general, the greater the structure, the more reliable and valid the panel interview will be. Reliability and validity are, of course, a positive attribute and one of our main goals in designing our assessments. Problems tend to occur when there is a lack of structure, leading to a lack of consistency across panels and across candidates. Unfortunately, many agencies use hastily put together interview questions with little thought to achieving consistency across raters or panels.

Structure is an important characteristic in both the questions asked and the evaluation of candidates. Thus, as with the individual interview, we would recommend:

• The use of a pre-set, consistent list of behaviorally-based and job-related questions.
• The use of predetermined follow-up questions or prompts.
• Behaviorally-based and anchored rating scales in order to score the interview (often referred to as BARS).
• Discriminatory, weird, and random questions should be avoided.

In addition, adequate training of the interviewers or raters is critical. Interviewers should be trained to observe behaviors, identify desired behaviors, and use scoring anchors.

This is especially true if there are multiple panels, in which case there needs to be a consistent understanding of the application of the rating scales. In other words, the panels must be trained to use the rating scales in a similar manner achieving a common frame of reference in ratings.

A System That Meets the Need for Structure and Training: the IPMA-HR Assessment Systems POSIS 

As an example of a system that provides structure and training, we will turn to a description of (POSIS), which was recently developed by IPMA-HR Assessment Services in order to meet the demands for an effective, standardized panel interview for entry level police officers. Based on extensive research, including studies with nearly 1,000 candidates, the POSIS is very reliable and valid. With a high rate of consistency across multiple interviewers and panels, the POSIS system has a remarkably high degree of acceptance from agency administrators, examiners and candidates.

The POSIS is worthy of our attention as it illustrates how jurisdictions can incorporate a reliable, valid panel interview into the assessment process. The POSIS was developed specifically for entry level police; however, the principles underlying the associated methodology can be generalized to any job, whether entry level or promotional. This innovative assessment system makes it easy for the user to plan the oral board as well as provide training to the panel members.

The POSIS system includes the following components:

• A comprehensive Administrator’s Manual including details on how to plan, administer and score the process.
• All the supporting materials and forms needed to administer a structured and successful process.
• A detailed Panelist Trainer’s Manual, which provides materials for training panelists on the interview content and the scoring process.
• For the purpose of providing training practice, a training PowerPoint including videotaped “candidates” answering each of the interview questions; such an approach helps raters in standardizing their understanding of the rating process and in achieving consistency across the panel.
• Panelist booklets with questions, scoring guidance, and easy to use benchmarks and rating scales to make scoring standardized.
• An excel spreadsheet for calculating final scores.

Features of the POSIS include:

• An easy to use, structured system, offering insightful results assessing abilities that have been shown, through job analysis and other validation research, to be important for police officers in all regions of the country and in departments of all sizes.
• A process that was carefully developed to promote job-related, fair, highly reliable and consistent interviews for police officer candidates.
• A documented, structured interview system, which attains an extremely high level of candidate acceptance.
• A structured training component that creates a common frame of reference for interviewers and leads to consistent ratings across panels.

User reactions to the POSIS have been extremely positive. For example, the Indianapolis Metropolitan Police Department commented that “Our department is very happy to have discovered POSIS. The training and instructions are clear and concise, and every form you could possibly need is supplied. Part of the training is a series of videos of mock candidates, which were particularly helpful to us. They provided our panel members with an in-depth understanding of each interview question and how to score in a consistent way — a very important feature given that we use several different panels in our interview process. POSIS is a welcome addition to our assessment process; it really helped in our selection of the right candidates. We’ll definitely be using it again!”

Next Blog Post

In the next blog post, I will answer what I see as some frequently encountered questions regarding the panel interview including:

• Who should be on the panel?
• How should I combine ratings to arrive at a final score?
• Record retention
• Timing

See you then.

Notes

I would like to thank Steven Tseng for his assistance in the preparation of this blog. In particular Steven assisted with the literature review. If you are interested in a list of references, please contact Dennis Doverspike. A literature search was conducted on Google Scholar, PsycInfo, and PsycNet using combinations of the terms board, panel, job, employment, and interview. Panel interviews also are sometimes referred to as board interviews. Not surprisingly very few articles were identified that dealt specifically with the interview.

As always, I welcome your comments, questions or criticisms.  There should be a reply box located at the end of the blog. Feel free to email me with comments or suggestions at dennisdoverspike@gmail.com.

I would also like to remind you that I will be offering Conducting a Job Analysis, a 1-day course at the IPMA-HR International Training Conference on Sunday, September 18th in Kansas City.  For more information on the conference, please click here.