In this blog, I will respond to what I see as practical questions that often arise in planning for a panel interview. I do apologize for the delay in the production of this third, and final, blog on the interview. Unfortunately, at times, real life intervenes.
I started this series by noting that no other selection device is as ubiquitous as the interview, while at the same time as misunderstood. Then, in Part 1, I discussed the individual selection interview. In Part 2, I discussed panel interview, including the availability from IPMA-HR of a product known as the Police Officer Structured Interview System or POSIS.
This month, in the third and final blog, I respond to what I see as some frequently encountered questions regarding the panel interview including:
- Should I train raters?
- Who should be on the panel?
- How should I combine ratings to arrive at a final score?
- What type of records should I keep?
- How long should it take?
As a warning, a lot of my answers will involve a combination of “it depends” and “on your local rules or procedures.”
The Panel Interview
In case you are new to HR or need a refresher, the distinguishing feature of the panel interview is the simultaneous involvement of multiple interviewers, usually three to five subject matter experts, instead of a single interviewer. Normally, the panel sits at a large table facing the lone interviewee. The panel asks questions of the interviewee and also scores or rates the interview. In some cases, there may be only one panel; for large test administrations, there will be multiple panels, each composed of multiple interviewers, with interviewees assigned to different panels.
Should I Train Raters?
Yes, you should develop a training program for your examiners. If possible, the topics covered should include:
- The interview process, including how to ask follow-up questions.
- A thorough review of the rating scales and anchors.
- Rater error training.
Research suggests that even relatively short rater training programs can be effective. The length or time devoted to training will depend on the complexity of your process, amount of risk involved in the selection situation, and how much past experience with interviewing your examiners possess.
Who Should Be On The Panel?
Wow, now this is a tough question. This question really gets at a number of issues:
- Should all my examiners be subject matter experts (HR professionals or members of the occupation) or can I use city council members or community members? Normally, my preference is to use HR professionals, attorneys, or members of the occupation (i.e., police officers in police testing) as panel members. However, there may be instances where it is beneficial to use community members. Certainly, one situation would be when selecting for higher level ranks, such as Chief. Additional advantages of using community members are that it can increase the diversity of the panel and encourage a different perspective.
- Should my panel members know the candidates? This is not usually an issue in entry-level testing, although it can be. It is more likely to arise in promotional testing. The answer here would depend on what you want to accomplish. The advantage of having local examiners, who then know the candidate, would be that such members know whether the examinees responses are consistent with day-to-day job performance. Local examiners also understand the procedures and rules of the department. Perhaps even more critically, local examiners will know and understand the values and mission of the department and, thus, can evaluate fit. However, the advantages I listed can also be disadvantages. If you want the evaluation to be free of any contamination from outside job performance, then you need to use outside individuals as raters. If you want to maintain a certain degree of anonymity (i.e., your rules state that examiners should not even know the candidate’s name), then you will have to use individuals from outside your community for the interview panels.
This leads to a related question, should you reimburse or pay the panel members in some way? Any pay offered would not have to be in dollars, it could be a gift card or a nice meal after the day of rating. It may be difficult to recruit outside interviewers, if you do not compensate them in some way, at least for travel, hotel, and food. Of course, these days, compensating panel members can get very expensive, very quickly. I arrange a lot of professional examinations at conferences where a single night for a hotel runs $300 or more. So, covering the hotel for one or two nights for three examiners, can run between $1000 and $2000, without even considering meals, mileage, or flights.
How Should I Combine My Ratings To Arrive At A Final Score?
Normally, each of your examiners or panel members will independently rate each of the candidates. A rating will be provided for each test taker on each dimension or scale, and perhaps an overall evaluation as well. This rating can occur right after the interview or at the end of the day. I prefer to have the panel take initial notes and make initial ratings, but then do an updated independent rating at the end of the daily session.
Once each panel member has rated each candidate, the ratings have to be combined across panel members. There are several ways to do this:
- Mathematically combine the scores using some formula or algorithm.
- From a research perspective, this is probably the most valid procedure.
- Have examiners discuss the candidates and then have each examiner independently revise and update their ratings based on the discussion; then, mathematically combine the ratings.
- This combines the advantages of mathematical combination with discussion.
- As with consensus, you need to make sure that one powerful member of the panel does not exert too much influence over the other members.
- Require the examiners to reach consensus on the rating of each candidate.
- This is a very popular approach.
- There are some advantages in terms of record retention (see below).
- You need to be careful to ensure that one powerful member of the panel does not exert too much influence over the other members.
- It increases the time needed to make ratings.
- You will need to develop rules that cover what should be done when the panel cannot reach consensus.
What Type of Records Should I Keep?
The answer to the questions of records retention will depend upon your local rules and a discussion with your attorney. I usually advise jurisdictions to keep only the final ratings (i.e., the official scores), and treat all other rating sheets or notes from discussion as working drafts. It can be very difficult to explain why an examiner may have changed an initial rating of 1, to a final rating of 5, even if it reflected an initial confusion over the rating scale. For that reason, if possible, I believe it is best to only retain the final rating.
How Long Should It Take?
The length of each interview will tend to vary depending on the level in the organization. An interview for Fire Chief might take three hours, while a panel for an entry-level firefighter might only require a half hour.
The time required for a panel interview will also be a function of the number of questions asked; the more questions you ask the more time it will take. For a typical promotional interview, I usually estimate about:
- One (1) minute or less to ask the question and have the candidate begin a response.
- Five (5) minutes for the initial response.
- Four (4) minutes for follow-up questions.
This adds up to ten (10) minutes per candidate. A survey I did of assessment professionals suggested that many allow closer to five (5) minutes per question. A little quick math tells us that in a fifty (50) minute interview, we could ask approximately five (5) to nine (9) questions. In addition to questions, you need to allow some time in the beginning for introductions and an explanation of the process, and at the end for any questions and final instructions for the examinee.
Why do I say a 50-minute interview? Well, you need to allow time for escorting the candidate into the room, escorting the candidate out of the room, initial ratings, note taking, and bathroom breaks for your panel.
Remember, you also have to allow time for training and rating. So, a typical day of interviewing might look something like:
8 to 9 Training
9 to 12 Three (3) interviews, fifty (50) minutes each.
12 to 1 Lunch
1 to 4 Three (3) interviews, fifty (50) minutes each.
4 to 5 Rating.
Good luck with your interviews. When well done, the interview can provide valuable information, which is extremely difficult to obtain from any other selection device.
I continue to welcome your comments, questions or criticisms. There should be a reply box located at the end of this blog. You can also email with comments or suggestions at firstname.lastname@example.org.
Remember, IPMA-HR Assessment Services offers interview systems for public safety positions and for the public sector. This includes the Police Officer Structured Interview System (POSIS)
I would like to thank Steven Tseng for his assistance in the preparation of this blog. In particular, Steven assisted with the literature review. If you are interested in a list of references, please contact Dennis Doverspike. A literature search was conducted on Google Scholar, PsycInfo, and PsycNet using combinations of the terms board, panel, job, employment, and interview. Panel interviews also are sometimes referred to as board interviews. Not surprisingly very few articles were identified that dealt specifically with the interview.