Well, not the only question. In this blog, we will consider a series of questions including:

  1. If someone takes the test again, will there score change?
  2. If a group of individuals are retested, will the rank-order of the scores change?
  3. How much time should there be between retests?
  4. How many retests should be allowed?
  5. Which test scores are the most valid for predicting performance?

My answers will be based primarily on the research literature.  However, retesting is one of those topics where the importance of the question to practitioners has far outpaced the quantity and applicability of the published research literature.

As with many of my blogs, our discussion will be divided up into two parts.  In Part 1, we will investigate what we know about the answers to our fundamental questions.  Part 2 will deal with the more difficult issue of how we can integrate professional and government guidelines, the public sector testing model, and the research findings in order to come up with recommendations for applied practice.

If someone takes the test again, will there score change?

The unpleasant truth is that of course test scores will change if a test is taken again.  No test is perfectly reliable and, therefore, across repeated test administrations we would expect to see a normal distribution of observed scores.  The question might be rephrased as “should we expect different results from what we would see due to simple unreliability?”  The answer would again be “yes.”

The research is fairly definitive in demonstrating that scores go up over readministrations of a test.  Thus, each time a person takes a test we would expect their score to go up.  Now, how much of an increase we should expect will vary as a function of the type of test, the time interval between retesting, the importance of the testing situation, the initial score obtained by the test taker, and whether the candidate passed the first time. Score increases may even vary as a function of sex and ethnicity.

Regardless of the specific details, test scores do tend to show improvement with retesting and the difference may be as much as half a standard deviation, or 5 percentage points on a typical test.  The scores will continue to go up on each subsequent test.  Some of this increase may be due to motivation, some of it may be due to memory, and some of it may be due to real changes in the construct of interest.

If a group of individuals are retested, will the rank-order of the scores change?

We know now that if we retest one individual, in all likelihood his/her score will be higher on the second administration.  If we retest a group of individuals, their mean score will increase.  But will the rank order of the individuals change on the second test; after all, everyone’s scores could increase but the rank ordering of individuals could stay the same.  Tests are not perfectly reliable, so we would expect some changes in the rank order, but are the changes greater than we should expect?

The answer is that the rank order of individuals will change over time.  Over extremely short time intervals, less than a week, the change in rank ordering may actually be less than expected.  However, after about two weeks we would expect changes in the rank order of individuals with additional time between the initial test and the retest or with additional retests.

How much time should there be between retests?

Great question, with the answer depending upon the stability of the construct.  As a general and very liberal rule, a minimum of two weeks should be allowed between retests.  For less than two weeks, there is too much of a memory effect.  However, from a practical perspective, two weeks is too short of a time period.  Recommendations of as much as one or two years can be found in the literature.  Based on simply personal opinion, I would argue for six months.

Time may not be the only question though.  An additional issue is whether individuals should be required to participate in some type of remedial activity before attempting a retest; an option that I endorse.  For example, in physical ability testing, the failing candidate may be instructed to engage in a program of physical training before being allowed to retake the test.  An applicant who fails a math test may be counseled to complete a basic math course.

How many retests should be allowed?

You make the decision to allow a retest, but how many retests? Should a candidate be allowed to take one retest, two retests, or unlimited retests?  I will not answer your question right now, but I can tell you an individual’s scores tend to stabilize with additional retests.  That is, over additional tests, the gain from retesting starts to reach an asymptote.  In addition, being willing to take a test over and over could be seen as an important behavioral indicator in its own right, as an indicator that the person really wants the job and will work hard to succeed if they are hired.  So, it could be argued that if you are going to allow one retest, then you could allow as many retests as the candidate is willing to endure.

Which test scores are the most valid for predicting performance?

We have previously seen that if we give a test to a group two or three times, the rank order of individuals will change over the repeated testing.  Will the results of one of the retests be more valid than the others?  I do not think there is a definitive answer in the literature to this question.  My reply would be that the initial test administration would be the most valid unless anxiety or some other significant environmental variable influences the scores on the original test administration.  On the other hand, if we expect change or instability in the construct being tested, then I would expect the most recent administration to be the most valid.  Except, in that case, I would ask you why you are selecting for a job based on a construct that you know is easily changed and unstable?

Conclusion

In this month’s blog we have looked at what the research literature can contribute to our knowledge of how to deal with the retesting of applicants. Nutshelling our answers:

  1. If someone takes a test again, his/her score will increase.
  2. If a group of individuals are retested, the rank-order will change.
  3. At least two months, but more realistically 6 months to a year should be required between most retests.
  4. Given a candidate is willing, there seems to be no reason to limit retests. The issues are really whether to even allow a first retest and time between retests.
  5. Under typical situations, where only a portion of the applicants may be taking the test a second time, the first administration will probably be the most valid; but there are many factors that may influence this conclusion.

Next month, we will apply our research results to making practical suggestions regarding practice.  In particular, we will investigate how we can integrate professional and government guidelines, the public sector testing model, and the research findings in order to come up with recommendations for applied practice.  This will include a discussion of how we should determine a score for someone who is retested.  Should we take the first score and if so, why allow the retest?  An average of the scores?  The highest score?  The last, is this fair to others who only take the test once?

Hope you join us next month.

I would like to thank Winfred Arthur, Jr., for his assistance in drafting this month’s blog.  Winfred, who is a full professor at Texas A&M University, has been doing some very informative research on whether a memory or learning explanation best explains retest effects.  I would also like to thank the students in my graduate personnel selection class at The University of Akron, who assisted me with the review of the research.