As indicated in the first article on this topic, a great deal can be learned from cheating schemes perpetrated on other agencies. This learning pursues two avenues, with the first being what the jurisdiction itself learned, what it did in response to minimize the damage and prevent further incidences in the future. The second avenue provides the path that you as an individual can take in terms of “brain storming” how to prevent such an occurrence within your own jurisdiction. Beyond that process, we should rely on the cumulative information available in the HR profession that is designed to provide test security and prevent cheating as the foundation for an anti cheating and test security program.
Test security agreements such as those developed by IPMA-HR and an agencies civil service rules provide a good starting blue print for establishing an effective test security/anti cheating plan. While these documents usually represent the best thinking of those who developed them, they should not be seen as exhaustive. So in addition to incorporating these guides in your own anti cheating plan, you can also benefit by using them as a basis for brain storming additional ideas that can deter cheating. In that regard, to develop your anti cheating program it is important to systematically look at the points in the test development, administration, and maintenance process that may be vulnerable. The other half of developing a sound program is to look at the other side as well in regard to who might be motivated to cheat. Dissecting the test process and potential cheaters provides a systematic approach to developing an effective preventative program.
In the test development phase it is very important to limit the number of people working on the test as well as the number of people having access to the background material and documents under development. Test development frequently involves the use of subject matter experts to help in the job analysis process as well as the creation of the test itself. Every one of these people, out of necessity, will be given access to confidential information. Therefore, anyone that is selected to be involved in developing the test should be carefully screened and required to sign some type of test security agreement. Such agreements should include penalties that can be invoked if confidentiality is breached with related civil service rules being cited.
In addition, some agencies limit the extent that any one subject matter expert (SME) can be involved in the development of the test. That is, individuals involved in the job analysis process are excluded from participating in item writing or item review. Similarly, individuals who serve as SME’s for item writing are not permitted to participate in final test review or pass point development. In addition, some agencies go so far as taking measures to safeguard the identity of department members involved in test development including asking those individuals to not disclose the fact that they are involved in the process.
During test development as well as test administration, it is critical to be aware of technology and how it may be used to cheat. It is important to be sure that all test development materials are kept under lock and key with access to the materials being restricted to the individual responsible for developing the test. Since computers are used extensively in developing test materials, it is also important to be sure that the computers utilized are secure and not vulnerable to any form of hacking. In addition, the test materials should be kept in secure, password protected files with access to those files being limited.
All staff working on the test development project should be briefed on ways to keep the tests secure during the development phase including not leaving tests on their desks or up on their computer screens unless they are working on them. They should be told that the tests should always be under lock and key, should be in password protected computer files, should not be left unattended in a printer, etc. These types of reminders can help avert disaster.
Since post test analysis of test performance is typically accomplished utilizing computers, it is also important to ensure that all data related to analysis is secure. This is particularly critical if the test or any part of it will be utilized again. Most tests developed under the professional guidelines utilized by IPMA-HR and other professional test developers make test development time-consuming and expensive so tests created under this model are intended for multiple uses. Therefore any breach of security of these tests is extremely costly since it not only impacts current results, but future use of the test as well.
In addition, tests that are intended for multiple use should have alternate versions. Alternate versions of a test can be as elaborate as creating one or more completely different tests from the original job analysis data or as simple as rearranging the order of items or sub tests. Anyone who has made an effort to cheat that stemmed from memorizing the key in item order will be foiled when presented with a different version of the test with a different key. Alternate versions of tests can also be utilized during a single session and thus prevent one individual from copying from another since their items would not be in the same order. In addition to changing the order of items; individual items can be changed by changing the order of responses or simply moving the keyed response from one location to another. That is, instead of the correct or keyed answer being located at response “C” it is moved to “A.” Similarly, true false items can be changed to move them from a correct answer of false to a correct answer of true or vice versa.
In my experience, I was able to use two versions of a test to prevent a cheating plan that involved a group of individuals intent on creating their own key. Their plan was for each member in the group to do what he could to memorize the keyed answers for ten items on the test. Then immediately following the test they were to get together to share their items. Fortunately, our test instructions, in addition to citing penalties and prohibitions about testing, also included a statement informing test takers that multiple versions of the same test were in use.
In addition to using alternate forms of a test during test administration, other forms of test security should be employed. As suggested earlier, this should include the prohibition of electronic equipment at the test site. With the proliferation of iPods, smart phones, note pads, lap tops, digital watches and cell phones, it is practically impossible to determine the multiple ways that these devices could be used to cheat on a test. The simplest prevention is to prohibit them at the test site. In particular you don’t want candidates having access to downloaded information during the test. Similarly, you don’t want candidates using cell phones to call friends for answers or take pictures of the test. My experience has convinced me that it is also important to limit the number of people using the restroom at any one given time and to escort them if possible. In that regard, it is important to have a good ratio of proctors to test takers as well as ensuring that proctors are qualified and receive some orientation on being a good proctor.
In the presentation “What’s New in Technology and Cheating,” Jamie Mulkey of Caveon Test Security outlines suggestions for limiting electronic devices as well as the creative ways in which students have attempted to cheat. Individuals in the test development and administration field should review the information in this presentation and incorporate what they can in their own anti cheating program along with visualizing other related ways test takers may attempt to cheat so they can develop counter measures.
Anecdotal information also suggests that strong admonishment against cheating provided either in writing or verbally during test taking instructions serve as a deterrent. As intended in these articles, some publicity regarding the consequences faced by those caught cheating also serves as a deterrent. As articles surface, it is good for those on both sides of the testing process to learn from them. To test developers they can serve as a reminder and as a lesson regarding consequences to those who may be tempted to cheat.
Post test challenges exist also in the form of test review. Any information provided to current test takers regarding test items and their keys compromises a test for future use. Civil Service Rules that do provide for test review should be written to greatly limit this process. It is important that if individuals are allowed to review test materials that they are not permitted to take any sort of written materials out of the review area. In that regard, I believe that individuals who truly have an issue with an item should be required to raise that issue before leaving the test area. That is, if someone believes an item is inappropriate or an item doesn’t have a correct answer as one of the choices or a related issue with the test she should put the issue in writing before she receives her test score. This prevents fishing expeditions by candidates who don’t get the score they thought they should.
While a great deal of anti cheating measures focus on individual cheaters, my experience along with the stories cited in the last article tell us that cheaters don’t always act alone. Therefore, as stressed, it is important to broaden the scope when looking at who has motive to cheat. In the police ranks, I have heard it said by some test takers that they don’t care about being promoted, but they sure don’t want to see so and so promoted. This suggests that they may consider participating in a scheme to affect their desired results. Unfortunately, experience has taught us that this is not necessarily limited to the lower ranks within organizations. Therefore, again, one must use a broad brush when looking at who may be motivated to influence the outcome of a test in their favor.
Hopefully, a systematic approach to developing an anti cheating program that incorporates all the materials available for writing the program along with an analysis of who might cheat and the methodology for preventing such efforts will prevent your jurisdiction from becoming a victim.
Test taking instructions at the start of the testing session should be strong and unequivocal regarding the agencies stance on cheating. In particular the instructions for taking the test should stress that cheating will not be tolerated and those “suspected” of cheating will be subject to strict penalties. Practice has provided anecdotal information suggesting that such admonishment do tend to prevent those who are considering cheating from actually attempting to do so. In particular, many agencies have found that instructions that state those, “suspected of cheating,” will be subject to penalties is quite powerful since it takes the pressure off the agency to prove cheating took place as well as requiring strict behavior from test participants.
IPMA-HR has two videos that outline our test security agreements as well as measures you can take to keep IPMA-HR tests secure.
For the third part in this series, Andrey Pankov will be writing an article on IPMA-HR security procedures, including a review of IPMA-HR’s Test Security Agreement (TSA) that all customers must read and sign before ordering and receiving tests, our Limited Access Security Agreement (LASA) that individuals have to sign to review or administer test materials but have no permission to actually order and receive tests, and our item challenge procedures.