It Pays to Take A Look: Item Analysis Part 1
Any organization using written exams as part of their selection processes that doesn’t take the time to review the information provided by an item analysis is overlooking a treasure trove of information. Without performing an item analysis on a written exam and acting upon the information gained from such an analysis, a jurisdiction truly does not know how that exam is performing.
There are two fundamental concepts involved in test utility and they are test validity and reliability. Simply put, validity refers to whether or not a test measures what it is intended to measure and reliability refers to how consistently a test measures what it is intended to measure. Item performance directly impacts these two concepts. Test items are the basic element for gathering information about test participants. In order for a test to have the necessary validity and reliability to make it worth using, test items have to perform optimally and gather the best information possible as accurately as possible. If test developers do not utilize information available from an item analysis, that test developer has no idea how items are performing. If items are performing poorly then all other statistical analyses become less meaningful and the test is not suited for its intended purpose. Hiring decisions based on the use of the test are questionable since they become another layer of what is ultimately a house of cards that does not have an adequate foundation. In this case, the written exam has not played its role in terms of optimizing accurate selections and supporting the mission of the organization. (more…)
Best Practice Standards: IPMA-HR Resources
Today, I would like to discuss how IPMA-HR’s assessment resources can be used to ensure your organization’s practices adhere to professional standards of practice.
Adverse impact is at the center of an increasing number of news stories written about selection practices. Unfortunately, a large number of departments and agencies don’t do enough to protect themselves. Over time, HR and assessment professionals have learned that it nearly impossible to eliminate adverse impact within cognitive testing. More than ever, it is important that your organization protect itself in situations where adverse impact may surface, opening the door to possible litigation.
Happy New Year!
Everyone here at IPMA-HR would like to wish you all a very happy and prosperous 2012!
In this New Year we will continue to innovate our assessment products as well as enhance the content brought to you through this blog.
We’ll be sending our new test product catalogs out in the mail in the next few weeks. We decided to split the catalog into five smaller catalogs to separate out the areas our tests cover: police, fire, corrections, dispatch and administrative. The new format will make it easier for you to share our products with co-workers by only giving them the catalog that focuses on their needs, while keeping the catalog you need on hand.
Also, we continue to improve our website, which you can always find at:
http://testing.ipma-hr.org
Over the course of the year, we’ll work on improving online availability of more of our documents, such as reading lists, technical reports and test response data reports.
Speaking of websites, we’ll have some big news in the coming weeks about a new site that we think you’ll love.
In addition to finding us online, you can get in touch with us by:
| Phone: | (800) 381-TEST (8378) |
| Fax: | (703) 684-0948 |
| Email: | assessment@ipma-hr.org |
| Mail: | IPMA-HR 1617 Duke St Alexandria, VA 22314 |
Thank you again for helping us making 2011 great! Let us know how you’re enjoying the new Assessment Services Review in the comments below.
In this New Year, don’t let your agency take a tumble back in time.
The recent article by Rich Tonowski certainly brought to the surface some of the thoughts I have had when reading and hearing about public safety departments all over the United States getting into large scale legal battles in regard to their selection processes.
Some of these litigation cases are certainly deep and complicated, such as Dukes v. Wal-Mart Stores, Inc. and Ricci v. DeStefano. However, others like Easterling v. Connecticut Department of Corrections and United States v. New Jersey Civil Service, are lessons in assessment basics that should always be at the forefront of an HR or assessment professional’s mind. Basics like ensuring that job analysis of a target position includes opinions and ratings from a diverse group of personnel and making sure to test candidates in areas that are directly related to job tasks.
During the month of January the ASR will focus on establishing standards you should keep in mind to ensure your department doesn’t end up as the lead story on tomorrow’s evening news. Our coverage kicks off with the next series from Robert Burd, on the topic of issues that might arise during test administration. Specifically he will discuss using successive hurdles, weighing different parts of the selection process and certification rules for eligibility lists. You can also look forward to a piece on the importance of Test Response Data, both to IPMA-HR and your agency as a whole. We will also be highlighting outside professional resources that are important for you and your staff to review and understand. Concluding the month long theme, we will bring you an in-depth look into non-cognitive testing, and how it can be implemented within your assessment process.
In Public Safety Selection Testing, It’s 1980 Again
The City of Chicago ran an entry-level firefighter test in 1995. After many subsequent uses of the results, a court ruled in 2004 that the test was discriminatory, based on race. The city did not try to defend the test. Instead, city officials argued that discrimination charges were time-barred. Finally in 2010 the U.S. Supreme Court agreed in Lewis v. Chicago that the test was bad but found the charges were good. The Chicago Tribune reported on August 17, 2011 that the decision will cost the city $45M. This does not include the city‘s legal costs.
In Ricci v. Destefano the City of New Haven not only did not defend its firefighter promotional tests, but portrayed them as somehow flawed to avoid certifying promotional lists. It was not until the final Supreme Court substantive brief that the city asserted that there was a problem with the tests’ validity. The Court ordered the city to live with the results. Along the way some well known industrial-organizational psychologists filed an amicus brief asserting that the tests could not be valid because they were content-deficient. And now the Second Circuit in Briscoe v. New Haven has cleared the way for the discrimination suit the city said it was trying to avoid. The court noted that Briscoe is arguing, among other things, that the “industry norm” for firefighter promotion is 30% for the written test and 70% for the oral component, rather than the 60/40 weighting that the city used. (more…)
New Promotional Fire Tests Now Available
We are pleased to announce the launch of four new fire service promotional examinations, designed to assess qualifications for promotion from firefighter to company officer.
Company officer is a first-line supervisory rank commonly called fire lieutenant or fire captain in different municipalities. The new tests incorporate the most current developments in the firefighting field — new technology, new regulatory requirements, hazardous materials updates, new developments in firefighting and rescue, as well as supervision and team leadership concepts. All test questions were selected using a thorough validation process, including linkage to the job analysis data and review by 18 or more expert practitioners in the field, from different departments and geographic regions, to assure each question’s accuracy and relevance. All questions were also linked to study references which were formulated into a recommended reading list for candidates.
The final products of the validation study were four 100-question tests. Two of the four tests are designed for departments which provide EMS services, and the other two tests are for non-EMS departments.
Each test covers areas such as fire behavior, firefighting procedures and tactics, rescue and safety, building construction, supervision, hazardous materials, fire prevention and fire safety, and (for two of the above forms) emergency medical services.
The final test forms are:
| FCO101-EM | Fire Company Officer test with EMS component |
| FCO102-EM | Fire Company Officer test with EMS component (alternate form of FCO101-EM) |
| FCO103 | Fire Company Officer test – no EMS component |
| FCO104 | Fire Company Officer test – no EMS component (alternate form of FCO103) |
Besides reflecting the most recent developments in the field, these tests provide more flexibility for the departments which use them. Departments can alternate the use of these tests over time to avoid overexposure of the questions in the promotional process.
A Technical Executive Summary is available for the FCO Series of tests in lieu of a full Technical Report. The full Technical Report is expected to be made available by the start of 2012.
Reading lists for all four tests are available immediately to Test Security Agreement signers.