Readability of Assessments in a Digital Age (Part 2): Practical Issues
- We want test questions that are very detailed, highly complex, engage the test taker, reflect the job, and, oh yes, are at a 6th grade reading level.
- Grade level = 5.9. Reading level from paragraphs from a 3rd grade reader from 1960, as calculated by Word.
This is part two of a blog dealing with the measurement of readability and the establishment of appropriate reading levels. For purposes of this blog, readability can be defined as the ability of material to be comprehended by its intended audience.
In Part 1, we investigated approaches to readability based on:
- The measurement of grammatical features or readability formulas.
- The linguistic perspective.
- Job analysis.
In Part 2, we turn our attention to more practical issues such as:
- How are readability indices used by assessment professionals?
- What adjustments can or should be made when evaluating multiple-choice tests?
- How has the changing nature of jobs impacted readability?
Essentials of Employment & Labor Law: Webinar Series from IPMA-HR
Readers of the ASR have expressed interest in articles detailing legal issues so we wanted to pass along this information from IPMA-HR.
You can join renowned employment and labor law attorney David Ritter of Barnes & Thornburg LLP, for IPMA-HR’s Essentials of Employment & Labor Law webinar series. This new webinar series is the most comprehensive, practical, up-to-date employment and labor law training available for public sector HR practitioners.
This 4-part series is geared to the real-world needs of human resources professionals and managers, providing “best practice” insights and information on some of the most pressing issues in employment and labor law today.
Available both live and on-demand, the four webinars include:
The Legal Implications of Social Media in the Workplace
March 25, 1:00 p.m. EST
The Family and Medical Leave Act (FMLA) & Americans with Disabilities Act (ADA)
June 17, 1:00 p.m. EST
Fair Labor Standards Act (FLSA)
October 14, 1:00 p.m. EST
Conducting Employee Investigations
December 9, 1:00 p.m. EST
Become an even more effective human resources administrator with this essential webinar series.
Purchase now and watch at your convenience — webinars are available both live and on-demand.
Earn Education Credits
HRCI | The Essentials of Employment & Labor Law webinar series, has been approved for 4 (HR (General)) recertification credit hours toward PHR, SPHR and GPHR recertification through the HR Certification Institute. For more information, visit www.hrci.org.
IPMA-CP | This webinar series has been approved for 4 points (1 point per webinar) toward IPMA-CP or IPMA-CS recertification. To learn more, please visit: www.ipma-hr.org/recertification.
Readability of Assessments in a Digital Age (Part 1): Bet You Won’t Read This Whole Blog
- People online don’t read.
- Olny samrt poelpe cna raed tish – cna you?
The opening epigraphs both deal with readability. The first is a commonly encountered claim that people scan rather than read when perusing material online. What does that mean for employment websites and the associated assessments? The second is a teaser that often makes the false claim that very few people can read the material, when in fact almost everyone can. It illustrates that individuals can make sense out of what appears to be unreadable or scrambled text. Both have implications for our topic for this two-part blog, which involves the readability of assessments.
The measurement of readability and the establishment of appropriate reading levels is a critical responsibility faced on a regular basis by many selection specialists and personnel managers in the public sector. This task involves an analysis of the assessment of materials used on the job and the tests or assessments used in selection. Readability can be defined as the ability of material to be comprehended by its intended audience.
Unfortunately, most of our knowledge of the impact of readability on assessments was developed in an era where we used paper-and-pencil, multiple-choice tests. Even that literature is limited in that most of it deals with educational tests. Very few studies look at the actual impact of readability on the difficulty of employment tests or potential racial bias in tests. I could spend this blog complaining ad nauseam about researchers conducting highly artificial studies of irreproducible phenomena of little generalizability, while ignoring questions of real practical importance, but that is another topic for another day or forum.
One of the questions we will examine in the second part of this blog is whether readability is still relevant for computer-based tests. However, before we do, we will review the more traditional literature on readability and how we measure readability.
In Part 1, we investigate approaches to readability based on:
- The measurement of grammatical features or readability formulas.
- The linguistic perspective.
- Job analysis.
ASR Bookshelf: Your Guide to Assessment-Related Resources
Periodically we will be posting links to resources that we have found from across the web. We hope you find this information helpful as you navigate through the everyday, and not so everyday, issues facing HR professionals today. Here’s a list of resources. (more…)
Welcome to the New Year and Forecasting the Future of Assessment
Traditionally, I have started the new year with a blog that recaps the past and looks to the future in assessment. My habit has been to insert a statement concerning how difficult it is to predict the future. However, this year I was surprised to find that some of the topics I would select for future trends, were actually covered in my blogs over the past year. So, maybe with age I am getting better at prophecy.
My predictions for future trends or hot topics over the coming year include:
- Mobile Devices and Technology
- Big Data and Predictive Analytics
- Branding
- Police Performance.
Mobile Devices and Technology
The Society for Industrial and Organizational Psychology (SIOP) recently listed their top ten trends for 2015. Number 1 was Mobile Assessments. It is clear that mobile assessments increase “flow through” or the number and diversity of individuals tested. On the other hand, mobile assessments present a number of challenges in terms of programming, comparability of scores, and the noise in the environment.
Practitioners have a large number of questions concerning the use of mobile devices, more questions than there are evidence-based answers. As a result, I can safely predict an increase in the number of publications we will see dealing with the issue of mobile testing, especially research looking at the question of measurement equivalence across device type. From a personal perspective, I see the issues involved in using mobile devices for assessment as one piece of the larger technology puzzle, which would include the movement to greater use of online assessment. Candidates have come to expect that assessments will be offered in an online version. Online assessments have many advantages including simplicity, efficiency, and cost savings.
IPMA-HR is now offering selected tests through the new Online Test Administration Service (OTAS). The plan is to have all tests available for online administration in the near future.
Big Data and Predictive Analytics
I must admit to having concerns regarding the faddish nature of the Big Data and Predictive Analytics movement. From my perspective, assessment professional have been engaged in Predictive Analytics for over 60 years. Nevertheless, Big Data is here to stay and finished Number 2 on the SIOP trend list. Assessment professionals will have to familiarize themselves with the language of Big Data and Predictive Analytics.
We will have to increase our awareness of advancements in the use of Big Data and Predictive Analytics and ensure that all selection decisions are made in a fair and valid manner, whether based on traditional models or empirical relationships discovered through Big Data analytics. In particular, within the public sector, we should remain wary of any attempt to substitute the measurement of demographic or Big Data-based variables for professionally developed and validated assessments of individual merit.
Branding
Related posts by Doverspike, D (2014):
The management of “brand” or “image” can be seen as particularly important in the public sector because there is a strong belief expressed in a variety of media that young job seekers are not attracted toward government agencies and jobs. Thus, human resource professionals must be concerned with the maintenance of a positive public sector image. Selection strategies can impact the image that applicants, employees, and the general public hold of your organization. Thus, we should take responsibility for the impact of our selection methods and decisions on the reputation of the organization.
An important part of branding is the impression that your employees make on the public. Those who work directly with the public are a major factor in shaping the image the public will hold of your agency. Effective and efficient customer service is key. With the improvement of customer service in mind, IPMA-HR will be working on rolling out a “generic” customer service test for 2015. The availability of this new instrument should aid organizations in the task of identifying and hiring the best customer service personnel available.
Finally, as promised several months ago, some results on branding in the public sector from my mini-survey. Unfortunately, I only received 19 complete responses; I thank those who took the time to complete the survey. Based on the readership of the blog, I would assume most individuals are employed in the public sector.
Now the results (again based on a small, mostly public sector sample):
- The public sector was seen as having the following positive attributes:
- A good image among respondents and being a good place to work.
- A socially responsible image.
- Hard working employees.
- The public sector was seen as having the following negative attributes:
- A poor or below average image in the mind of the general public.
- A below average financial image and future.
- An image as failing to pay fairly, although with good benefits.
- An image as failing to pay based on performance or merit.
Looking at the results, I believe they suggest a divide between the way public sector employees see the government as an employer, which would be in a generally positive fashion, and the way employees believe the public sees the government, which is not very positive. The respondents were also concerned about the financial shape and future of the government, which seems tied to the general issue of the fairness of pay, as well as the ability to attract future employees.
Compared to many private sector companies, the reaction of public sector employees to the government as an employer tends to be pretty positive. The results of this mini-survey support this viewpoint. Hopefully, the positive message of work in the public sector can be communicated to future candidates for employment.
Police Performance
As I write this blog, the topic of police performance dominates the news cycle and social media. Police work is incredibly difficult and the educational requirements associated with the job continue to rise. At the same time, many communities are reporting that they are experiencing a shortage of applicants; recent events will probably exacerbate that trend. As assessment professionals, we are under continued pressure to recruit, screen, select, train, and retain highly qualified individuals to serve the public through police work.
In order to assist you, IPMA-HR Assessment Services has added to its already existing suite of products for police selection. In addition to the previously mentioned availability of online testing, Assessment Services is introducing a Police Officer Structured Interview System (POSIS). Based on extensive studies with nearly 1,000 candidates, POSIS will provide in one package everything you need to conduct structured and defensible oral interviews, thereby adding a new item to your assessment arsenal.
The POSIS guides you through all the steps of the process including planning, training raters, delivery of the interviews, and scoring. The POSIS should greatly aid communities in delivering a standardized, valid interview process, which should lead to the hiring of highly competent police officers and a more positive image for both human resources and the police department.
Conclusion
My final prediction is that public sector assessment professionals will have to expand their competency in various areas of emerging technologies, including an expanded knowledge of online assessment, Big Data, and the development of “branded” assessments. This will require that we learn to communicate and work on teams with members of allied professions, especially those in the Information Technology areas. As we move into the future, you can count on IPMA-HR Assessment Services to develop and deliver innovative products and selection systems, while continuing to support their traditional battery of tests.
This brings to a close my first year as a blogger for Assessment Services. I hope you, the reader, found my blogs to be informative and enjoyable. I am always willing to consider any feedback you have or suggestions for future topics.