The Value of Employee Assessment (Part 4 of 5)

If you’ve been tracking this series of posts, you know we’ve covered some historical perspective on the use of assessments, and spent a couple weeks digging deeper into the world of employee selection.  This week, we’ll shift our focus to a brief discussion of how assessment is being used to support ongoing employee development.

While people typically associate the term "assessment" with selection, organizations are increasingly using assessments for developmental purposes. Here are a few of the ways assessments are adding value:

• Ensure employees have acquired the necessary knowledge and skills during training. Organizations spend about $60 billion on training each year, and a modest investment in assessments can significantly leverage those resources. The data are also very helpful in fine-tuning training programs.
• Better target training resources. Rather than assuming that all employees need the same training, assessments can help identify the specific areas where training would be most beneficial on an individual basis.
• Promote continuous learning and development. Good assessment programs periodically measure employees' capabilities and provide specific feedback to help them further develop. Some assessments measure the degree to which knowledge and skills are actually being applied on the job, which, of course, is the real point of development.
• Empower managers. Managers are just as responsible for developing the talent pools in their organizations as they are for accomplishing operational goals. Unfortunately, while they get a lot of information pertaining to how they're doing on operational goals, they get no information on the knowledge and skills of their people. Good assessment programs can provide this valuable data.
• Improve organizational effectiveness. For example, a large financial institution Censeo works with administers a product knowledge test every six months to all of its employees who interact with customers. Follow-up research showed very high relationships between product knowledge and a variety of financial measures. This propelled the organization to an even greater commitment to employee development.

Increasing the Value of Assessments in Development

This graphic illustrates the strategies involved in increasing the value of assessments.

The main point relevant to this discussion is how assessments can be used strategically to build a performance-oriented organizational culture. Here are a few suggestions on how to accomplish that:

1. Understand the business drivers and strategies of the organization. Examine what the organization is trying to achieve and what metrics will be used.
2. Identify the activities to be performed and the KSAs employees need to carry out the strategies. Set performance targets and then measure proficiency
   on the critical KSAs using appropriate assessments.
3. Assess the gaps between actual performance and set targets, and then implement performance feedback sessions in which employees and their managers agree on the development activities necessary for building capabilities linked to the organization's strategies.
4. Once employees have gone through training and other developmental activities targeted to improve knowledge and performance, measure their acquired knowledge and skills to ensure learning took place.
5. Finally, after an appropriate interval, measure the effect of the assessment process and all of the developmental activities on the organization. Did the metrics important to the organization significantly improve? Were customers more satisfied? Were employees more content in their jobs? The process is a continuous loop, as all the data should be used to refine the business strategies, required KSAs, targets, etc.

From the employee's perspective, this performance-oriented system ensures that their development is focused on the things that matter — things that will help them do their jobs better and faster, possibly opening the door to greater rewards and faster advancement. The value of assessment to the organization is in the cost savings and productivity gains from a targeted development process. Giving employees exactly what they need in development when they need it will make the organization that much more effective.

The Value of Employee Assessment (Part 3 of 5)

Last week’s post made the point that including assessments in the selection process adds value. How much value depends on several actions you can take, and I’ll outline a few of these best practices in this week’s post.

1. Success profile. Build an accurate profile of the job by specifying the KSAs and personal attributes most important to success. Understand the goals of the organization and how they impact the jobs being staffed. For example, what skills are critical to the organization today, and how will that change over the next few years?

 

2. Selection tools. Based on the success profile, choose the selection tools that will provide the data needed to make sound hiring decisions. With respect to assessments, decide whether you should obtain good, valid tools from a vendor or develop your own. Aim for more assessments rather than fewer. Cost-benefit studies consistently show that even if a selection tool only marginally increases predictive validity, it will yield a very high ROI.

 

 

3. Selection process flow. Organize the tools in a logical way that will be effective, efficient and reduce costs. The graphic below illustrates the "selection funnel" approach to selection.

 

Assessments are typically very fast and relatively inexpensive. Tools that are fast and low cost are used early in the process to screen out less qualified candidates. The more time consuming and costly tools, such as interviews and assessment centers, are used later in the process.

 

4. Administration. Ensure that consistent practices are followed during the selection process, particularly with respect to how assessments are administered. Inconsistent administration can lead to lower utility and legal problems. Read and carefully follow the instructions in the manual.

 

5. Adverse impact. Examine whether or not the assessment adversely impacts protected groups. (Is the minority group selection ratio less than four-fifths of the majority group's selection ratio?) This is more often found in instruments measuring reasoning or cognitive ability. Use the instrument if it's a valid predictor of job performance, and a better, less adverse instrument cannot be found.

 

 

6. Follow-up validation. After some time on the job, collect performance-related data (as well as other indices like turnover and employee attitudes) and compare it with scores from the assessment. Use the results to make any necessary adjustments to the selection process.

 

With next week’s post, I’ll shift gears a bit and focus on the employee development side of things.

The Value of Employee Assessment (Part 2 of 5)

The Value of Assessments in Selection

Imagine the repercussions to an airline if its pilots were improperly certified, or if city administrations failed to screen the people who police our streets and protect our property. The high levels of on-the-job failure would lead to a loss of life, harm to the public, poor public image, and eventually, serious financial implications for those organizations.

While the consequences of error in such jobs could be fatal, the consequences of error in less hazardous and less complex jobs are also significant. For example, mistakes made in hiring customer service representatives can lead to low performance, high turnover, unhappy customers, lost business and higher labor costs.

The primary reason we assess people in a selection process is to increase the probability of hiring the best candidates - people with the skills to perform well, and who are more likely to be retained. Some assessments measure knowledge, skills and abilities (KSAs), while other assessments measure personal attributes such as personality, work preferences and motivation. Both KSAs and personal attributes are important, but the latter are particularly important in reducing turnover. Call centers, for example, typically struggle with very high turnover, but it's more of a fit issue than a skills issue.

What are the costs associated with making a bad hire? It depends on the job and how the costs are calculated. The American Management Association estimates that the cost is 1.4 times an employee's annual salary. The California Strategic HR Partnership (a consortium of Fortune 500 executives) calculated that, for certain positions, the cost can be almost 10 times an employee's annual salary (approximately 1 million dollars in the case of a software engineer).

What's behind these figures? They are comprised of some obvious costs such as advertising and recruiting, administrative costs for managing the selection process, travel and relocation, and the training and development new employees need to get up to speed. However, the less obvious costs are even higher - opportunity costs and lost business, lower performance until new hires are trained, negative impact on morale, and so forth. As the complexity and level of the job increases, the cost of a bad hire also increases. In some cases, making the wrong hiring decision at the CEO level can cost hundreds of millions of dollars. From the organization's perspective, there are huge financial benefits to including assessments in the selection process, insofar as the information will lead to better hiring and placement decisions. The individuals being assessed also benefit. Employees are more satisfied and advance further in their careers when their skills and personal attributes match the requirements of their jobs.

There are other benefits to selecting the right people. Jim Collins, author of "Good to Great," talks about "getting the right people on the bus." He notes that most people assume that great business leaders spark change by announcing to the people "on the bus" where they're going - by setting a new direction or by articulating a fresh corporate vision. Based on his research, the companies that moved from good to great were those that began their journey by asking "with whom" rather than "where." In other words, great companies start by getting the right people on the bus, the wrong people off, and the right people in the right seats.

Until next week...

The Value of Employee Assessment (Part 1 of 5)

Today I kick off a five part series on the value of leadership assessment.  Sure, folks tend to agree that assessing skills is generally a good thing, but why?  What role does assessment play and what value does it lend from a business perspective?  These are the questions I'd like to address over the next few weeks... hope you find it helpful, and I'd love to hear your thoughts on the topic.

With that, let's get started with a short post to provide some historical perspective.

For thousands of years, humans have measured work-oriented skills. As far back as the Chan dynasty in 1115 BC, civil servant candidates were subject to competitive examinations to test their proficiency in areas such as archery, horsemanship, military affairs, civil law, writing and arithmetic - the knowledge, skills and abilities (KSAs) deemed important for success in a civil servant role.

It wasn't until World Wars I and II, however, that the science of assessing people for the purposes of selection, placement and training was applied on a truly large scale. The US military used aptitude tests, among other tools, to quickly and efficiently place recruits into roles that matched their abilities. They also developed and used simulations (assessment centers) to screen and train secret agents. The value of assessment was quite evident at that time because of the large number of soldiers that needed to be quickly and effectively screened.

Today organizations spend millions of dollars on assessment tools to help attract, select, promote, train and develop their staff. The motives for assessing people are related to several criteria including productivity, retention, absenteeism, legal requirements and employee development.

Next week we'll get into the meat of the topic by discussing the use of assessment to select better leaders.

The US Supreme Court Ruling on Ricci v. DeStefano – An overview of the case and what the decision really means for test users (Part 2 of 2)

Last week I summarized the Ricci case - the issues and sequence in rulings all the way to the USSC decision. I also noted that the case points out the growing tension between making "race-conscious" decisions when there is adverse impact and making decisions which are "color blind" (ensuring equal protection).

It's clear that this case was really about ensuring "equal protection" and so the court's ruling was in alignment with what it was given to deliberate (unfair treatment of the firefighters who would have been selected had the results not been thrown out). That said, there were some major issues with the construction and validity of the exam, which led to the lower courts upholding the City's decision to throw out the results. What exactly were some of those issues and why did 5 of the Justices disregard that information? Let's deal with some of the issues first:

Issue 1: There seemed to be little expert advice/input on the relatedness of the exam to the job. Specifically, the exam (combination of a multiple choice test and an oral interview) left out some critical competencies required for success - e.g., "Command Presence."

Issue 2: The passing score was arbitrarily set at 70%. Having no rationale other than it's equivalent to getting a "C" has many negative repercussions, the most important being that it leaves the employer open to litigation when certain groups or individuals fail.

Issue 3: The exam's components were weighted (60% Written, 40% interview) solely based on the collective bargaining agreement and not on any rigorous methodology. Ideally the weights should reflect the relative importance of the KSAOs, measured in each component, to the jobs being performed.

Issue 4: The City used a strict rank ordering approach on the tests to identify the best candidates for promotion. In other words, the firefighter with the highest test score would be the first considered for an open position, second highest score would be considered for the next open position... and so on. According to the EEOC "Uniform Guidelines on Employee Selection Procedures" you cannot infer incremental differences in job performance from small differences in test scores without research evidence. In this particular case it's especially problematic since the job-relatedness of the exam was in question. Given that the City collected no such evidence, this selection method was not justified.

So why did these issues not get weighed more heavily in the ruling of the 5 supporting Justices? Essentially, they point to the fact that there was not a "strong basis in evidence" to invalidate the results. The City only threw out the results when they saw the possibility of an adverse impact claim coming (which discriminates against those who performed well) and did not present a record of thorough due diligence on the exam prior to the results. Their attempts to undermine the validity of the exam all happened after the fact. Some evidence to support these assertions include:

• A formal validation study was never submitted to the City or the court
• A substantive challenge regarding the job-relatedness of the exam was never presented to any court for review

Some concluding thoughts about the case:

1. The City of New Haven put themselves in a "no win" situation - certifying the results, would have surely brought an adverse impact claim, which I believe would have been easily won, given the issues with the exam. We know the answer to the reverse scenario.
2. Had the results looked different (no adverse impact) the City may have certified them, however, given what we've learned about the issues with the exam it probably would not have been a very good predictor of job success.
3. Had the City followed a more rigorous process of due diligence with regards to the development of the exam the USSC might have ruled differently... along with the lower courts.  

Some implications for employers/test users:

1. The ruling upholds the use of tests for selection and promotion - it is not a ruling against test use.
2. Well constructed and valid tests will sometimes adversely impact protected groups. In these cases, users cannot just throw out the results.
3. A "strong basis in evidence" is necessary to demonstrate poor test validity and support the decision to throw out results when there is adverse impact.
4. Follow the principles for creating valid and legally defensible selection/promotion tests. Seek out experts to help you in designing these tests, as well as the systems in which they will be used.
5. Beforehand, research if equally valid assessments exist that might have lower adverse impact. For example, simulation type assessments (assessment centers, situational judgment tests) historically have lower incidence of adverse impact.

The Ricci v. DeStefano case will be discussed for many months and years to come. I welcome your comments and opinions on how you perceive the case and its potential impact on employee selection/promotion systems.

Note: An in-depth coverage of the exams issues in the Ricci case is covered in the amicus brief submitted by fellows from the Society of Industrial/Organizational Psychology http://www.abanet.org/publiced/preview/briefs/pdfs/07-08/08-328_RespondentAmCuIndus-OrgPsychologists.pdf)

The US Supreme Court Ruling on Ricci v. DeStefano – An overview of the case and what the decision really means for test users (Part 1 of 2)

I've been reading quite a bit about the US Supreme Court (USSC) ruling over the last few weeks and have received questions from our clients on what the case was really about and what the ruling might mean for them.

This week I'll provide a brief overview of the case for our readers (for specific details, visit: http://www.lawmemo.com/supreme/case/Ricci/) and in part 2 of my post next week I'll discuss some of the issues with the test and the implications of the case. Here goes...

In 2003, 118 firefighters took examinations to qualify for promotions to the rank of lieutenant or captain in the New Haven Fire Department. The results of the exams showed that white candidates passed the exam at a much higher rate than African American candidates... by almost a 2 to 1 margin. A public debate ensued where some firefighters argued that the test results should be discarded as they appeared to be discriminatory and threatened to file a lawsuit if the City made promotions based on the tests. Another group of firefighters said the exams were fair and threatened a discrimination lawsuit if the City ignored the test results and denied promotions to the highest scoring candidates.

After looking at the disparity in results and some design flaws in the test, the City made the choice not to certify the test results. Here now are the various lawsuits and court rulings leading up to the USSC's decision: 

1. White (and 1 Hispanic) firefighters responded to the City's decision by filing a reverse discrimination suit claiming that discarding the results discriminated against them and was in violation of Title VII of the Civil Rights Act and the Equal Protection Clause of the Fourteenth Amendment.

2. The City defended their stance and argued that if they had used the results to make promotions, they could have been in violation of Title VII by using a selection method that demonstrated disparate (adverse) impact. The case was heard by the New Haven District Court and the plaintiffs' (white firefighters) claims rejected, citing prior cases where "race conscious" interventions were used to remedy adverse impact.

3. The 2^nd Circuit Court upheld the District Court's decision.

4. Enter the US Supreme Court (USSC), which then reversed the decisions of the lower courts (by 5 to 4 margin) stating that the City of New Haven did not have enough evidence/justification ("strong basis in evidence") to throw out the results for fear of being potentially sued for discrimination.

Essentially, the USSC's decision sets the potential precedent that test users cannot just discard results when there is a hint of adverse impact. The case further brings to light the looming conflict between remedying adverse impact (race based decision) and facilitating equal protection (creating a color blind society). It seems like the City of New Haven was in a "no win" situation... either avoid the adverse impact suit at the risk of facing disparate treatment claims or vice versa.

Next week I'll discuss some of the issues with the firefighter exams, why these issues were not fully considered in the USSC's arguments and also what test users should take away from the case.

Leadership Assessment Validity: What is it? Why Does it Matter?

Rarely does a week go by that one of our clients doesn't ask questions around the validity of assessments, be it leadership assessments or other tools... and the topics of validity and fairness have become even more front-and-center as a result of the recent Supreme Court decision in Ricci vs. DeStafano.

In my opinion, all this attention is a good thing... and clients should always be asking vendors these questions about validity.  Why?  Well, the driving factor in many cases is a company's desire to protect itself in the event of legal action, and that's understandable in today's litigious environment. 

However, there's a larger, extremely important, issue regarding the validity of assessments which companies seem to give less attention.  Simply put, valid assessments work better!  They are scientifically proven to predict on-the-job success or failure better than their non-valid (or unvalidated) counterparts.  They are also more accurate in assessing an employee's unique strengths and areas for improvement.

What's this mean for companies?  Better quality hires, more targeted employee development, and better ROI - just to name a few things.

Given it's such a timely topic; we thought you might benefit from a concise, practical definition of "validity."  In this video, I take a couple minutes to interview Industrial and Organizational Psychologist, Dr. Michael Zia Mian on the topic.

Hope you find it beneficial!

Also - if this topic is of interest, be sure to see our next two posts (August 28^th and September 4^th).  Dr. Zia Mian will be discussing the Ricci vs. DeStafano case in detail, and it's implications on HR practitioners.

Should 360 Degree Feedback be used in Performance Appraisals? (Part 2 of 2)

It's hard to refute the basic logic of arming both the manager and the employee being appraised with the most accurate data possible, and to use that information intelligently to help understand past performance and determine how performance can be improved in the future.

360 degree feedback can provide such data. Here are three suggestions on how to capitalize on that opportunity, while minimizing the risks:

1. Initially use 360s only for development - If 360 degree feedback is new to your organization, focus on using the process for development purposes only for a couple of years. Give people time to get used to the whole concept of 360 degree feedback, and to appreciate how valuable such feedback can be. Then consider the possibility of using it in some fashion as part of the PA system.

2. Don't tightly link 360s to the PA system - This means several things. First, 360 degree feedback should be only one of the data sources used by the manager in evaluating performance. Second, 360 degree feedback results should never be hard-wired to PA ratings in the sense that there's any kind of mathematical relationship. Third, don't link the timing of 360 degree feedback to the timing of PAs. Most PA systems use an annual cycle tied in with the company's fiscal year, but conducting 360 degree feedback every year is too often. Conduct 360 surveys when it makes sense to do so for development purposes. Then use the results as one of the inputs to the PA system, if they happen to be timely enough.

3. Clearly communicate how the 360 tool will be used - Clearly communicate to participants and raters how the 360 degree feedback will and won't be used with respect to the PA system. Don't underestimate the challenges in communicating these matters, particularly if you're announcing a policy change (e.g., a shift from "strictly for development purposes" to something else). It will take numerous and repeated communications to allay the fears and concerns on the part of all stakeholder groups - participants, managers and raters.

A reminder to those of you thinking that I've just provided great rationale for you to now quickly integrate your 360 and PA systems, or maybe you've been integrating them for some time and this just validates your stance - the success of the integration really will depend on following the guidelines mentioned and paying close attention to what happens during and after the performance review. A well planned structure is still necessary to ensure that employees actually engage in developing their competencies. While there is too much to cover in this post, some critical questions you should be asking include - Do employees understand your performance review process and how 360 ratings will be used? Are managers equipped to conduct effective feedback discussions in the review? Is there a development planning template and is regular follow-up part of your process?  

It's the age old question ... how effective is your system in helping people change behaviors? Essentially.... that's the bottom line!

Should 360 Degree Feedback be used in Performance Appraisals? (Part 1 of 2)

In the second half of every year, as annual performance reviews approach, I'm frequently faced with the client question "is it appropriate to integrate employee performance appraisals (PA) and 360 results?" There is considerable debate and controversy about the relationship between the two, but from where I stand, 360 degree feedback can play a beneficial role in PA systems, if several issues are carefully managed.

Before covering those issues, let me first describe the broader context of PA systems and one of the reasons they often don't work very well. Most PA systems include two components - evaluation of operational/business goals to be achieved and evaluation of performance factors (how the job is accomplished and the competencies required). Typically evaluations of both components are then integrated into some overall "score" or rating, and there are usually consequences linked to those ratings, such as merit salary increases, bonus allocations, and so forth.

The rationale for looking at performance factors is that companies want managers and their employees to place an emphasis on improving competencies and skills, in addition to operational goals, because improving them increases the potential for future business success.

Let's now talk about why this traditional PA process often doesn't work very well. The one most relevant to my post is that evaluations of performance factors (the behaviors and competencies demonstrated) are not as accurate when the assessment information comes only from the manager. (This is usually less of a problem for operational objectives because more quantitative, objective measures are available.) In an effort to improve this important part of their PA systems, some companies use 360 degree feedback to increase the accuracy of assessments and evaluation. Why? Because having several different perspectives on an individual is better than just one - this is the driving premise of the 360 tool. So, what's the problem and why is there controversy? Here are the four main issues:

1. Decreased rater honesty and accuracy - When they know that significant consequences are tied to their ratings, people providing 360 ratings may be less honest and accurate (i.e., positive or negative bias could be introduced).

2. Participants focus more on ratings than development - Participants might be less inclined, not more, to proactively engage in developmental activities when their PA ratings are influenced by the 360 process. In fact, their main motivation could be to do everything possible to select raters who will be predisposed to give favorable ratings, versus those who will give honest feedback.

3. 360 results lose their "future focus" - 360 degree feedback surveys are more focused on the future (using feedback to improve future performance) whereas PAs are more focused on the past (evaluating goals accomplished and behaviors demonstrated over the past year). Therefore, when put together, there's a risk of the 360 degree feedback process becoming "tainted," and losing its positive focus on the future.

4. 360 tool loses its positive appeal - Associating 360 degree feedback with the PA system (which typically has a very negative reputation in the organization) could jeopardize positive views people hold about the former.

My general advice to clients is to be very careful about incorporating 360 degree feedback into appraisal systems. The advantages of making modest improvements in the PA system could be outweighed by the potential damage done to the 360 degree feedback program. Carefully think through what you're trying to accomplish and why, and evaluate the advantages and disadvantages of moving in that direction.

Looking ahead to next week, I'll dive deeper into how 360 degree feedback can be leveraged while, at the same time, minimizing the risks.

Job Simulation: The future of leadership assessment? (Part 4 of 4)

As noted in my last post, there are many powerful advantages to using simulations for selecting and developing leaders. The problem is that many organizations are finding it more difficult to justify the cost of wide spread use. That said, recent technological developments in the field have made it possible to use established online solutions that are not dependent on human administration and scoring and allow participants to be remote. They can also assess an unlimited number of candidates at once, streamlining the selection process for organizations.

These "next generation" simulations utilize rich media (streaming video and audio) to present critical situations that managers frequently encounter. Participants view these situations and are typically then instructed to select the most appropriate action from a series of options. Since scores have already been assigned to the options, once the assessment is complete, the online system can automatically generate a report for the participant and/or administrator. 

In order for these solutions to generate positive ROI, however, they must be robust, predictive of leadership success, on target for the company's skill requirements, and be multi-purposed (can be used for both selection and development).

It's possible that some organizations might believe in the need to develop their own online leadership simulation. While this may be appropriate for some, leadership skill requirements and the contexts in which they occur are, for the most part, generalizable across companies. I would encourage you to thoroughly research what's already out there, before incurring the time and cost.

Censeo's online leadership simulation - TalentSIM (http://www.talentsim.com/) gets at several critical skills managers need to be successful and is an example of the "next generation" of simulations.

Well, that concludes my series on job simulations - the future of leadership assessment. I hope you have come away from this excited about how far we have come, what we can currently do in the world of simulations and leadership assessment and what might be possible in the near future.