Leadership Assessment with 360-Degree Feedback: Ensuring a High-Quality Rater Group

This time of year, many of Censeo's clients turn their attention to leadership assessment via 360 degree feedback.  It's a great time to take stock of the past year, establish a new benchmark of leadership performance, and use assessment results to identify and prioritize goals for the coming year.

One important factor in conducting an effective and accurate 360 degree feedback process involves the selection of high-quality raters - raters who are well positioned to provide candid and objective feedback.  So how can organizations accomplish this?  If 360 participants are given full control to assign their own raters does the quality of the process suffer?

In this short video, Censeo's Director of Consulting, Dr. Michael Zia Mian addresses this issue, and offers a specific best practice of ensuring a high-quality 360 rater population.

The Value of Employee Assessment (Part 2 of 5)

The Value of Assessments in Selection

Imagine the repercussions to an airline if its pilots were improperly certified, or if city administrations failed to screen the people who police our streets and protect our property. The high levels of on-the-job failure would lead to a loss of life, harm to the public, poor public image, and eventually, serious financial implications for those organizations.

While the consequences of error in such jobs could be fatal, the consequences of error in less hazardous and less complex jobs are also significant. For example, mistakes made in hiring customer service representatives can lead to low performance, high turnover, unhappy customers, lost business and higher labor costs.

The primary reason we assess people in a selection process is to increase the probability of hiring the best candidates - people with the skills to perform well, and who are more likely to be retained. Some assessments measure knowledge, skills and abilities (KSAs), while other assessments measure personal attributes such as personality, work preferences and motivation. Both KSAs and personal attributes are important, but the latter are particularly important in reducing turnover. Call centers, for example, typically struggle with very high turnover, but it's more of a fit issue than a skills issue.

What are the costs associated with making a bad hire? It depends on the job and how the costs are calculated. The American Management Association estimates that the cost is 1.4 times an employee's annual salary. The California Strategic HR Partnership (a consortium of Fortune 500 executives) calculated that, for certain positions, the cost can be almost 10 times an employee's annual salary (approximately 1 million dollars in the case of a software engineer).

What's behind these figures? They are comprised of some obvious costs such as advertising and recruiting, administrative costs for managing the selection process, travel and relocation, and the training and development new employees need to get up to speed. However, the less obvious costs are even higher - opportunity costs and lost business, lower performance until new hires are trained, negative impact on morale, and so forth. As the complexity and level of the job increases, the cost of a bad hire also increases. In some cases, making the wrong hiring decision at the CEO level can cost hundreds of millions of dollars. From the organization's perspective, there are huge financial benefits to including assessments in the selection process, insofar as the information will lead to better hiring and placement decisions. The individuals being assessed also benefit. Employees are more satisfied and advance further in their careers when their skills and personal attributes match the requirements of their jobs.

There are other benefits to selecting the right people. Jim Collins, author of "Good to Great," talks about "getting the right people on the bus." He notes that most people assume that great business leaders spark change by announcing to the people "on the bus" where they're going - by setting a new direction or by articulating a fresh corporate vision. Based on his research, the companies that moved from good to great were those that began their journey by asking "with whom" rather than "where." In other words, great companies start by getting the right people on the bus, the wrong people off, and the right people in the right seats.

Until next week...

The Value of Employee Assessment (Part 1 of 5)

Today I kick off a five part series on the value of leadership assessment.  Sure, folks tend to agree that assessing skills is generally a good thing, but why?  What role does assessment play and what value does it lend from a business perspective?  These are the questions I'd like to address over the next few weeks... hope you find it helpful, and I'd love to hear your thoughts on the topic.

With that, let's get started with a short post to provide some historical perspective.

For thousands of years, humans have measured work-oriented skills. As far back as the Chan dynasty in 1115 BC, civil servant candidates were subject to competitive examinations to test their proficiency in areas such as archery, horsemanship, military affairs, civil law, writing and arithmetic - the knowledge, skills and abilities (KSAs) deemed important for success in a civil servant role.

It wasn't until World Wars I and II, however, that the science of assessing people for the purposes of selection, placement and training was applied on a truly large scale. The US military used aptitude tests, among other tools, to quickly and efficiently place recruits into roles that matched their abilities. They also developed and used simulations (assessment centers) to screen and train secret agents. The value of assessment was quite evident at that time because of the large number of soldiers that needed to be quickly and effectively screened.

Today organizations spend millions of dollars on assessment tools to help attract, select, promote, train and develop their staff. The motives for assessing people are related to several criteria including productivity, retention, absenteeism, legal requirements and employee development.

Next week we'll get into the meat of the topic by discussing the use of assessment to select better leaders.

The US Supreme Court Ruling on Ricci v. DeStefano – An overview of the case and what the decision really means for test users (Part 1 of 2)

I've been reading quite a bit about the US Supreme Court (USSC) ruling over the last few weeks and have received questions from our clients on what the case was really about and what the ruling might mean for them.

This week I'll provide a brief overview of the case for our readers (for specific details, visit: http://www.lawmemo.com/supreme/case/Ricci/) and in part 2 of my post next week I'll discuss some of the issues with the test and the implications of the case. Here goes...

In 2003, 118 firefighters took examinations to qualify for promotions to the rank of lieutenant or captain in the New Haven Fire Department. The results of the exams showed that white candidates passed the exam at a much higher rate than African American candidates... by almost a 2 to 1 margin. A public debate ensued where some firefighters argued that the test results should be discarded as they appeared to be discriminatory and threatened to file a lawsuit if the City made promotions based on the tests. Another group of firefighters said the exams were fair and threatened a discrimination lawsuit if the City ignored the test results and denied promotions to the highest scoring candidates.

After looking at the disparity in results and some design flaws in the test, the City made the choice not to certify the test results. Here now are the various lawsuits and court rulings leading up to the USSC's decision: 

1. White (and 1 Hispanic) firefighters responded to the City's decision by filing a reverse discrimination suit claiming that discarding the results discriminated against them and was in violation of Title VII of the Civil Rights Act and the Equal Protection Clause of the Fourteenth Amendment.

2. The City defended their stance and argued that if they had used the results to make promotions, they could have been in violation of Title VII by using a selection method that demonstrated disparate (adverse) impact. The case was heard by the New Haven District Court and the plaintiffs' (white firefighters) claims rejected, citing prior cases where "race conscious" interventions were used to remedy adverse impact.

3. The 2^nd Circuit Court upheld the District Court's decision.

4. Enter the US Supreme Court (USSC), which then reversed the decisions of the lower courts (by 5 to 4 margin) stating that the City of New Haven did not have enough evidence/justification ("strong basis in evidence") to throw out the results for fear of being potentially sued for discrimination.

Essentially, the USSC's decision sets the potential precedent that test users cannot just discard results when there is a hint of adverse impact. The case further brings to light the looming conflict between remedying adverse impact (race based decision) and facilitating equal protection (creating a color blind society). It seems like the City of New Haven was in a "no win" situation... either avoid the adverse impact suit at the risk of facing disparate treatment claims or vice versa.

Next week I'll discuss some of the issues with the firefighter exams, why these issues were not fully considered in the USSC's arguments and also what test users should take away from the case.

Leadership Assessment Validity: What is it? Why Does it Matter?

Rarely does a week go by that one of our clients doesn't ask questions around the validity of assessments, be it leadership assessments or other tools... and the topics of validity and fairness have become even more front-and-center as a result of the recent Supreme Court decision in Ricci vs. DeStafano.

In my opinion, all this attention is a good thing... and clients should always be asking vendors these questions about validity.  Why?  Well, the driving factor in many cases is a company's desire to protect itself in the event of legal action, and that's understandable in today's litigious environment. 

However, there's a larger, extremely important, issue regarding the validity of assessments which companies seem to give less attention.  Simply put, valid assessments work better!  They are scientifically proven to predict on-the-job success or failure better than their non-valid (or unvalidated) counterparts.  They are also more accurate in assessing an employee's unique strengths and areas for improvement.

What's this mean for companies?  Better quality hires, more targeted employee development, and better ROI - just to name a few things.

Given it's such a timely topic; we thought you might benefit from a concise, practical definition of "validity."  In this video, I take a couple minutes to interview Industrial and Organizational Psychologist, Dr. Michael Zia Mian on the topic.

Hope you find it beneficial!

Also - if this topic is of interest, be sure to see our next two posts (August 28^th and September 4^th).  Dr. Zia Mian will be discussing the Ricci vs. DeStafano case in detail, and it's implications on HR practitioners.

Should 360 Degree Feedback be used in Performance Appraisals? (Part 2 of 2)

It's hard to refute the basic logic of arming both the manager and the employee being appraised with the most accurate data possible, and to use that information intelligently to help understand past performance and determine how performance can be improved in the future.

360 degree feedback can provide such data. Here are three suggestions on how to capitalize on that opportunity, while minimizing the risks:

1. Initially use 360s only for development - If 360 degree feedback is new to your organization, focus on using the process for development purposes only for a couple of years. Give people time to get used to the whole concept of 360 degree feedback, and to appreciate how valuable such feedback can be. Then consider the possibility of using it in some fashion as part of the PA system.

2. Don't tightly link 360s to the PA system - This means several things. First, 360 degree feedback should be only one of the data sources used by the manager in evaluating performance. Second, 360 degree feedback results should never be hard-wired to PA ratings in the sense that there's any kind of mathematical relationship. Third, don't link the timing of 360 degree feedback to the timing of PAs. Most PA systems use an annual cycle tied in with the company's fiscal year, but conducting 360 degree feedback every year is too often. Conduct 360 surveys when it makes sense to do so for development purposes. Then use the results as one of the inputs to the PA system, if they happen to be timely enough.

3. Clearly communicate how the 360 tool will be used - Clearly communicate to participants and raters how the 360 degree feedback will and won't be used with respect to the PA system. Don't underestimate the challenges in communicating these matters, particularly if you're announcing a policy change (e.g., a shift from "strictly for development purposes" to something else). It will take numerous and repeated communications to allay the fears and concerns on the part of all stakeholder groups - participants, managers and raters.

A reminder to those of you thinking that I've just provided great rationale for you to now quickly integrate your 360 and PA systems, or maybe you've been integrating them for some time and this just validates your stance - the success of the integration really will depend on following the guidelines mentioned and paying close attention to what happens during and after the performance review. A well planned structure is still necessary to ensure that employees actually engage in developing their competencies. While there is too much to cover in this post, some critical questions you should be asking include - Do employees understand your performance review process and how 360 ratings will be used? Are managers equipped to conduct effective feedback discussions in the review? Is there a development planning template and is regular follow-up part of your process?  

It's the age old question ... how effective is your system in helping people change behaviors? Essentially.... that's the bottom line!

Should 360 Degree Feedback be used in Performance Appraisals? (Part 1 of 2)

In the second half of every year, as annual performance reviews approach, I'm frequently faced with the client question "is it appropriate to integrate employee performance appraisals (PA) and 360 results?" There is considerable debate and controversy about the relationship between the two, but from where I stand, 360 degree feedback can play a beneficial role in PA systems, if several issues are carefully managed.

Before covering those issues, let me first describe the broader context of PA systems and one of the reasons they often don't work very well. Most PA systems include two components - evaluation of operational/business goals to be achieved and evaluation of performance factors (how the job is accomplished and the competencies required). Typically evaluations of both components are then integrated into some overall "score" or rating, and there are usually consequences linked to those ratings, such as merit salary increases, bonus allocations, and so forth.

The rationale for looking at performance factors is that companies want managers and their employees to place an emphasis on improving competencies and skills, in addition to operational goals, because improving them increases the potential for future business success.

Let's now talk about why this traditional PA process often doesn't work very well. The one most relevant to my post is that evaluations of performance factors (the behaviors and competencies demonstrated) are not as accurate when the assessment information comes only from the manager. (This is usually less of a problem for operational objectives because more quantitative, objective measures are available.) In an effort to improve this important part of their PA systems, some companies use 360 degree feedback to increase the accuracy of assessments and evaluation. Why? Because having several different perspectives on an individual is better than just one - this is the driving premise of the 360 tool. So, what's the problem and why is there controversy? Here are the four main issues:

1. Decreased rater honesty and accuracy - When they know that significant consequences are tied to their ratings, people providing 360 ratings may be less honest and accurate (i.e., positive or negative bias could be introduced).

2. Participants focus more on ratings than development - Participants might be less inclined, not more, to proactively engage in developmental activities when their PA ratings are influenced by the 360 process. In fact, their main motivation could be to do everything possible to select raters who will be predisposed to give favorable ratings, versus those who will give honest feedback.

3. 360 results lose their "future focus" - 360 degree feedback surveys are more focused on the future (using feedback to improve future performance) whereas PAs are more focused on the past (evaluating goals accomplished and behaviors demonstrated over the past year). Therefore, when put together, there's a risk of the 360 degree feedback process becoming "tainted," and losing its positive focus on the future.

4. 360 tool loses its positive appeal - Associating 360 degree feedback with the PA system (which typically has a very negative reputation in the organization) could jeopardize positive views people hold about the former.

My general advice to clients is to be very careful about incorporating 360 degree feedback into appraisal systems. The advantages of making modest improvements in the PA system could be outweighed by the potential damage done to the 360 degree feedback program. Carefully think through what you're trying to accomplish and why, and evaluate the advantages and disadvantages of moving in that direction.

Looking ahead to next week, I'll dive deeper into how 360 degree feedback can be leveraged while, at the same time, minimizing the risks.

Job Simulation: The future of leadership assessment? (Part 4 of 4)

As noted in my last post, there are many powerful advantages to using simulations for selecting and developing leaders. The problem is that many organizations are finding it more difficult to justify the cost of wide spread use. That said, recent technological developments in the field have made it possible to use established online solutions that are not dependent on human administration and scoring and allow participants to be remote. They can also assess an unlimited number of candidates at once, streamlining the selection process for organizations.

These "next generation" simulations utilize rich media (streaming video and audio) to present critical situations that managers frequently encounter. Participants view these situations and are typically then instructed to select the most appropriate action from a series of options. Since scores have already been assigned to the options, once the assessment is complete, the online system can automatically generate a report for the participant and/or administrator. 

In order for these solutions to generate positive ROI, however, they must be robust, predictive of leadership success, on target for the company's skill requirements, and be multi-purposed (can be used for both selection and development).

It's possible that some organizations might believe in the need to develop their own online leadership simulation. While this may be appropriate for some, leadership skill requirements and the contexts in which they occur are, for the most part, generalizable across companies. I would encourage you to thoroughly research what's already out there, before incurring the time and cost.

Censeo's online leadership simulation - TalentSIM (http://www.talentsim.com/) gets at several critical skills managers need to be successful and is an example of the "next generation" of simulations.

Well, that concludes my series on job simulations - the future of leadership assessment. I hope you have come away from this excited about how far we have come, what we can currently do in the world of simulations and leadership assessment and what might be possible in the near future.

Job Simulation: The future of leadership assessment? (Part 3 of 4)

Last week I summarized the history and evolution of simulations. I'm always struck by the fact that one of its earliest uses was to select spies in WWII. To some degree we might say that simulations played a very important role in the defeat of the Axis powers and the liberation of Europe and other regions. It's also neat to see how this concept was adopted by the business world to assess managers and professionals.

Let's now take a look at some of the specific pros and cons associated with leadership assessment via simulations.

Let's deal with the cons first. Customized assessment centers for managers are still very expensive to develop, usually more than $100K. Off-the-shelf assessment centers may be affordable from a development standpoint, but they're still very time consuming and expensive to administer, considering all the costs involved for participants, assessment center staff, facilities, travel, etc. Even hybrid assessment centers, where some exercises are automated and others are live (e.g., role plays conducted via phone), may cost about $1,500 per participant.

Related to the cost and time requirements, another disadvantage of traditional leadership assessment centers is that they're not very scalable. It may be practical for a company to assess 20 leaders within a short period of time, but not 200. Any assessment process that requires staff for administration and scoring will have scalability problems.

Finally, with some simulations, like leadership assessment centers, in order to make the process more efficient, usually 6-8 participants are assessed at the same time. That makes it very difficult to use the approach as part of a selection system, particularly with external candidates.

Turning to the pros of leadership simulation (and simulations in general), these are the main advantages:

• Most importantly, leadership simulations work better than other types of assessments with respect to their predictive validity. That is, the correlation between assessment center scores and measures of job performance is higher than for other types of assessments. This shouldn't be surprising; since we're directly measuring people's ability to perform the job versus making inferences, such as is done in relating personality profiles to job performance.
• They have high "face validity" and receive positive participant reactions. Job simulations make intuitive sense to HR professionals, managers, participants, and the legal community (compliance agencies, courts, and juries) as being fair and job-related. In our increasingly litigious society, this is an important advantage.
• Job simulations provide a realistic job preview. The opportunity for participants to "try on" the job for a while affords the opportunity for those who may not really want that job to self-select out of the process.
• They lead naturally to development planning because we're gaining information on skills and competencies, which can be developed. Personality tests would be at the opposite end of the spectrum.
• When job simulations are objectively scored, they can help reduce the bias present in selection interviews, or in nominating processes for identifying high potential employees.
• Simulations are very engaging. This may not seem to be an important advantage, but an assessment that participants have fun taking is better than one where they're bored to death. And the assessment process itself is sort of a branding for the company to both candidates, and to current employees taking the assessment for development purposes.

Stay tuned next week for the final installation of my series on job simulations, when I'll talk about the integration of online technology and its impact on leadership assessment.

Job Simulation: The future of leadership assessment? (Part 2 of 4)

Assuming you now understand the notion of job simulation and the different types that exist from part 1 in my series, let's briefly review its remarkable history and evolution.

The history goes back thousands of years, at least to 1115 BC in the Chan dynasty where civil service candidates participated in simulations to test their proficiency in skills such as archery, horsemanship, and military affairs. It wasn't until World War II, however, that scientific rigor was applied to job simulations - interestingly, to help select spies. Candidates were placed in realistic "spy situations," such as leading a team to accomplish some mission. But, unbeknownst to the candidates, some of the team members were actually confederates, intentionally thwarting their attempts to accomplish the mission.

Based on the military's work, Industrial-Organizational Psychologists at AT&T and General Electric, conducted research in the 60's and 70's on the use of job simulations in measuring managerial effectiveness in business settings. These leadership assessments became known as the "assessment center method." The research demonstrated that assessment centers worked better in terms of predictive validity (and other criteria, discussed later) than just about anything else. There was a sharp increase during this period in the number of companies using assessment centers as part of their leadership selection and development programs. While assessment centers are still popular for higher level leadership positions, interest in using them for lower level positions has waned because of the very high costs for development and administration.

The resurgence of job simulation in the last ten years is due primarily to one factor - the application of computer technology to make simulations cheaper, faster, easier, and, with the latest products on the market, completely scalable. It started with using computers to help assessors score participants' performance and generate feedback reports, and then evolved to presenting at least part of the simulation via the computer (e.g., electronic in-baskets). Today, advances are being made in creating highly predictive, and yet still very realistic, job simulations where everything happens online via the internet (I'll say more about this in the last part of the series).

The challenge assessment developers face is that it is much more difficult to create good online job simulations for complex jobs, such as managers than it is for say a call center customer service rep. The latter job is perfectly aligned with what one can do in a simulation - working at a computer and looking up information, interacting with customers via phone versus the many face-to-face interactions required in a managerial role.

Hopefully you learned some interesting facts about the evolution of simulations. Be on the look out for part 3 of my series next week, when I'll present the pros and con's of this type of assessment.