Job Simulation: The future of leadership assessment? (Part 4 of 4)

As noted in my last post, there are many powerful advantages to using simulations for selecting and developing leaders. The problem is that many organizations are finding it more difficult to justify the cost of wide spread use. That said, recent technological developments in the field have made it possible to use established online solutions that are not dependent on human administration and scoring and allow participants to be remote. They can also assess an unlimited number of candidates at once, streamlining the selection process for organizations.

These "next generation" simulations utilize rich media (streaming video and audio) to present critical situations that managers frequently encounter. Participants view these situations and are typically then instructed to select the most appropriate action from a series of options. Since scores have already been assigned to the options, once the assessment is complete, the online system can automatically generate a report for the participant and/or administrator. 

In order for these solutions to generate positive ROI, however, they must be robust, predictive of leadership success, on target for the company's skill requirements, and be multi-purposed (can be used for both selection and development).

It's possible that some organizations might believe in the need to develop their own online leadership simulation. While this may be appropriate for some, leadership skill requirements and the contexts in which they occur are, for the most part, generalizable across companies. I would encourage you to thoroughly research what's already out there, before incurring the time and cost.

Censeo's online leadership simulation - TalentSIM (http://www.talentsim.com/) gets at several critical skills managers need to be successful and is an example of the "next generation" of simulations.

Well, that concludes my series on job simulations - the future of leadership assessment. I hope you have come away from this excited about how far we have come, what we can currently do in the world of simulations and leadership assessment and what might be possible in the near future.

Job Simulation: The future of leadership assessment? (Part 3 of 4)

Last week I summarized the history and evolution of simulations. I'm always struck by the fact that one of its earliest uses was to select spies in WWII. To some degree we might say that simulations played a very important role in the defeat of the Axis powers and the liberation of Europe and other regions. It's also neat to see how this concept was adopted by the business world to assess managers and professionals.

Let's now take a look at some of the specific pros and cons associated with leadership assessment via simulations.

Let's deal with the cons first. Customized assessment centers for managers are still very expensive to develop, usually more than $100K. Off-the-shelf assessment centers may be affordable from a development standpoint, but they're still very time consuming and expensive to administer, considering all the costs involved for participants, assessment center staff, facilities, travel, etc. Even hybrid assessment centers, where some exercises are automated and others are live (e.g., role plays conducted via phone), may cost about $1,500 per participant.

Related to the cost and time requirements, another disadvantage of traditional leadership assessment centers is that they're not very scalable. It may be practical for a company to assess 20 leaders within a short period of time, but not 200. Any assessment process that requires staff for administration and scoring will have scalability problems.

Finally, with some simulations, like leadership assessment centers, in order to make the process more efficient, usually 6-8 participants are assessed at the same time. That makes it very difficult to use the approach as part of a selection system, particularly with external candidates.

Turning to the pros of leadership simulation (and simulations in general), these are the main advantages:

• Most importantly, leadership simulations work better than other types of assessments with respect to their predictive validity. That is, the correlation between assessment center scores and measures of job performance is higher than for other types of assessments. This shouldn't be surprising; since we're directly measuring people's ability to perform the job versus making inferences, such as is done in relating personality profiles to job performance.
• They have high "face validity" and receive positive participant reactions. Job simulations make intuitive sense to HR professionals, managers, participants, and the legal community (compliance agencies, courts, and juries) as being fair and job-related. In our increasingly litigious society, this is an important advantage.
• Job simulations provide a realistic job preview. The opportunity for participants to "try on" the job for a while affords the opportunity for those who may not really want that job to self-select out of the process.
• They lead naturally to development planning because we're gaining information on skills and competencies, which can be developed. Personality tests would be at the opposite end of the spectrum.
• When job simulations are objectively scored, they can help reduce the bias present in selection interviews, or in nominating processes for identifying high potential employees.
• Simulations are very engaging. This may not seem to be an important advantage, but an assessment that participants have fun taking is better than one where they're bored to death. And the assessment process itself is sort of a branding for the company to both candidates, and to current employees taking the assessment for development purposes.

Stay tuned next week for the final installation of my series on job simulations, when I'll talk about the integration of online technology and its impact on leadership assessment.

Job Simulation: The future of leadership assessment? (Part 2 of 4)

Assuming you now understand the notion of job simulation and the different types that exist from part 1 in my series, let's briefly review its remarkable history and evolution.

The history goes back thousands of years, at least to 1115 BC in the Chan dynasty where civil service candidates participated in simulations to test their proficiency in skills such as archery, horsemanship, and military affairs. It wasn't until World War II, however, that scientific rigor was applied to job simulations - interestingly, to help select spies. Candidates were placed in realistic "spy situations," such as leading a team to accomplish some mission. But, unbeknownst to the candidates, some of the team members were actually confederates, intentionally thwarting their attempts to accomplish the mission.

Based on the military's work, Industrial-Organizational Psychologists at AT&T and General Electric, conducted research in the 60's and 70's on the use of job simulations in measuring managerial effectiveness in business settings. These leadership assessments became known as the "assessment center method." The research demonstrated that assessment centers worked better in terms of predictive validity (and other criteria, discussed later) than just about anything else. There was a sharp increase during this period in the number of companies using assessment centers as part of their leadership selection and development programs. While assessment centers are still popular for higher level leadership positions, interest in using them for lower level positions has waned because of the very high costs for development and administration.

The resurgence of job simulation in the last ten years is due primarily to one factor - the application of computer technology to make simulations cheaper, faster, easier, and, with the latest products on the market, completely scalable. It started with using computers to help assessors score participants' performance and generate feedback reports, and then evolved to presenting at least part of the simulation via the computer (e.g., electronic in-baskets). Today, advances are being made in creating highly predictive, and yet still very realistic, job simulations where everything happens online via the internet (I'll say more about this in the last part of the series).

The challenge assessment developers face is that it is much more difficult to create good online job simulations for complex jobs, such as managers than it is for say a call center customer service rep. The latter job is perfectly aligned with what one can do in a simulation - working at a computer and looking up information, interacting with customers via phone versus the many face-to-face interactions required in a managerial role.

Hopefully you learned some interesting facts about the evolution of simulations. Be on the look out for part 3 of my series next week, when I'll present the pros and con's of this type of assessment.

Job Simulation: The future of leadership assessment? (Part 1 of 4)

Job simulations have a long history and have proven to be one of the most effective methods for selecting and developing leaders, however, they have not experienced the same degree of popularity as other assessments, mainly because of their complexity and cost. The growing integration of online technology and assessment tools, however, has created a renewed excitement about the feasibility of simulating these types of jobs.

In the coming weeks, I'll discuss, in a 4-part series, the concept of job simulations and whether I believe it represents the future of leadership assessment.

I'll start with an overview of simulations... and the different types. In Part 2, I'll look back at the history and evolution of simulations. In part 3, I'll present the pros and cons of using this assessment approach.  And to conclude the series, I'll look at how online technology is making simulations more accessible, acceptable and applicable to more organizations. 

Ok... let's get started. In case you are wondering what I mean by "job simulation", this week's focus provides you with an overview of the concept and the different types that exist, including simulations associated with leadership assessment.

In its simplest form, a job simulation is putting a person in situations where you collect data on how well he/she can perform the job, or certain tasks that are part of the job. A typing test is an example of a job simulation, and was heavily used back in the days when people were hired primarily to do typing (and before spell checking). Another example of a simulation would be presenting an automotive mechanic with a vehicle that had several known problems, and asking him/her to identify and fix all the problems.

The most complex types of simulations include flight simulators for training and qualifying pilots, and nuclear power station simulators for plant operators. They are particularly useful in simulating critical, but rarely occurring, incidents (e.g., losing both engines shortly after takeoff, reactor overheating). These are not things you want to practice in the "real world." Finally, an entire industry has sprung up to build complex battlefield simulations for the military.

In terms of leadership assessment, the assessment center method has been used for decades to evaluate leadership skills. It is a combination of several different simulations - in-baskets, role plays, group exercises, problem solving exercises, etc. - and may last from 1-3 days. Sometimes non-simulation measures are also included in assessment centers, such as ability tests and style inventories.

So, regardless of the job, simulations share several things in common. First, they all place participants in realistic situations that occur on the job, and they must use the same skills in the simulation as they would on the job. Second, systematic methods are used to collect data on participants' performance and to evaluate the skills being measured - computer-based scoring for flight simulators and trained staff in assessment centers. Third, structured feedback is given to participants for development purposes, typically on overall performance and on specific skills.

In part 2 we'll explore the history and evolution of simulations and in so doing touch on the reasons for its predictive power and the challenges in developing these kinds of assessments.