Snake Oil or Science

To catch a crook or untether a dark horse en route to corporate stardom, Robert Hogan, an organizational psychologist and president of Hogan Assessments Inc. in Tulsa, Oklahoma, relies on personality inventory tests. A leader in the industry, he crafts tests meticulously, asking all kinds of questions–and sometimes repeating the same questions in subtly altered ways–to ascertain a job candidate’s tastes and preferences, level of self-confidence, achievement in school, even reactions to wounded animals and big noisy crowds. His purpose is to improve staffing decisions and thus increase ROI by giving Fortune 1000 corporations and government groups a sense of individual personality and “fit” with job requirements or corporate culture. Hogan calls his tests a probative revelation of the “bright side, the dark side and the inside.”

For example, one company, Overnight Transportation in Atlanta, was able to reduce on-the-job delinquency such as fighting, drunkenness and damage to goods or vehicles by 50 to 100 percent by using the Hogan Personality Inventory to perform mass screenings of 1,500 job candidates. Another 1,500 candidates were used as untested “controls.” Savings: more than $1 million each year. “A single accident alone can cost a company $100,000,” Hogan says.

Hogan is the editor of the book Personality Psychology in the Workplace, the bible of a movement that makes personality testing and “behavioral competence” a job-recruitment necessity. But although personality testing now touches the lives of millions of job candidates and employees at every level, not all of it is defensible. “Of the 2,500 test publishers out there, only three or four are legitimate,” Hogan says. “Most of these people are selling snake oil.”

He and other experts cite poor test construction, what Hogan calls “laughable” research-validation techniques, confused goals and test misapplication and abuse as testing problems. Even more, “lots of people are selling tests and lots of people in human resources don’t know what they’re looking at,” says Glenn DeBiasi, an industrial psychologist and corporate vice president of human resources at Alex Lee Inc., a $2.4 billion food company in Hickory, North Carolina. “While many human resources folks lack the background in statistics or research methodology to evaluate the tests, if you look behind the scenes, most test publishers themselves haven’t done a good job of demonstrating that the tests predict anything of value.”

A hedge against bad hires
Hogan believes that his tests, at the very least, measure five key dimensions of personality, a now widely accepted psychological construct that names likability, extroversion, prudence (conscientiousness), “intellectance” (Hogan’s term for intellectual openness and creativity) and capacity to adjust as the most important. Various job descriptions call for different dimensional profiles. For example, job recruits with a high dose of likability are ideal for customer service. “You get charming, affable, helpful people,” he says. Executives with high levels of prudence, he says, “follow rules, plan ahead and respect authority.” At the low end, you get the former Enron executives who were impulsive, manipulative and deceitful. Hogan’s methods of research validation tie individual test results to projections of performance in specific job roles.

But according to Ben Dattner, who earned a Ph.D. in organizational and industrial psychology at New York University in 1999, tests mirror the people who design and commission them. Screening tools provide a political hedge and seeming defense for human resources, ostensibly to lower the risk factors associated with “wrong hires” and “undesirable turnover,” says Dattner, president of Dattner Consulting Inc., an executive performance consultancy in New York. “A conservative estimate of the cost of wrong hiring decisions or undesirable turnover is one year’s [employee] salary,” he says, citing Saratoga Institute figures. The estimate reflects the time it takes to recruit, staff, train and develop individual talents.

Gary Snodgrass, an executive vice president for human resources at Excelon Corp., a Chicago energy company formed in October 2000 from the merger of PECO Energy and Commonwealth Edison, uses multiple personality measures, work simulations, biographically based interviews and leadership-assessment measures to reveal what he describes as a composite “behavioral competency.” He likens the complex management evaluation process to “peeling an onion.”

“The process includes assessments of behavior patterns, leadership skills, delegation skills and problem-solving. We use different kinds of assessment tools based on position.” For example, the company’s candidates for middle-management positions get a battery of assessment tests that stress realistic work scenarios (also known as situational tests), structured interviews and role-playing. Industry assessment leaders such as Personnel Decisions Inc. and Development Dimensions International craft and validate these tests. “We’ve based them on assessments within our business units in which we’ve looked at the success factors involved in these jobs and measured incumbent performance ratings,” Snodgrass says. In high-level management, the company utilizes biographically based assessment interviews, structured questions that take several hours to complete. The tests work well, he says; since 2002 more than 80 percent of Excelon employees picked for selection and development have been promoted or retained their current positions. The company also scored a 15 percent improvement in “ready now” candidates deemed prepared to take on an incumbent’s job. But while composite testing methods may be an effective approach to evaluation, personality alone certainly does not determine behavior on the job, Dattner says.

“Basically, many research studies have shown that role is a better predictor of [workplace] behavior than is personality, and that when people change roles, their behavior changes. For example, people moving into sales become more extroverted because their roles demand that they be outgoing.

“There is an ongoing debate in psychology about the respective importance of ‘person’ versus ‘situation’ which is analogous to the debate between ‘nature’ and ‘nurture,’ ” Dattner says. “Organizations [produce] ‘strong’ situations that can heavily influence behavior, which is why corporations acquire their own unique cultures.” This leads Dattner to his pet criticism, namely that while corporations are well intentioned in devising tests and assessments, most of them measure the wrong things.

The problem is “fundamental attribution error,” which means that companies tend to overplay the contribution of individual traits and under-analyze the “big normative personality” of the organization.

The next big thing
John Scott, a test designer and vice president of Applied Psychological Techniques Inc., an assessment company in Darien, Connecticut, says that “situational” or “simulations” testing is becoming more popular, but much of it is bleeding edge.

“Organizations that want to put simulation-based assessment systems in place are getting very busy with us now, but as testing organizations, we need to show that these tests are valid,” Scott says. “To do that, [we] need to go through a structured process, to follow the proper test procedures, to be consistent in measurement and to go in and do a job analysis to find out what are the required skills for the job.”

Test designers must also prove that the findings of a proposed test relate back to job performance, first by testing incumbents, then by conducting additional research. To ensure predictive accuracy, tests must be designed so that high-performing incumbents should do well on the test. Test designers must prove not only that the tests are legitimate but also that they eliminate cultural and ethnic bias wherever possible, Scott says. “The bottom line is if you do your job right, you’re protected under the law; you’re going to increase ROI and reduce turnover.”

Nonetheless, no assessment tool is perfect. John Fey, a human resources manager of operations at Ameren Corp., a gas and electric utility company in St. Louis, says that “pure” psychological tests are too risky. He prefers APT’s “situational” tests, which are about to become a regular part of Ameren’s management-screening battery. “When you look at a test and say someone doesn’t have a good fit for this organization, what does that mean? It’s one thing when you don’t have the [technical] aptitude to do a power plants job, and that’s demonstrated on a test. People can accept that. But when you say, ‘Your personality doesn’t fit our culture,’ you set yourself up for legal challenges.”

Workforce Management, October 2004, pp. 90-92 — Subscribe Now!