Making Training More Accountable

Because of heightened skepticism in reporting, it is more important than everto develop a training return-on-investment methodology that will stand up underintense scrutiny. The ROI methodology must meet certain operating standards tohelp ensure that there is consistency in the evaluation process and that aconservative approach is taken. Standards and guiding principles keep theevaluation credible and allow for replication of the methodology.

When implementing training ROI, here are some guiding principles to use asoperating standards.

Report the complete story: ROI is a critical measure, but it is only one ofmany levels of evaluation necessary to explain the full impact of a program.Once the program has been implemented, you should evaluate how participantsreacted (including their perceived ability to put the training into action), theextent to which participants improved their knowledge and skill levels, how wellpeople are applying the skills on the job, and finally the business impact. Ifmeasurements are not taken at each of these stages, it is difficult to concludethat the results achieved are actually a result of the training and performanceimprovement program.

Enhance credibility: When collecting and analyzing data, use only the mostcredible source. Credibility is the most important factor in the measurement andevaluation process. Without it, the results are meaningless. Using the mostcredible source (often the participants) will enhance the perception of thequality and accuracy of the data analysis and results.

    Be conservative: When analyzing data, select the most conservativealternative for calculations. This principle is at the heart of the evaluationprocess. A conservative approach lowers the ROI but helps build the neededcredibility with the target audience. It is always better to be conservativethan to provide a generous estimate and have results that are not credible.

Account for other factors: At least one method must be used to isolate theeffects of the program. This step is imperative. Without some method to isolatethe effects of the program, the evaluation results will be considered highlyinaccurate and overstated. Some commonly used strategies include:

• A pilot group of participants in a training program is compared with acontrol group not participating in the program to isolate the impact of theprogram.
• Forecasts of anticipated results without the training program are comparedto actual post-intervention results.
• Participants estimate the influence a training program has on key measuresof impact.

Account for missing data: Sometimes training participants leave theorganization or change their job function. If training participants cannot or donot provide post-intervention improvement data, assume that little or noimprovement has occurred. It damages the credibility of the evaluation to makeassumptions about improvements for which no substantiating data exists.

Adjust estimates for error: It’s common to use estimates in reportingfinancial and cost-benefit information. To enhance the credibility of estimateddata, weigh the estimates based on the level of confidence you have in the dataand adjust accordingly.

Omit the extremes: Extreme data items can skew results. To eliminate theinfluence of extreme data items, omit them from the analysis. For example, ifyou have a list of numbers that all range from 30 to 70 except for one instanceof the number 100, the number 100 would be considered an “outlier” orextreme data item and should be eliminated.

Capture annual benefits for short-term programs: Only use the first year ofbenefits in the ROI analysis of short-term programs. If benefits are not quicklyrealized for most training and performance improvement programs, they areprobably not worth the cost. Therefore, for short-term programs, consider onlyannual benefits. Reserve multiple-year ROI analysis for more extensive programswhere implementation spans a year or more.

Tabulate all program costs: The ROI methodology must include all of the costsassociated with the training and performance improvement programs. These costsinclude the initial needs assessment; development; delivery costs includingfacilitator, facility and participant costs; opportunity costs associated withemployees being absent from their jobs during training; and evaluation costs.Although the term ROI has been used loosely to express any of the benefits of atraining and performance improvement program, a credible ROI methodologyincludes monetary costs. Omitting or understating costs will destroy thecredibility of the ROI results.

Collectively, these guiding principles will ensure that the ROI methodologyis credible and that it produces accurate values and consistent outcomes. Italso ensures that the impact study can be replicated–when two or morepractitioners evaluate the same program, they should always result in the samemeasurement.

Workforce Online, September 2002 — Register Now!