Evaluating Training and Results (ROI of Training)

Sections of this topic

    Evaluating Training and Results (ROI of Training)

    © Copyright Carter McNamara, MBA, PhD, Authenticity Consulting, LLC.

    Sections of This Topic Include

    Also consider
    Related Library Topics

    Learn More in the Library’s Blogs Related to Evaluating Training and Result (ROI)

    In addition to the articles on this current page, also see the following blogs that have posts related to Evaluating Training and Results (ROI). Scan down the blog’s page to see various posts. Also see the section “Recent Blog Posts” in the sidebar of the blog or click on “next” near the bottom of a post in the blog. The blog also links to numerous free related resources.

    Preparation for Evaluating Training Activities and Results

    The last phase of the ADDIE model of instructional design, or systematic training, is evaluation. However, the evaluation really should have started even during the previous phase — the implementation phase — because the evaluation is of both the activities of the trainer as they are being implemented and of the results of the training as it nears an end or is finished. Evaluation includes getting ongoing feedback, e.g., from the learner, trainer and learner’s supervisor, to improve the quality of the training and identify if the learner achieved the goals of the training.

    Perspective on Evaluating Training

    Evaluation is often looked at from four different levels (the “Kirkpatrick levels”) listed below. Note that the farther down the list, the more valid the evaluation.

    1. Reaction – What does the learner feel about the training?
    2. Learning – What facts, knowledge, etc., did the learner gain?
    3. Behaviors – What skills did the learner develop, that is, what new information is the learner using on the job?
    4. Results or effectiveness – What results occurred, that is, did the learner apply the new skills to the necessary tasks in the organization and, if so, what results were achieved?

    Although level 4, evaluating results and effectiveness, is the most desired result from training, it’s usually the most difficult to accomplish. Evaluating effectiveness often involves the use of key performance measures — measures you can see, e.g., faster and more reliable output from the machine after the operator has been trained, higher ratings on employees’ job satisfaction questionnaires from the trained supervisor, etc. This is where following sound principles of performance management is of great benefit.

    Suggestions for Evaluating Training

    Typically, evaluators look for validity, accuracy and reliability in their evaluations. However, these goals may require more time, people and money than the organization has. Evaluators are also looking for evaluation approaches that are practical and relevant.

    Training and development activities can be evaluated before, during and after the activities. Consider the following very basic suggestions:

    Before the Implementation Phase

    • Will the selected training and development methods really result in the employee’s learning the knowledge and skills needed to perform the task or carry out the role? Have other employee’s used the methods and been successful?
    • Consider applying the methods to a highly skilled employee. Ask the employee of their impressions of the methods.
    • Do the methods conform to the employee’s preferences and learning styles? Have the employee briefly review the methods, e.g., documentation, overheads, etc. Does the employee experience any difficulties understanding the methods?

    During Implementation of Training

    • Ask the employee how they’re doing. Do they understand what’s being said?
    • Periodically conduct a short test, e.g., have the employee explain the main points of what was just described to him, e.g., in the lecture.
    • Is the employee enthusiastically taking part in the activities? Is he or she coming late and leaving early. It’s surprising how often learners will leave a course or workshop and immediately complain that it was a complete waste of their time. Ask the employee to rate the activities from 1 to 5, with 5 being the highest rating. If the employee gives a rating of anything less than 5, have the employee describe what could be done to get a 5.

    After Completion of the Training

    • Give him or her a test before and after the training and development, and compare the results?
    • Interview him or her before and after, and compare results?
    • Watch him or her perform the task or conduct the role?
    • Assign an expert evaluator from inside or outside the organization to evaluate the learner’s knowledge and skills?

    One Approach to Calculate Return On Investment (ROI)

    (This section was written by Leigh Dudley. The section mentions HRD — activities of human resource development — but the guidelines are as applicable to training and development.)

    The calculation of ROI in [training and development] or HRD begins with the basic model, where sequential steps simplify a potentially complicated process. The ROI process model provides a systematic approach to ROI calculations.

    The step-by-step approach keeps the process manageable so that users can tackle one issue at a time. The model also emphasizes that this is a logical process that flows from one step to another. ROI calculation to another provides consistency, understanding, and credibility. Each step of the model is briefly described below.

    Collecting Post-Program Data

    Data collection is central to the ROI process and is the starting point of the ROI process. Although the ROI analysis is (or should be) planned early in the training and development cycle, the actual ROI calculation begins with data collection. (Additional information on planning for the ROI analysis is presented later under “Essential Planning Steps).

    The HRD staff should collect both hard data (representing output, quality, cost, and time) and soft data (including work habits, work climate, and attitudes). Collect Level 4 data using a variety of the methods as follows:

    • Follow-up Questionnaires – Administer follow-up questionnaires to uncover specific applications of training. Participants provide responses to a variety of types of open-ended and forced response questions.
    • Use questionnaires to capture both Level 3 and Level 4 data. The example below shows a series of level 4 impact questions contained in a follow-up questionnaire for evaluating an automotive manufacturer’s sales training program in Europe, with appropriate responses. HRD practitioners can use the data in an ROI analysis
    • Program Assignments – Program assignments are useful for simple, short-term projects. Participants complete the assignment on the job, using the skills or knowledge learned in the program. Report completed assignments as evaluation information, which often contains Level 3/Level 4 data. Convert Level 4 data to monetary values and compare the data to cost to develop the ROI
    • Action Plans – Developed in training and development programs, action plans on the job should be implemented after the program is completed. A follow-up of the plans provides evaluation information. Level 3/Level 4 data are collected with action plans, and the HRD staff can develop the ROI from the Level 4 data.
    • Performance Contracts – Developed prior to conducting the program and when the participant, the participant’s supervisor, and the instructor all agree on planned specific out-comes from the training, performance contracts outline how the program will be implemented. Performance contracts usually collect both Level 3/and Level 4 data and are designed and analyzed in the same way as action plans.
    • Performance Monitoring – As the most beneficial method to collect Level 4 data, performance monitoring is useful when HRD personnel examine various business performance records and operational data for improvement.

    The important challenge in this step is to select the data collection method or methods that are appropriate for both the setting and the specific program and the time and budget constraints.

    Isolating the Effects of Training

    Isolating the effects of training is an often overlooked issue in evaluations. In this step of the ROI process, explore specific techniques to determine the amount of output performance directly related to the program. This step is essential because many factors influence performance data after training. The specific techniques of this step will pinpoint the amount of improvement directly related to the program, increasing the accuracy and credibility of the ROI calculation. Collectively, the following techniques provide a comprehensive set of tools to tackle the important and critical issue of isolating the effects of training.

    • Control Group – use a control group arrangement to isolate training impact. With this technique, one group receives training while another similar, group does not receive training. The difference in the performance of the two groups is attributed to the training program. When properly set up and implemented, control group arrangement is the most effective way to isolate the effects of training.
    • Impact Estimates – When the previous approach is not feasible, estimating the impact of training on the output variables is another approach and can be accomplished on the following 4 levels.
    • Participants – estimate the amount of improvement related to training. In this approach, provide participants with the total amount of improvement, on a pre- and post-program basis, and ask them to indicate the percent of the improvement that is actually related to the training program.
    • Supervisors – of participants estimate the impact of training on the output variables. Present supervisors with the total amount of improvement, and ask them to indicate the percent related to training.
    • Senior Managers – estimate the impact of training by providing an estimate or adjustment to reflect the portion of the improvement related to the training program. While perhaps inaccurate, having senior management involved in this process develops ownership of the value and buy-in process.
    • Experts –estimate the impact of training on the performance variable. Because these estimates are based on previous experience, experts must be familiar with the type of training and the specific situation.

    Customers sometimes provide input on the extent to which training has influenced their decision to use a product or service. Although this approach has limited applications, it can be quite useful in customer service and sales training.

    Converting Data to Monetary Values

    A number of techniques are available to convert data to monetary values; the selection depends on the type of data and the situation.

    • Convert output data to profit contribution or cost savings. With this technique, output increases are converted to monetary value based on their unit contribution to profit or the unit of cost reduction. These values are readily available in most organizations and are seen as generally accepted standard values.
    • Calculate the cost of quality, and covert quality improvements directly to cost savings. This standard value is available in many organizations for the most common quality measures (such as rejects, rework, and scrap).
    • Use the participants’ wages and employee benefits as the value for time in programs where employee time is saved. Because a variety of programs focus on improving the time required to complete projects, processes, or daily activities, the value of time becomes an important and necessary issue. The use of total compensation per hour provides a conservative estimate for the
      value of time.
    • Use historical costs when they are available for a specific variable. In this case, use organizational cost data to establish the specific value of an improvement.
    • Use internal and external experts, when available, to estimate a value for an improvement. In this situation, the credibility of the estimate hinges on the expertise and reputation of the individual.
    • Use external databases, when available, to estimate the value or cost of data items. Research, government, and industry databases can provide important for these values. The difficulty lies in finding a specific database related to the situation.
    • Ask participants to estimate the value of the data item. For this approach to be effective, participants must understand the process and be capable of providing a value for the improvement.
    • Require supervisors and managers to provide estimates when they are willing and capable of assigning values to the improvement. This approach is especially useful when participants are not fully capable of providing this input or in situations where supervisors or managers need to confirm or adjust the participant’s estimate.

    Converting data to monetary value is very important in the ROI model and is absolutely necessary to determine the monetary benefits from a training program. The process is challenging, particularly with the conversion of soft data, but can be methodically accomplished using one or more of the above techniques.

    Tabulating Program Costs

    The other part of the equation in a cost/benefit analysis is the cost of the program. Tabulating the costs involves monitoring or developing all of the related costs of the program targeted for the ROI calculation. Include the following items among the cost components.

    • Cost to design and develop the program, possibly prorated over the expected life of the program
    • Cost of all program materials provided to each participant
    • Cost for the instructor/facilitator, including preparation time as well as delivery time.
    • Cost of the facilities for the training program.
    • Cost of travel, lodging and meals for the participants, if applicable.
    • Salaries, plus employee benefits of the training function, allocated in some convenient way.

    In addition, specific cost related to the needs assessment and evaluation should be included, if appropriate. The conservative approach is to include all of these costs so that the total is fully loaded.

    Calculating the ROI

    Calculate the ROI using the program benefits and costs. The BCR is the program benefits divided by costs:

    • BCR = program benefits / program costs
    • (Sometimes this ratio is stated as a cost/benefit ratio, although the formula is the same as BCR).

    The net benefits are the program benefits minus the costs:

    • Net benefits = program benefits – program costs

    The ROI uses the net benefits divided by programs costs:

    • ROI (%) = net benefits / program costs x 100

    Use the same basic formula in evaluating other investments where the ROI is traditionally reported as earnings divided by investment. The ROI from some training programs is high. For example, in sales training, supervisory training, and managerial training, the ROI can be quite large, frequently over 100 percent, while ROI value for technical and operator training may be lower.

    Additional Resources to Guide Evaluation of Your Training

    Evaluating Online Learning

    Learn More in the Library’s Blogs Related to This Topic

    In addition to the articles on this current page, also see the following blogs that have posts related to this topic. Scan down the blog’s page to see various posts. Also see the section “Recent Blog Posts” in the sidebar of the blog or click on “next” near the bottom of a post in the blog. The blog also links to numerous free related resources.

    Go to main Training and Development page.

    For the Category of Training and Development:

    To round out your knowledge of this Library topic, you may want to review some related topics, available from the link below. Each of the related topics includes free, online resources.

    Also, scan the Recommended Books listed below. They have been selected for their relevance and highly practical nature.