Training Evaluation Strategy

Like This Article? Share It!

Our community is filled with passionate and educated individuals who are true examples of problem solvers. It is without question that training employees is incredibly important to the business function and future of the organization as a whole.

Since this is the case, it is important for the training function to be ringing it’s bells in-tune with overall business objectives and desired outcomes. From the chart below (Figure 12), it is obvious that something is missing in connecting training to performance within the enterprise. It seems we can identify lack of insightful data as the culprit in not being able to predict business outcomes, verify learning objectives, and prove ROI for educational efforts. Specifically, each piece of learning content that maps to the behavior or skill that it is trying to effect should have a data story to tell.

ATD2.pngSource: ATD; Evaluating Learning: Getting to Measurements That Matter, 2016

 

If you missed it, our last blog post covered an interoperable data specification that enables you to obtain quantifiable learning data and performance data to prove business outcomes and tell an insightful story. Assuming contextual learning and performance activity data is available, this post will discuss some ways that this data can now enable you in your business. We are also assuming that you want the activity data. Comprehensive learning activity data can expose information that you are now culpable for. For instance, here is a business case: leadership does not want to understand that an employee, Jack, struggled with the ethics and compliance anti-bribery training and attestation because if Jack violates the policy that he failed the training on, the organization may be culpable. The result of this business case is that the ethics and compliance training produces no data other than if Jack completed the training or not. There are elephants such as this in the room. Again, we are assuming you want all of the data in this post; and if you want the data, you are likely aware of the ways you can start using the data. If you are using Kirkpatrick’s levels of training evaluation, an immediate way we can use contextual training and performance data is to better inform the levels of the model.

If you are unfamiliar with the 4 levels of the Kirkpatrick model, there are many in the industry today who evaluate at least Level 1 and 2. Level 1 is “Reaction” – which is essentially surveys about how satisfied learners felt with the training, how engaged they felt with the training, and how relevant they felt the training was to their jobs. If you want to better inform Level 1 you must put thought into the design and psychology of the surveys to ensure honest results. Then you must be willing to evaluate the data honestly. Level 2 is “Learning” – This can be through assessments and pre/post tests to evaluate learners on the material before and after the training. To level set, very often the assessment strategy is not doing as much good as it could be. For instance, offering a 5 question quiz after 40 minutes of training may not net you the data you really need. If you want to better inform Level 2 you can include data in the submitted answers that indicates which learning objectives and content the question came from. In our Learnpoints and Waypoints products we handle assessments with a question pool with question id’s and we include the question and answer in the data. If you look at Figure 3 below you see a 20 question quiz pool that has been answered by a couple thousand learners. The quiz is delivered 10 questions at a time and if the user does not pass they are given 10 more randomly. The red plot line is showing the anomalies. The dips in the line are questions that many people are having a hard time answering. Is it because of a bad question? What training content did this question come from? Proper sustainment of training involves this kind of evaluation for validating your assessment and your training approach. This is an example of a better informed Level 2 Kirkpatricks.

PassRate.png

One of ATD’s more recent studies “Evaluating Learning” suggested that Levels 1 and 2 “do not cover the actual application of skills to work activities or the business results of learning programs.” If you refer to Figure 2 below, you see that Levels 1 and 2 are the most used but the least valued (refer to Figure 10). This tells us the issue for many organizations lies in honestly getting from Levels 1 and 2 into levels 3 through 5. Note that level 5 was added in recent years which helps determine ROI.

5levelsused.png

Source: ATD; Evaluating Learning: Getting to Measurements That Matter, 2016

Although these first two levels are important – what’s missing to cross the bridge from qualitative into quantitative?

Here are a few honest questions we can start to ask ourselves:

  1. Do the learning professionals in your organization want to get into the world of data-driven results?
  2. Is your organization able to map contextual training content (learning objectives) to the desired outcomes (performance)?
  3. If “Yes” to 2, are you able to get at the learning activity data and the performance data so that you can evaluate it?

If you’re still wondering what your first step should be in getting the data to unlock Levels 1 and 2, read our blog post on xAPI and the LRS. Essentially, when you are able to gain insights and correlate the training and performance of each piece of learning content that maps to the specific piece of knowledge, behavior, or skill that it is trying to effect, you kick open the door to better informing all the levels of evaluation. This post is not solely about Kirkpatrick’s, it is about informing you that there is an easy path to gaining insights. It is easy and you can start doing it today.

Stay tuned for “Better Inform Your Training Evaluation (Part Two)”

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>