The Case for Net Promoter Score as a Measure of Presentation Effectiveness

When it comes to post-training evaluation forms, the rule of thumb to which I’ve adhered is: high scores may not guarantee learning happened, but low scores often guarantee learning didn’t happen.

For years I’ve tabulated and delivered feedback for countless sessions that have received Likert-scale scores well above 4 (on a 5-point scale), yet I knew deep down that some of these presentations weren’t as effective as they could be. How can we tell if the presentation was engaging and effective if the post-training evaluation scores are always high?

Several weeks ago I attended a Training Magazine-sponsored webinar on training metrics (view the recording here) and I was introduced to the idea of Net Promoter Score as a way to evaluate presentations. After some deliberation, my colleagues and I decided to test this concept out during a recent 2-day meeting. We added one question to our evaluation forms: Would you recommend this session to a colleague?

Following are the scores from our traditional question about whether people were exposed to new insights, information or ideas on a scale of 1-5 (5 representing “Strongly Agree”):

NPS 1

Not too shabby. Apparently we were exposing people to new insights, information and ideas! So that’s good, right? Who cares whether presenters were engaging or boring, stuck to their lesson plans or went off script, all the scores averaged between 4 and 5. Yay for us!

Then we took a look at the same sessions through the lens of Net Promoter Score, and this is what we found:

NPS 2

These scores showed some variance, but it didn’t tell much of a story until we put these scores side-by-side:

NPS 3

People may have been exposed to some new ideas or insights in each session, but would they put their own reputation on the line and recommend the session to any of their colleagues? It depends. There’s a huge difference between presentations that scored above 4.5 and presentations that drew an average of 4.2.

Here are three reasons why I think this matters:

1. A Wake-up Call. In the past, someone could walk away from a meeting with a score of 4.13 and think to himself: “Well, I wasn’t as sharp as I could have been, but people still liked that session, so I don’t really need to work to improve my delivery… and furthermore, who cares about all these adult learning principles that people keep telling me I need to include?!”

However, if that same presenter sees a Net Promoter Score of 6 or 19 or 31 (with a high score potential of 100), the reaction is very different. People suddenly seem a little more interested in tightening up their next presentation – rehearsing a little more seriously, having instructions for activities down cold, sticking more closely to their lesson plans.

2. Before On-the-Job Application, People Need To Remember Your Content. Some L&D practitioners care less about whether a presentation was engaging, instead being wholly focused on whether or not someone actually does something new or differently or better on the job. To this, I say: “Yes, and…”

Yes, better performance is the ultimate goal for most training programs.

And, you can’t do something new or differently or better if you weren’t paying attention to the presentation. You can’t do something better if you forgot what you learned before you go to bed that evening. While better job performance matters, the presenter plays a big role in whether or not people remember the content and are excited to use it when they return to their offices.

3. Marketing Matters. The principle objection to Net Promoter Score as a training evaluation tool, articulated very well in this post from Dr. Will Thalheimer, is that it is designed for marketing, not for training effectiveness. I would argue that L&D professionals must have some sort of marketing chops in order to generate interest in their programs. After all, Dr. Thalheimer also cited a significant body of research that “found that one misdirected comment by a team leader can wipe out the full effects of a training program.”  If influential people wouldn’t recommend your presentation, research shows that you have a problem.

What do you think? Is Net Promoter Score something you’ve used (or are thinking about using)? Or is it a misguided metric, not suitable for L&D efforts?

 

 

 

4 thoughts on “The Case for Net Promoter Score as a Measure of Presentation Effectiveness

  1. Brian: I have been using this question for about a year. I have received similar scores about the “recommendation.” The results need further analysis regarding the learners. Was the course timed correctly for them? Did they think they were at the level to which the material was addressed? Do they think colleagues would benefit in the same way? It is a useful question.

    • Thanks Paul.

      Yes, these numbers don’t necessarily tell the whole story, but they do provide an alternate ending to the story that traditional Likert-scale evaluation questions offer.

      For example, the session that had the highest NPS may never produce real results. We knew it would be a popular session going in, and we knew we didn’t have a ton of follow-up for that session planned. But we did want to simply expose attendees to those concepts.

      The session with the lowest NPS will have a LOT of follow-up and will probably (we hope) lead to a change in the way people do things… and next time, we need to be sure we tighten up the way that content is delivered.

  2. Nice article Brian. Certainly some food for thought. Measuring training and it’s ROI is the holy grail of L&D. Asking if they would recommend a session does change the learner’s perspective and might provide a more considered answer, yet there are many factors in play.
    I would appreciate some clarification on the following:
    Were the sessions different sessions with different groups, or was this a series of sessions to the same people at the same time? I’m interested to understand the reducing trend. If it is the same group of people I would have expected the score to increase towards the end rather than decrease.

    • Thanks Richard!

      You’ve pointed out a flaw in the way I presented the data. The data was organized by session from highest to lowest scores (it didn’t take into account *when* each session appeared on the program). The session receiving the top score actually occurred as the first session on the second day of this conference.

      A more direct answer to your question is that this was a 2-day meeting with a common theme, but each session was about 30-45 minute in length, each covering a separate topic.

      Hope that helps, but let me know if you have more questions.

      One final note (just to re-iterate what I mentioned at the end of the article) – just because there were high scores doesn’t mean we’re expecting better results from those sessions. We’re being intentional about which topics get more follow-up and follow-through. But these scores help us understand a little better which types of instructional strategies worked for our learners, and they helped us understand which presenters may need to tighten up their preparation and delivery next time around.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s