Net Promoter Score Methodology Is Popular But Has Many Vocal Critics

Measuring customer satisfaction is a critical yet surprisingly complex endeavor for a company that wants to deliver "Perfect Service." It is the process that measures accomplishment in acheiving vision-critical goals, while fueling improvement initiatives in priority areas.

Obviously, asking customers how satisfied they are with a recent transaction is the most basic question of all, but for it to be truly meaningful, the question(s) itself must be just right....simple and specific.

One measurement technique, Net Promoter, has become increasingly popular over the past few years. According to its website, Net Promoter was developed by Satmetrix, Bain & Company, and Fred Reichheld, and the concept was first popularized through Reichheld's book "The Ultimate Question." The link to the site is http://www.netpromoter.com/np/index.jsp

In short, Net Promoter "is based on the fundamental perspective that every company's customers can be divided into three categories: Promoters, Passives, and Detractors.

By asking one simple question — How likely is it that you would recommend [Company X] to a friend or colleague? — you can track these groups and get a clear measure of your company's performance through its customers' eyes.

Customers respond on a 0-to-10 point rating scale and are categorized as follows:

---Promoters (score 9-10) are loyal enthusiasts who will keep buying and refer others, fueling growth.
---Passives (score 7-8) are satisfied but unenthusiastic customers who are vulnerable to competitive offerings.
---Detractors (score 0-6) are unhappy customers who can damage your brand and impede growth through negative word-of-mouth.

To calculate your company's Net Promoter Score (NPS), take the percentage of customers who are Promoters and subtract the percentage who are Detractors."

For example, if a company's survey results found that out of 100 surveys, 50% scored 9-10, and 30% were 7-8, and the remaining 20% were 0-6, then the company's Net Promoter Score would be 30. The higher the score, the better. A negative score indicates real trouble. (Almost every Healthcare company scored a negative NPS, with CIGNA scoring a low -30.)

Companies use this score to determine, on a relative basis, how well they are satisfying their customers. It is a simple, easy-to-understand, method. Customer Satisfaction surveys around the world now include this one question.

However, the score methodology has numerous critics, citing flaws in its approach and lack of depth of information.

Barry Dalton, in a recent post from his blog "Customer Service Stories...And Other Posts", declared "there seems to still be an increasing number of customer service practitioners talking about NPS and staking a significant portion of their customer service time and resources on collecting it. At the same time, oversimplification of the concept is epidemic, creating a clouded view of the company's performance vis-a-vis the customer's perception." http://custservicestories.blogspot.com/2010/04/ultimate-question.html#comments

His main criticism is that the information gathered from this question is too generic to be meaningful. He believes that surveys should evaluate the specific interaction that took place, and not the intended bahavior (the recommendation to friend or colleague) after the fact.

Taking the critique a step further, Ron Shevlin, posted a scathing comment on Dalton's blog:

Net Promoter Score is the biggest bunch of snake oil to come to the world of management in 25 years. There's so much wrong with the metric, it would take a couple of hours to just begin to list the reasons.

Shevlin goes into detail on his objections with the score in his blog "Marketing Whims." http://marketingroi.wordpress.com/2007/01/11/stop-measuring-your-net-promoter-score/

In his post, he lists the reasons why companies should stop using Net Promoter Score in their surveys:

• Doesn’t help explain WHY a customer would recommend the firm.
• Measures intention, not behavior.
• Doesn’t capture inherent consumer differences.
• Can incent undesirable behavior.
• Uses funds better deployed elsewhere

"It’s been said that you can’t manage what you don’t measure. But a metric that doesn’t help you manage isn’t worth measuring. And the Net Promoter Score is a measure that doesn’t help you manage."

Next Post: My View On Net Promoter Score

Important Lessons In Collecting Satisfaction Feedback--Purchasing A Used Car From CarMax

After looking at the prices of larger, well-equipped new cars at local dealerships, my wife and I decided it made sense to check out CarMax, where we thought we could find what we wanted at a reasonable price.

Of course, we were nervous about buying used...I always think I am being taken advantage of whenever I am buying a car. The end result is typically an expensive nice car with expensive extra features I don't want and expensive warranties I don't need.....but at least we have a nice car. With a used car, I'm not even sure we'll get that!

So three weeks later, we are driving home in our newly-purchased auto from CarMax. (Yes, I bought the warranty....but the car price was fixed so I knew I wasn't paying more than anyone else...).

My wife was happy that she is now driving a nice car, and I was happy that I didn't have to pay a new car price for the "luxury" vehicle. I was also intrigued by the business model at CarMax and was overall pleased by the process that didn't make me feel like prey waiting to be pounced on by carnivorous salesmen.

So now it is several weeks later, and the CarMax Customer Satisfaction Survey arrives in the mail. When my wife and I sit at the kitchen table this morning to evaluate, I begin to understand that we view the experience very differently. And rating CarMax was not going to be easy.

What Is Being Evaluated?

CarMax's stated mission is to sell "great quality cars at low prices with exceptional customer service." To achieve that goal, the survey states that our "honest assessment" of the buying experience was needed. On the survey is our name, our salesman's name, and some coding presumably tracking back to the purchase...so nothing anonymous here. Also of note is that the survey is from CarMax itself, and not a third party like Dalbar.

My approach to the survey is to evaluate the "buying experience," using past miserable transactions as my benchmark. In that regard, I found the experience positive.

My wife's approach is to evaluate her satisfaction with "the car" as her primary satisfaction criterium, presumably using a problem-free new car as her benchmark. Since her car has been in the shop for a week fixing things we did not see on the lot, she is finding the experience problematic.

Both are legitimate approaches, based on the customer's expectations and definitions of satisfaction. And both viewpoints must be addressed for CarMax to achieve its stated mission.

The CarMax Survey

The CarMax mission can be broken down into three parts: great quality cars, low prices, exceptional customer service. Presumably, satisfaction must be evaluated across all three categories. Of course, CarMax may view the three categories with unequal weighting, such as focusing mostly on "exceptional customer service." If they did, they might be missing something important!

CarMax's survey starts with Satisfaction and Loyalty questions, including the ubiquitous "How likely is it that you would recommend CarMax to a friend or colleague?"

The survey then dissects the buying experience:

• Greeting at the Store
• Wait List
• Selecting A Vehicle
• Product Knowledge
• Communication Skills
• Appraisal Process (in case I wanted to sell my car to CarMax)
• Competitive Performance (versus other dealerships)
• Business Office/Paperwork

Then the survey veers into gathering information about how we became aware of CarMax and our shopping process and past experiences with CarMax, none of which evaluate our experience.

The survey, using a quantitative bubble answer format, leaves no room for explanation. In its instructions, however, the respondent is told he or she may use a separate piece of paper or log onto the website.

Clearly, CarMax has a business formula that is carefuly crafted in its mission statement: great quality cars at low prices with exceptional customer service. I am wondering, however, whether the survey will capture what it is looking for? No survey questions asked about our satisfaction with the vehicle we bought. Only one question asked about the price, and that only in relation to other dealers.

Our response to the survey?

Experience with sales process--Very positive (5 out of 5).
Will we recommend CarMax?--Not at all likely (1 out of 11).
We we buy from CarMax in future?--Not at all likely (1 out of 11).

My View

CarMax has identified the right buttons to push for a great buying experience. Its mission is simple and clear: great cars, low prices, exceptional customer experience. But all of these elements must be working in order for the customer to be truly satisfied. If one of these traits goes awry, then the whole experience is sour.

It appears that CarMax has focused its attention on its service, and has done a great job at making used cars sales a "less risky" and more "professional" transaction. Clearly the sales process has been carefully scripted and choreographed, and its survey asks for evaluation of each step. Good job here.

But it also appears that the other two elements of its mission (price and quality of car) may need additional attention. The best way to understand this is to ask. And CarMax missed the opportunity in its survey.

CarMax customer experience analysts will scratch their heads when they read our survey. It will say:

You did everything great BUT we are not satisfied nor loyal.

And they will not know why.

How Do You Know If You Are Truly Satisfying Your Customers?

How do you tell if you are truly satisfying your customers? It is a very difficult question, one that many companies struggle to answer. In his excellent blog "Buzz Tank" http://www.buzz-tank.com/ John Oswald describes the dilemma:

A lot of the companies that I work with measure their customer experience by measures that wouldn’t really make sense to a customer. Things like First Call Resolution, Right First Time, Cycle Time and so on. Great measures (sometimes) but not terribly meaningful from a customer point of view.

If these common measurements are not the right measurements, then what should we measure?

It could be as simple as what Frederick F. Reichheld suggests in his book "Ultimate Question." Here he believes the answer to one question is a key measure: Would you recommend this company to a friend? The question has interesting implications, but ultimately requires additional queries to find out "Why" or Why Not?" to make the responses actionable.

But is it okay to be directionally correct with your questions? I think that companies that strive to be service leaders must be more precise in their satisfaction measurements. The data that is gathered is not just for "scoreboard" purposes, but also to drive analysis and improvement. Not having good fuel for this entire process is like saying that your car will be okay as long as you put liquid in the gas tank. The better the fuel/data, the better the car/organization will run!

My View

I took the time to comment on the Buzz-Tank blog about measurements, and the quest for "Perfect Knowledge." In that response, I state that I believe that by establishing the "Perfect Guarantee," companies are forced to get to the core of what satisifies customers. One technique of doing that is for customers to describe the "Perfect Service Provider." All efforts--measurement and improvement--can be geared toward that goal.

The response is below:

John--

In my own design, called “Perfect Service,” I believe that discovering what is really important to the customer is so critical that it must almost be forced on the company delivering the service.

Therefore, I advocate using a very strong and material guarantee of customer satisfaction as the rallying point for customer and delivery alike. No sense answering the phone quickly if the conversation is not satisfying, even if you have complied with the letter of a service agreement.

Once you have established the “metric” of customer satisfaction, then the company must discover what is meaningful for that client. Another technique I use is to survey potential and current clients and ask them to describe the “perfect” provider. They will use characteristics that are meaningful to them, not production metrics that are meaningful to the service operation.

By setting up measurement systems and continual improvement processes, you are prioritizing based on a customer satisfaction target. Of course, if you don’t get it right, you end up paying on the guarantee.

Good stuff.

"Perfect Knowledge"--The Data Lives! (Part 2)

As I mentioned in previous posts, the client satisfaction data is the blood or the fuel that circulates through the system, firing up activity. (Sorry for the mixed metaphors....) I cannot stress this enough: the data tells you how you are doing, tells you where you need to work, and tells you if you are making any improvement.

Asking To Be Criticized

Getting that data is so important that companies need to ask for feedback often, and thank customers for providing the feedback...especially if it is bad! Most companies will shy away from receiving bad news, acting almost personally hurt if a customer tells them they are not satisfied.

Your customers must understand that part of the service "contract" you have with them is that they will provide you with their "truth" so you can improve it. If you do not get that truth, the whole service process will break down.

We should "invite" criticism. We reward customers who respond to surveys...no matter what they indicate. We celebrate a partner customer who feeds us the fuel for our service engine. Maybe we even award a customer with a certificate the points out a flaw in our process. Companies that desire top-tier service must open the floodgates of client evaluation.

When The Data Is Received

There are several tactical steps that must take place once a client response is collected:

1. Must acknowledge the receipt of the feedback. Since most likely the feedback is via internet or email, a thank you note should be sent immediately. In the message is the reinforcement about the importance of the survey and the next steps that will follow.

2. If the survey result is less than stellar (I used any score less than a 4 or 5 rating on any question on 5 point scale), a followup contact is made with the client. The purpose of the call is to seek more "context" on the score. This detail is carefully documented since it will be used by other teams for future root cause analyses. If the survey result is a "3 rating," this contact will be made by the "relationship or client manager" and the service manager responsible for the type of transaction.

3. Should the response on the survey be a 2 or 1 rating, senior management will attend the call. This is important because it indicates the seriousness to the client and team of the urgency the company has in understanding and resolving the issue.

4. The client manager commits to the customer that a formalized report will be created, with actions outlined, and presented to the client in the near future.

The result of this activity is that some aspect of service did not meet the client expectation, the service provider asked for and received this feedback in realtime, the service team quickly contacted the client for more information, and a commitment was made to present solutions quickly that will prevent a reoccurance.

This process alone will show your client your company is serious about satisfaction. Few, if any, of your competitors are going this far. But your company has just started because you have an asset that no other company possesses...realtime, detailed satisfaction data that you can use to focus everything you do.

Next Post: Perfect Improvement