August, 2013

This week has seen lots of coverage on the first published results from the NHS’s ‘Friends and Family’ test.

Some love it and some are critical, and a few others are arguing for more use of free text responses online, like TripAdvisor.

They are in luck for there is already growing promotion and use of free text feedback and response sites online. These range from what feel like the more edited and impersonal contributions on NHS Choices (this, for example), to the less moderated (and moderate) ‘Rate my teachers’, complete with troll like behaviour and comments. Why not have a look at a school and teacher you know?

However, even these sort of resources, that do ape the well-known and industry leading TripAdvisor in some respects, can be hard to decipher. What are they telling us? Even TripAdvisor, with the summary statistics and vast numbers of contributions can be hard to interpret and I notice how I draw different conclusions to my ‘family and friends’ when reading about a hotel…

So, let’s test ourselves…

This is the summary review page from the top scoring hotel in Cambridge. Is the hotel any good?

Well it is top in Cambridge. And the recommendation score is 82%. But, in many places round the world the top ones in any city hit 95% or more. Have a look. Oxford has one at 93%. So, The Varsity isn’t looking that good.

And have a look at the graph….what do you look for? Well, for me, a top scoring hotel (at whatever star rating the hotel is operating at) has a ‘5’ (excellent) bar that accounts for over three quarters of all ratings – with the majority of those left in bar 4 (very good). Again, this top scoring hotel is looking a bit dodgy…

What else do you look for? Well I like the free text responses too, see them here for the Varsity. I expect that for any stay some people have a propensity to rate something well, to justify their choice. And some, when disappointed or upset, flip the other way and give a place a 1 or 2 (terrible and poor) for what appears to be a minor misdemeanour, probably due to buyer’s remorse or dashed high expectations (‘satisfaction is a function of expectation’ we know).

I tend to read the latest 10 or 20 reviews, is the average going up or down? And I also look at most of the mid ranking ones – do people score down a place for things that are not or no longer important to me (e.g the lack of a kids club, indoor pool). Actually, when reading the text there are things about the Varsity that sound good (roof terrace, spa overlooking the river, central location, bar and grill with river view) and frustrations that don’t bother me usually (e.g valet parking, pricey car park) and some that do (value for money, noise).

So if TripAdvisor on hotels can be hard to interpret, what about health care? We have written about this a bit last year and in 2008 toward the end of this Business Briefing. Taking some of the logic from my approach to Trip Advisor, I wrote this in 2008 explaining why I thought patient satisfaction with anything less than scores of 90% was misplaced. I repeat it here, what do you think?
“Our ‘five-fold discount’ hypothesis contends that for five reasons the average scores that many organisations are proud of (or even complacent about) probably need to be discounted downward. So what is the case for a ‘five-fold discount’ of satisfaction scores to a lower net level:
1. First, there is a natural patient predisposition to satisfaction. Emotionally, we need the experience to be good. We start with a positive pre-disposition to the experience of health care (unlike the expectation of car sales and estate agency). In some other sectors (eg home improvement, retail) there is a tendency to blame the supplier if it goes wrong — anxiety overwhelms and individuals project all their fears and regrets onto the supplier. We argue that in health care this is a less likely consumer response when completing an evaluation form, due to factors 2 and 3 below.
2. Second, users fear retribution when they next need the service, so are more likely to score high so as not to ‘alienate staff’ that they might depend on again.
3. Third, there is an inbuilt reluctance to blame frontline NHS staff and services: “it can’t be the staff’s fault” is a starting mindset. Poor NHS staff attitudes are often tolerated. There is a common patient perspective that discounts poor performance, almost as if staff are doing it for love and voluntarily. If the actual remuneration and cost of services was clear we wonder if this discount would lessen.
4. Fourth, the methods used (i.e. tick box forms) tend to mask discontent that would emerge with a more open approach (eg listening to patient stories). Take the case of older people. They are often noted to have positive scores, but listening to their anecdotes highlights the many hidden negative narratives. It might be argued that when a service is free at the point of use then consumers are more forgiving. Again, we believe that the right methods will still elicit the deeper views at the heart of any service experience, be it in the public or private sector
5. Fifth, the information available to most service users to make an informed judgement of satisfaction is limited and likely to over-estimate satisfaction due to limited awareness of what could be done or what great practice looks like. This asymmetry may explain the fact that when we have asked many groups of NHS managers and clinicians what their average rating of satisfaction — or compassion — is we get an average (both mean and mode) of 50%.
There is a need for much greater honesty about the level of poor performance. It is essential to break the addiction to excusing current scores. This is even more striking when we realise in the commercial sector that experts in customer service regard any score less than 7 out of 10 as a negative rating.”

And the idea that 7 is the new zero, brings us right back to the controversy about the Net Promoter Score in the NHS…and the insight from TripAdvisor (even before the five fold discount is applied) that what seems like a good score might not be so.