Lies, damn lies & the wrong statistics, measuring what matters, part 1

Building on this month’s focus, the reality of applying analytics in businesses, let’s tackle the issue of the wrong statistics.

False comfort can be achieved by completing analysis or implementing metrics that look useful but are misleading. In the field of CX especially, you need to be careful what you measure.

So, on this topic I am delighted to welcome back guest blogger Gerry Brown. He has extensive hands on experience of improving customer experiences. We’ve heard from Gerry before on Big Data, the benefits of personalisation & at the Royal Albert Hall.

This time, Gerry shares an excerpt from his latest book “When a Customer Wins, Nobody Loses“. This section on being careful you’re not using the wrong statistics, when applying analytics to CX. Over to Gerry.

The legendary 60 and a need for relevant insights

“It’s really hard to design products by focus groups. A lot of times, people don’t know what they want until you show it to them” Steve Jobs

In 1927 two men, separated by an ocean, but united in desire and ambition, set records in their respective sports that have stood as enduring standards of true talent and immeasurable value to their team. Baseball legend Babe Ruth hit 60 home runs for the New York Yankees, a record that stood for 34 years.

In the same year Dixie Dean playing for Everton in the old First Division of English Football scored 60 goals, a record that has never been beaten, nor equalled. Goals, home runs and baskets tend to generate the most sports headlines, and they have generally been the currency to measure both individual and team success and to drive idolatry.

However, these are frequently overrated in terms of the overall impact on a team, especially if other issues such as a leaky defence, frequent injuries or underperforming supporting players are taken into consideration.

This has parallels in business where similarly spectacular headlines and over-hyped metrics, such as stock price, total sales, and customer numbers, have historically been used as the key indicators to rate success. For many businesses and investors, these have been found to be less dependable, dangerously unstable and unpredictable over the longer term.

We only have to look at the banking or utilities industries in the UK and more recently, share price roller coasters such as BT and Centrica, for evidence of that. While the big numbers can still resonate, more attention is being focused on data that delivers the most relevant and timely insights – the combination of data sets that enable effective and intelligent investment – that lead to success, whether that’s on the field, at the box office or the board room.

The Best Figure – Where Contribution and Value Meet

When Babe Ruth earned $80,000 in 1930 he was asked him why he had made more money than the President, Herbert Hoover, Ruth famously answered, “I had a better year.”  In a statistics driven game this was an early example of having the right insight or data to back-up his claim and measure his success, however glibly it was stated.

Back in England, where scoring was reputationally valued but financially less rewarding, Dixie Dean’s weekly wage topped out at £8, (yes really!) and it was many years before footballers in England were paid a living wage. Dixie once remarked to George Best when discussing players’ salaries, “When I was playing, I couldn’t afford a pair of boots never mind boutiques.” Now it’s off the scale in the other direction and has no relationship to actual sporting talent, organizational intelligence or value to the team. Value is measured more in shirt sales, stadium naming rights and other spurious commercial links that significantly distort the contribution-reward ratio.

Computer says “yes” and “no”

Let’s return to North America, where there has been some game changing (literally) activity in the sporting financial landscape. Although there is still some silly money being thrown around, the concept of salary caps and greater financial prudence is beginning to gain currency and acceptance, although probably out of necessity rather than increased common sense or discovery of a social conscience. This has led to a slow, subtle, but seemingly irreversible change in how sports teams rate, evaluate, and ultimately sign and pay for players.

Major League Baseball is where this change has been most keenly felt and where statistics have generated legions of anoraks for many years. The catalyst for this change actually started in the late 70’s when Bill James, who had no experience as a writer, but had a huge obsession with baseball, started collecting his own brand of statistics and began publishing his Baseball Abstract. What made his approach radically different, was that he took serious issue with many of the statistics that had been historically used to rate players and teams and to demonstrate and value success.

It took some time before James’s statistics had a measurable effect on the game, and many people, both inside and outside baseball, thought of him as an eccentric and misguided journalist or just a bored number cruncher. That was until Billy Beane became the general manager of the Oakland A’s in 1997. The best-selling author Michael Lewis tells the story of Billy, the Oakland As, and his discovery of James’ statistical approach, in his excellent book Moneyball.

He hits an early and resounding home run in the book when he states that Bill James found that “The statistics were not merely inadequate: they lied. And the lies they told led the people who ran baseball to misjudge their players and mismanage their games.”

An intelligent approach to the price of success

The book’s main theme is charting the success of the Oakland A’s and Billy Beane’s role in it. The A’s, who, as a smaller market club, similar perhaps to Bournemouth or Watford in the EPL, simply didn’t have the resources of the bigger teams such as the New York Yankees or Boston Red Sox to attract the top stars and newly minted and often overrated prospects. He tells how Billy Beane used a whole new range of metrics to help the A’s to sign players who not only didn’t show up on other teams’ radar and scouting systems but were thought to be significantly inferior to the highly paid stars and prospects that the other teams were courting.

And this wasn’t just a one-year phenomenon; Oakland has consistently out-performed the richer teams and the long-term financial results speak for themselves. One statistic that puts this into perspective is the amount of money each Major League Baseball team has ‘paid’ for a win. Based on a formula developed by Doug Pappas, a leading authority on baseball finance, over a three-year period the A’s paid around $500 thousand per win. Compare this with the nearly $3 million that richer teams such as the Baltimore Orioles and Texas Rangers spent, for far less success and far more player aggravation and mediocrity.

A new way of thinking – Powerful combinations lead to hidden valuation

Billy Beane started to rethink baseball and looked for new baseball knowledge. He used a systematic, scientific investigation of the sport to utilize data and drive insight that hadn’t traditionally been used to value players. In doing this, he uncovered and mined hidden gems of players that might have otherwise been left to languish in lower leagues, or never make it, all because of the historic talent evaluation prejudices rooted in baseball traditions.

He was able to start looking at players in very different ways and to use measures and evaluation, both physical and psychological, in powerful combinations that showed a player’s true worth and made a lasting impact on baseball history. Billy has just become a minority owner at Barnsley in the English Championship and their fans are clearly hoping that some of his magic can make the transatlantic trip along with him.

What can you learn from Moneyball, to improve your strategy?

Thanks to Gerry for sharing that with us. A useful reminder of the real world reality, that using the wrong statistics can be as misleading as a lie. How could that caution impact your plans for applying analytics? Have you ensured you understand the domain dynamics well enough to measure the right things?

In part two, Gerry will look at how the lessons learnt from Billy Beane and the Oakland As, can help business gain more insight and drive positive customer outcomes.

As I mentioned this blog post is an excerpt from a chapter of “When a Customer Wins, Nobody Loses“, I praised that book in my recent book review & you can get your own paperback copy on Amazon for under £10: