My Thoughts | Analytics v Anecdotes
Jeff Bezos (at a recent forum on leadership):
We have tons of metrics, we have weekly business reviews, metric decks, we know so many things about the customers — whether we’re delivering on time, whether the packages have too much air in them — we have so many metrics that we monitor. And the thing I have noticed is that when the anecdotes and the data disagree, the anecdotes are usually right. There’s something wrong with the way you’re measuring it. When you’re shipping billions of packages a year, for sure you need good data and metrics, but then you need to check that data with your intuition and your instincts.
And that got me thinking... I tend to pay little attention to anecdotes at work, when I invest, and here online. But years ago, I was the total opposite.
Is one better than the other? Do you need both in equal measure?
Defining Analytics and Anecdotes
Let's baseline what analytics and anecdotes mean for this post.
analytics | noun
1. the systematic computational analysis of data or statistics.
2. information resulting from the systematic analysis of data or statistics.
anecdote | noun
1. a short amusing or interesting story about a real incident or person.
2. an account regarded as unreliable or hearsay.
3. the depiction of a minor narrative incident in a painting.
The dictionary definition of an anecdote is a little off for what I'm getting at here, but with synonyms like story, tale, and narrative hopefully, you get the picture.
In a nutshell, analytics are the logic/rules-based, unemotional, data-driven, and quantitative assessments of a given subject or activity. Whereas anecdotes are the opinion based, emotional, human-driven, and qualitative assessments of the same thing.
Same Same, but Different
You could argue that analytics and anecdotes are basically the same (or at least very similar) because they both tell a story about something. But they don't tell the same story. Why? Because of the different ways of formulating (sources, methods, and paths) each one. There can be crossover in all three areas and you should see similarities (correlations) in the analytics and anecdotes arising from a single issue.
Analytics (as a product) are generally arrived at as a result of data being passed through a rules-based and referential model so we can say that logic plays the largest part in their formulation. So the structure and boundaries of the data set and model set the basis for the final product. Assuming that anecdotes are "collected in a casual or informal manner and relying heavily or entirely on personal testimony", we could say that human nature plays a large part in their formulation. So the thinking, emotional state, personality, beliefs, etc of those contributing the anecdotes would form part of the final product.
Know Thy Enemy
Without an interpretive context, the facts we get from an analytics-based assessment misses the 'so what' which can dislocate the user of an analytical output from the reality of what is actually going on. Datasets and rule-based models all have limitations, even if you are using artificial intelligence to add new data and/or adapt the model. Those limitations have a direct impact on relevance or applicability of the outcomes depending on the scenario being assessed. Models also have a tendency to reflect the views of their creators, that's not a bad thing - it just needs to be factored into the usability of the output. Reliance on a model-based interpretation of data to produce facts can be dangerous.
Equally dangerous is how a person perceives the situation and/or circumstances that they are commenting on. Those perceptions bring an inherent level of bias into the truth we get from anecdotes. Those biases combined with a lack of knowledge or understanding (subject matter expertise) on the part of the person using the anecdotes can result in a false reality being created and promulgated.
So Which One is Better?
That depends on what they are being used for. In reality, one is not substantially better than the other when you look at the outputs over a variety of subjects. Every situation will require a mixture of both analytics and anecdotes for an effective decision or prediction. Bayesian Search Theory is a great example of using anecdotal inputs from experts as part of an analytical model to make high-quality predictions.
Do You Need Both in Equal Measure?
Not at all. The situation and circumstances for their use should be used to assess the mixture of analytical and anecdotal inputs. Layering the two together or using one to inform the other look like being the best ways of leveraging Analytics and Anecdotes. Without using both, you won't get the full picture.
And Your Point Is?
I reckon that I haven't been paying enough attention to anecdotal inputs. In a world obsessed with quantitative measurement the 'human factor' remains and is often forgotten.
Join Manna